sam package speed degrade when artifact size is large

[deadmanhead1602]

Description

My case is the artifact size around 120MB and >30 lambda functions defined in the template. The package command takes around 8-9 mins to complete.

Steps to reproduce

Using sam package command as follow
sam package --template-file ./template.yaml --output-template-file serverless-output.yaml --s3-bucket dev-sample-pipeline --s3-prefix xxx-lambda-api --debug

A single sample for lambda definition

SearchFunction:
  Type: 'AWS::Serverless::Function'
  Properties:
    Handler: controller/apply_search.handler
    Runtime: nodejs8.10
    CodeUri: ./
    Description: search
    MemorySize: 256
    Timeout: 60
    Role: !GetAtt SampleLambdaRole.Arn
    Events:
      SearchApi:
        Type: Api
        Properties:
          Path: /apply/list
          Method: any
          RestApiId: !Ref AuthGatewayApi

Observed result

Observation 1: Different combinations are tried with variated artifact size and the number of lambda functions.

Case A:
Artifact size = 113 MB
Lambda totel = 37
Time taken = ~9 mins

2018-12-20 10:04:21 package command is called
Uploading to xxx-lambda-api/dcbf39ab78cafcd01580c0927ae4c023  119252142 / 119252142.0  (100.00%)
Successfully packaged artifacts and wrote output template to file serverless-output.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file ./serverless-output.yaml --stack-name <YOUR STACK NAME>
2018-12-20 10:13:26 package command successful

Case B:
Artifact size = 113 MB
Lambda totel = 10
Time taken = ~2 mins

2018-12-20 10:24:17 package command is called
Uploading to xxx-lambda-api/41e7bd78dbe8a008afd8da02f4937407  119251592 / 119251592.0  (100.00%)
Successfully packaged artifacts and wrote output template to file serverless-output.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file ./serverless-output.yaml --stack-name <YOUR STACK NAME>
2018-12-20 10:26:26 package command successful

Case C:
Artifact size = 64MB
Lambda totel = 37
Time taken = ~3mins

2018-12-20 10:35:50 package command is called
Uploading to xxx-lambda-api/fa216743595aa67fad7e2dd27da91876  66809134 / 66809134.0  (100.00%)
Successfully packaged artifacts and wrote output template to file serverless-output.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file ./serverless-output.yaml --stack-name <YOUR STACK NAME>
2018-12-20 10:38:52 package command successful

Case D:
Artifact size = 64MB
Lambda totel = 10
Time taken = 40 secs

2018-12-20 10:40:22 package command is called
Uploading to xxx-lambda-api/a5f08b2f2f8928512d6c895ffb1d2116  66808100 / 66808100.0  (100.00%)
Successfully packaged artifacts and wrote output template to file serverless-output.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file ./serverless-output.yaml --stack-name <YOUR STACK NAME>
2018-12-20 10:41:02 package command successful

Observation 2:
When replace the CodeUri in lambda defination with s3 url instead of './' e.g. s3://bucket-name/xxx-lambda-api/fa216743595aa67fad7e2dd27da91876 . The package command completed instantly.

Expected result

Excluded the S3 upload time, I expected the package (CodeUri replacement with S3 uri) should be fast or not affected by artifact file size.

Additional environment details

1. OS: MacOS 10.14 and ubuntu 14.04.5
2. sam --version: 0.6.0 and 0.9.0

[jhstephe]

I'm seeing the same behavior with running sam package on yaml files with many lambdas defined. Observed in sam version 0.43.0. Any update on a possible fix?

[rafikiassumani]

I'm seeing the same issue. 15-20 small lambda functions with almost only AWS dependencies taking approximately 38 min.