Request for Comments: Auto Create ECR Repos in Guided Deploy (#2675)

CoshUS · web-flow · commit 58a9427867ef · 2021-04-16T15:03:28.000-06:00
* Added Auto Create ECR Design Doc

* Updated Format

* Addressed feedback
diff --git a/designs/auto-create-ecr.md b/designs/auto-create-ecr.md
@@ -0,0 +1,324 @@
+Auto Create ECR Repos in Guided Deploy
+====================================
+
+
+What is the problem?
+--------------------
+
+With the release of Lambda Container Images Support in SAM CLI, customers today have to specify a ECR Repo URI location where images will need to be uploaded by SAM CLI after having been built. This means that customers need to have pre-created resources ready (in this case ECR repos) to go, so that they can supply them during the deploy process. This introduces friction and break in the seamless workflow that `sam deploy --guided` normally offers, with customers having to figure out how to create a ECR repo and how to find correct ECR URI to specify.
+
+Current flow for deploying a template with image based function:
+
+1. Create ECR repo: `aws ecr create-repository --repository-name myecr_repo`
+
+2. Deploy with SAM CLI guided: `sam deploy --guided`
+
+3. Specify image repo: `Image Repository for HelloWorldFunction []: 12345.dkr.ecr.us-west-2.amazonaws.com/helloworldfunctionrepo`
+
+What will be changed?
+---------------------
+
+When deploying with guided, SAM CLI will prompt the option to auto create ECR repos for image based functions.
+The auto created ECR repos will reside in a companion stack that gets deployed along with the actual stack. 
+
+During each guided deploy, the functions and repos will be synced.
+
+
+Each function without an image repo specified will have a corresponding repo created in the companion stack.
+If a function is deleted from the template and has an auto created image repo previously associated with it, the auto created image repo will also be removed.
+
+
+
+There will be an escape hatch to use non SAM CLI managed repos by specifying `--image-repositories` or changing `samconfig.toml`.
+
+Success criteria for the change
+-------------------------------
+
+* No context switching from SAM CLI to creating resources using other tools.
+
+Out-of-Scope
+------------
+
+* SAM CLI will not manage lifecycles, creation and deletion, of the auto created resources outside of SAM CLI. For auto created image repos, modifications to the functions throught other means like console will not modify the associated image repo until the next deployment with SAM CLI.
+* Auto create repo only concerns about guided experience. Repos are assumed to be provided in CI/CD situations. However the option --resolve-image-repos will be added for auto creating repos without going through guided.
+
+User Experience Walkthrough
+---------------------------
+
+`sam deploy --guided`
+
+**Creating New Repos**
+
+
+```
+======================
+Looking for config file [samconfig.toml] : Not found
+
+Setting default arguments for 'sam deploy'
+=========================================
+Stack Name [sam-app]: images-app
+AWS Region [us-east-1]: us-east-2
+#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
+Confirm changes before deploy [y/N]: y
+#SAM needs permission to be able to create roles to connect to the resources in your template
+Allow SAM CLI IAM role creation [Y/n]: y
+Save arguments to configuration file [Y/n]: y
+SAM configuration file [samconfig.toml]:
+SAM configuration environment [default]:
+Looking for resources needed for deployment:
+ S3 bucket: Found! (aws-sam-cli-managed-default-samclisourcebucket-abcdef)
+ Image repositories: Not found.
+ #Managed repositories will be deleted when their functions are removed from the template and deployed
+ Create managed ECR repositories for all functions? [Y/n]: Y
+ Creating the required resources...
+ Successfully created resources for deployment!
+
+Successfully saved arguments to config file!
+ #Running 'sam deploy' for future deployments will use the parameters saved above.
+ #The above parameters can be changed by modifying samconfig.toml 
+ #Learn more about samconfig.toml syntax at 
+ #https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
+
+Uploading to abcdefg/7dd33d96eafcae1086a1356df982d38e 539284 / 539284.0 (100.00%)	
+
+Deploying with the following values
+===================================
+Stack name                 : test-stack
+Region                     : us-east-2
+Confirm changeset          : True
+Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-abcdef
+Image repositories         : [
+ {“helloWorldFunction1”:"12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction1-abcde"},
+ {“helloWorldFunction2”:"12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction2-abcde"},
+ {“helloWorldFunction3”:"12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction3-abcde”}
+]
+Capabilities               : ["CAPABILITY_IAM"]
+Parameter overrides        : {}
+Signing profiles           : {}
+
+Initiating deployment
+```
+
+**Deleting Unreferenced Repos**
+
+```
+Configuring SAM deploy
+======================
+
+        Looking for config file [samconfig.toml] :  Found
+        Reading default arguments  :  Success
+
+        Setting default arguments for 'sam deploy'       
+        =========================================        
+        Stack Name [test-stack]:
+        AWS Region [us-east-2]: 
+        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
+        Confirm changes before deploy [Y/n]: y
+        #SAM needs permission to be able to create roles to connect to the resources in your template
+        Allow SAM CLI IAM role creation [Y/n]: y
+        HelloWorldFunction4 may not have authorization defined, Is this okay? [y/N]: y
+        HelloWorldFunction5 may not have authorization defined, Is this okay? [y/N]: y
+        Save arguments to configuration file [Y/n]: y
+        SAM configuration file [samconfig.toml]: 
+        SAM configuration environment [default]: 
+
+        Looking for resources needed for deployment: Found!
+
+                Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-abcdef
+                A different default S3 bucket can be set in samconfig.toml
+
+                Image repositories: Not found.
+                #Managed repositories will be deleted when their functions are removed from the template and deployed
+                Create managed ECR repositories for all functions? [Y/n]: y
+                Checking for unreferenced ECR repositories to clean-up: 2 found
+                 12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction1-abcde
+                 12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction2-abcde
+                 Delete the unreferenced repositories listed above when deploying? [y/N]: y
+
+        helloworldfunction4:python3.8-v1 to be pushed to 12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction4-abcde:helloworldfunction4-7bfff073dfcf-python3.8-v1
+        helloworldfunction5:python3.8-v1 to be pushed to 12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction5-abcde:helloworldfunction5-7bfff073dfcf-python3.8-v1
+
+
+        Saved arguments to config file
+        Running 'sam deploy' for future deployments will use the parameters saved above.
+        The above parameters can be changed by modifying samconfig.toml
+        Learn more about samconfig.toml syntax at
+        https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
+
+
+        Deploying with following values
+        ===============================
+        Stack name                   : auto-ecr-guided-test
+        Region                       : us-west-2
+        Confirm changeset            : True
+        Deployment image repository  :
+                                       {
+                                           "HelloWorldFunction3": "12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction3-abcde",
+                                           "HelloWorldFunction4": "12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction4-abcde",
+                                           "HelloWorldFunction5": "12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction5-abcde"
+                                       }
+        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-abcde
+        Capabilities                 : ["CAPABILITY_IAM"]
+        Parameter overrides          : {}
+        Signing Profiles             : {}
+```
+
+--resolve-image-repos
+---------------------------
+This new option will have the exact behavior as running guided with confirmation for all image repo related prompts.
+
+To be more specific, this option will:
+1. Create image repos for functions are not linked to image repos with the `--image-repositories`
+2. Delete auto created image repos that do not have functions associated with them anymore
+
+Destructive operations are done for the following reasons:
+1. This will keep a consistent behavior as the guided. In guided, SAM CLI will abort deployment if deletion of auto created image repos is denied.
+2. For UX, this will avoid image repos and functions mapping to an invalid state where orphaned image repos exist. For this case, we will also need to track which repos should be kept and makes the sync less robust.
+3. From security perspective, keeping old image repos will increase the impact radius of an information leakage. A customer might expect a sensitive image repo will be deleted as soon as the function itself is also removed like the guided experience.
+
+Implementation
+==============
+
+CLI Changes
+-----------
+
+* Add new prompt for guided deploy. 
+  * `Create managed ECR repositories for all functions? [Y/n]: y`
+* Add `--resolve-image-repos` to non-guided deploy
+
+
+### Breaking Change
+
+* Not a breaking change.
+
+Design
+------
+
+**Companion Stack Naming Scheme**
+```
+#Escaped StackName with only common accpetable characters
+escaped_stack_name = re.sub(r"[^a-z0-9]", "", stack_name.lower())
+#Escaped LambdaName with only common accpetable characters
+escaped_lambda_logical_id = re.sub(r"[^a-z0-9]", "", lambda_logical_id.lower())
+#MD5 of the original StackName.
+stack_md5 = hash.str_checksum(stack_name)
+#MD5 of the original LambdaName
+function_md5 = hash.str_checksum(lambda_logical_id)
+
+#MD5 is used to avoid two having the same escaped name with different Lambda Functions
+#For example: Helloworld and HELLO-WORLD
+repo_logical_id =
+    lambda_logical_id [:52] + function_md5 [:8] + "Repo"
+    #52 + 8 + 4 = 64 max char
+    
+repo_name = 
+    escaped_stack_name + stack_md5[:8] + "/" + escaped_lambda_logical_id + function_md5[:8] + "repo"
+    #128        +         8      +  1  +     64      +        8      +      4 = 213 max char
+    
+companion_stack_name = 
+    stack_name[:104] + "-" + stack_md5[:8] + "-" + "CompanionStack"
+    #104 + 1 + 8 + 1 + 14 = 128 max char
+    
+repo_output_logical_id = 
+    lambda_logical_id[:52] + function_md5 [:8] + "Out"
+    #52 + 8 + 3 = 63 max char
+
+Exmaple:
+    Input:
+        Customer Stack Name: Hello-World-Stack
+        Function 1 Logical ID: TestFunction01
+        Function 2 Logical ID: AnotherTestFunction02
+    Output:
+        Companion Stack Name: Hello-World-Stack-925976eb-CompanionStack
+        Function 1 Repo Logical ID: TestFunction0150919004Repo
+        Function 1 Repo Name: helloworldstack925976eb/testfunction0150919004repo
+        Function 2 Repo Logical ID: AnotherTestFunction025c2cfd8cRepo
+        Function 2 Repo Name: helloworldstack925976eb/anothertestfunction025c2cfd8crepo
+```
+
+**Companion Stack Structure**
+```
+AWSTemplateFormatVersion : '2010-09-09'
+Transform: AWS::Serverless-2016-10-31
+Description: AWS SAM CLI Managed ECR Repo Stack
+Metadata:
+  SamCliInfo: 1.18.0
+  CompanionStackname:  Hello-World-Stack-925976eb-CompanionStack
+
+Resources:
+
+  TestFunction0150919004Repo:
+    Type: AWS::ECR::Repository
+    Properties:
+      RepositoryName: helloworldstack925976eb/testfunction0150919004repo
+      Tags:
+        - Key: ManagedStackSource
+          Value: AwsSamCli
+        - Key: AwsSamCliCompanionStack
+          Value: Hello-World-Stack-925976eb-CompanionStack
+
+      RepositoryPolicyText: 
+        Version: "2012-10-17"
+        Statement:
+          -
+            Sid: AllowLambdaSLR
+            Effect: Allow
+            Principal:
+              Service:
+                - "lambda.amazonaws.com"
+            Action:
+                - "ecr:GetDownloadUrlForLayer"
+                - "ecr:GetRepositoryPolicy"
+                - "ecr:BatchGetImage"
+
+  AnotherTestFunction025c2cfd8cRepo:
+    Type: AWS::ECR::Repository
+    Properties:
+      RepositoryName: helloworldstack925976eb/anothertestfunction025c2cfd8crepo
+      Tags:
+        - Key: ManagedStackSource
+          Value: AwsSamCli
+        - Key: AwsSamCliCompanionStack
+          Value: Hello-World-Stack-925976eb-CompanionStack
+
+      RepositoryPolicyText: 
+        Version: "2012-10-17"
+        Statement:
+          -
+            Sid: AllowLambdaSLR
+            Effect: Allow
+            Principal:
+              Service:
+                - "lambda.amazonaws.com"
+            Action:
+                - "ecr:GetDownloadUrlForLayer"
+                - "ecr:GetRepositoryPolicy"
+                - "ecr:BatchGetImage"
+
+Outputs:
+
+  None:
+    Value: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.${AWS::URLSuffix}/${TestFunction0150919004Repo}
+
+  None:
+    Value: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.${AWS::URLSuffix}/${AnotherTestFunction025c2cfd8cRepo}
+```
+
+Documentation Changes
+=====================
+* New option `--resolve-image-repos`. This option will auto create/delete repos without the needs of going through guided experience. 
+
+Open Issues
+============
+https://github.com/aws/aws-sam-cli/issues/2447
+
+Task Breakdown
+==============
+
+-   \[x\] Send a Pull Request with this design document
+-   \[ \] Build Companion Stack Manager
+-   \[ \] Change Deploy CLI
+-   \[ \] Unit tests
+-   \[ \] Functional Tests
+-   \[ \] Integration tests
+-   \[ \] Update documentation
