Skip to content

Commit 036b41b

Browse files
awood45sriram-mv
awood45
authored and
sriram-mv
committed
Managed S3 Bucket via optional bootstrap command (#1526)
* WIP: Managed S3 Stack * Managed S3 Bucket Command With Tests Missing: Integration with config files. We may also want to move some of the echo commands to debug logging. * Setup Design Doc * Setup Design Document * Rename to and remove CLI interface * Fix lint errors * Adding metadata to stack * fixing black formatting
1 parent 0451b59 commit 036b41b

File tree

7 files changed

+374
-0
lines changed

7 files changed

+374
-0
lines changed

‎designs/sam_setup_cmd.md

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# `sam setup` command
2+
3+
As a part of packaging Lambda functions for deployment to AWS, users of the AWS SAM CLI currently need to provide an S3 bucket to store their code artifacts in. This creates a number of extra setup steps today, from users needing to go and set up an S3 bucket, to needing to track which bucket is appropriate for a given region (S3 bucket region must match CloudFormation deployment region). This project aims to simplify this experience.
4+
5+
## Goals
6+
7+
1. AWS SAM CLI users should be able to set up an S3 bucket for their SAM project entirely through the AWS SAM CLI.
8+
2. The AWS SAM CLI, in setting up such a bucket, should choose an appropriate region and populate the users’s SAM CLI config file in their project.
9+
3. A user doing the interactive deploy experience should be able to be completely separated from the S3 bucket used for source code storage, if the user does not wish to directly configure their source bucket.
10+
11+
## Design
12+
13+
We propose creating a new SAM CLI command, sam setup for this process. The underlying functionality would also be accessible to other commands, such as package itself.
14+
15+
The `sam setup` command would have the following parameters:
16+
17+
* `--region` This parameter is **CONDITIONALLY REQUIRED**, because the primary goal of this command is to ensure that the user’s region has an S3 bucket set up. We will also accept the `AWS_REGION` environment variable, or the default region in a user’s profile. In short, a region must be provided in some way, or we will fail.
18+
* `--profile` This is associated with a user’s AWS profile, and defaults to `"default"` if not provided. It will be used for sourcing credentials for CloudFormation commands used when setting up the bucket, and for doing S3 ListBucket calls to see if a suitable bucket already exists.
19+
20+
## Challenges
21+
22+
Both S3 buckets and CloudFormation stacks do not have sufficiently efficient ways to search by tags. Simply put, there’s likely to be some computational inefficiency as up to hundreds of API calls might be required to identify an existing bucket that was created to be a source bucket. This means that to avoid severe performance issues, we need to make compromises. Proposed:
23+
24+
* The default managed bucket uses a fixed stack name per region, such as “aws-sam-cli-managed-source-bucket”. If the user for some reason has a stack with that name, then we cannot support a managed bucket for them.
25+
* Alternatively, when doing sam setup, the user providing a bucket name would mean that we just check for it to exist and if it does and is in the correct region, populate the config file.

‎samcli/cli/command.py

+2
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@
2121
"samcli.commands.deploy",
2222
"samcli.commands.logs",
2323
"samcli.commands.publish",
24+
# We intentionally do not expose the `bootstrap` command for now. We might open it up later
25+
# "samcli.commands.bootstrap",
2426
]
2527

2628

‎samcli/commands/bootstrap/__init__.py

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""
2+
`sam setup` command
3+
"""
4+
5+
# Expose the cli object here
6+
from .command import cli # noqa

‎samcli/commands/bootstrap/command.py

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
"""
2+
CLI command for "bootstrap", which sets up a SAM development environment
3+
"""
4+
import click
5+
6+
from samcli.cli.main import pass_context, common_options, aws_creds_options
7+
from samcli.lib.telemetry.metrics import track_command
8+
from samcli.lib.bootstrap import bootstrap
9+
10+
SHORT_HELP = "Set up development environment for AWS SAM applications."
11+
12+
HELP_TEXT = """
13+
Sets up a development environment for AWS SAM applications.
14+
15+
Currently this creates, if one does not exist, a managed S3 bucket for your account in your working AWS region.
16+
"""
17+
18+
19+
@click.command("bootstrap", short_help=SHORT_HELP, help=HELP_TEXT, context_settings=dict(max_content_width=120))
20+
@common_options
21+
@aws_creds_options
22+
@pass_context
23+
@track_command
24+
def cli(ctx):
25+
do_cli(ctx.region, ctx.profile) # pragma: no cover
26+
27+
28+
def do_cli(region, profile):
29+
bucket_name = bootstrap.manage_stack(profile=profile, region=region)
30+
click.echo("Source Bucket: " + bucket_name)

‎samcli/lib/bootstrap/__init__.py

Whitespace-only changes.

‎samcli/lib/bootstrap/bootstrap.py

+121
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
"""
2+
Bootstrap's user's development environment by creating cloud resources required by SAM CLI
3+
"""
4+
5+
import json
6+
import logging
7+
import boto3
8+
9+
from botocore.config import Config
10+
from botocore.exceptions import ClientError
11+
12+
from samcli import __version__
13+
from samcli.cli.global_config import GlobalConfig
14+
from samcli.commands.exceptions import UserException
15+
16+
17+
LOG = logging.getLogger(__name__)
18+
SAM_CLI_STACK_NAME = "aws-sam-cli-managed-stack"
19+
20+
21+
def manage_stack(profile, region):
22+
session = boto3.Session(profile_name=profile if profile else None)
23+
cloudformation_client = session.client("cloudformation", config=Config(region_name=region if region else None))
24+
25+
return _create_or_get_stack(cloudformation_client)
26+
27+
28+
def _create_or_get_stack(cloudformation_client):
29+
stack = None
30+
try:
31+
ds_resp = cloudformation_client.describe_stacks(StackName=SAM_CLI_STACK_NAME)
32+
stacks = ds_resp["Stacks"]
33+
stack = stacks[0]
34+
LOG.info("Found managed SAM CLI stack.")
35+
except ClientError:
36+
LOG.info("Managed SAM CLI stack not found, creating.")
37+
stack = _create_stack(cloudformation_client) # exceptions are not captured from subcommands
38+
# Sanity check for non-none stack? Sanity check for tag?
39+
tags = stack["Tags"]
40+
try:
41+
sam_cli_tag = next(t for t in tags if t["Key"] == "ManagedStackSource")
42+
if not sam_cli_tag["Value"] == "AwsSamCli":
43+
msg = (
44+
"Stack "
45+
+ SAM_CLI_STACK_NAME
46+
+ " ManagedStackSource tag shows "
47+
+ sam_cli_tag["Value"]
48+
+ " which does not match the AWS SAM CLI generated tag value of AwsSamCli. "
49+
"Failing as the stack was likely not created by the AWS SAM CLI."
50+
)
51+
raise UserException(msg)
52+
except StopIteration:
53+
msg = (
54+
"Stack " + SAM_CLI_STACK_NAME + " exists, but the ManagedStackSource tag is missing. "
55+
"Failing as the stack was likely not created by the AWS SAM CLI."
56+
)
57+
raise UserException(msg)
58+
outputs = stack["Outputs"]
59+
try:
60+
bucket_name = next(o for o in outputs if o["OutputKey"] == "SourceBucket")["OutputValue"]
61+
except StopIteration:
62+
msg = (
63+
"Stack " + SAM_CLI_STACK_NAME + " exists, but is missing the managed source bucket key. "
64+
"Failing as this stack was likely not created by the AWS SAM CLI."
65+
)
66+
raise UserException(msg)
67+
# This bucket name is what we would write to a config file
68+
return bucket_name
69+
70+
71+
def _create_stack(cloudformation_client):
72+
change_set_name = "InitialCreation"
73+
change_set_resp = cloudformation_client.create_change_set(
74+
StackName=SAM_CLI_STACK_NAME,
75+
TemplateBody=_get_stack_template(),
76+
Tags=[{"Key": "ManagedStackSource", "Value": "AwsSamCli"}],
77+
ChangeSetType="CREATE",
78+
ChangeSetName=change_set_name, # this must be unique for the stack, but we only create so that's fine
79+
)
80+
stack_id = change_set_resp["StackId"]
81+
LOG.info("Waiting for managed stack change set to create.")
82+
change_waiter = cloudformation_client.get_waiter("change_set_create_complete")
83+
change_waiter.wait(
84+
ChangeSetName=change_set_name, StackName=SAM_CLI_STACK_NAME, WaiterConfig={"Delay": 15, "MaxAttempts": 60}
85+
)
86+
cloudformation_client.execute_change_set(ChangeSetName=change_set_name, StackName=SAM_CLI_STACK_NAME)
87+
LOG.info("Waiting for managed stack to be created.")
88+
stack_waiter = cloudformation_client.get_waiter("stack_create_complete")
89+
stack_waiter.wait(StackName=stack_id, WaiterConfig={"Delay": 15, "MaxAttempts": 60})
90+
LOG.info("Managed SAM CLI stack creation complete.")
91+
ds_resp = cloudformation_client.describe_stacks(StackName=SAM_CLI_STACK_NAME)
92+
stacks = ds_resp["Stacks"]
93+
return stacks[0]
94+
95+
96+
def _get_stack_template():
97+
gc = GlobalConfig()
98+
info = {"version": __version__, "installationId": gc.installation_id}
99+
100+
template = """
101+
AWSTemplateFormatVersion : '2010-09-09'
102+
Transform: AWS::Serverless-2016-10-31
103+
Description: Managed Stack for AWS SAM CLI
104+
105+
Metadata:
106+
SamCliInfo: {info}
107+
108+
Resources:
109+
SamCliSourceBucket:
110+
Type: AWS::S3::Bucket
111+
Properties:
112+
Tags:
113+
- Key: ManagedStackSource
114+
Value: AwsSamCli
115+
116+
Outputs:
117+
SourceBucket:
118+
Value: !Ref SamCliSourceBucket
119+
"""
120+
121+
return template.format(info=json.dumps(info))

‎tests/unit/lib/bootstrap/test_bootstrap.py

+190
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
from unittest import TestCase
2+
3+
import botocore.session
4+
5+
from botocore.exceptions import ClientError
6+
from botocore.stub import Stubber
7+
8+
from samcli.commands.exceptions import UserException
9+
from samcli.lib.bootstrap.bootstrap import _create_or_get_stack, _get_stack_template, SAM_CLI_STACK_NAME
10+
11+
12+
class TestBootstrapManagedStack(TestCase):
13+
def _stubbed_cf_client(self):
14+
cf = botocore.session.get_session().create_client("cloudformation")
15+
return [cf, Stubber(cf)]
16+
17+
def test_new_stack(self):
18+
stub_cf, stubber = self._stubbed_cf_client()
19+
# first describe_stacks call will fail
20+
ds_params = {"StackName": SAM_CLI_STACK_NAME}
21+
stubber.add_client_error("describe_stacks", service_error_code="ClientError", expected_params=ds_params)
22+
# creating change set
23+
ccs_params = {
24+
"StackName": SAM_CLI_STACK_NAME,
25+
"TemplateBody": _get_stack_template(),
26+
"Tags": [{"Key": "ManagedStackSource", "Value": "AwsSamCli"}],
27+
"ChangeSetType": "CREATE",
28+
"ChangeSetName": "InitialCreation",
29+
}
30+
ccs_resp = {"Id": "id", "StackId": "aws-sam-cli-managed-stack"}
31+
stubber.add_response("create_change_set", ccs_resp, ccs_params)
32+
# describe change set creation status for waiter
33+
dcs_params = {"ChangeSetName": "InitialCreation", "StackName": SAM_CLI_STACK_NAME}
34+
dcs_resp = {"Status": "CREATE_COMPLETE"}
35+
stubber.add_response("describe_change_set", dcs_resp, dcs_params)
36+
# executing change set
37+
ecs_params = {"ChangeSetName": "InitialCreation", "StackName": SAM_CLI_STACK_NAME}
38+
ecs_resp = {}
39+
stubber.add_response("execute_change_set", ecs_resp, ecs_params)
40+
# two describe_stacks calls will succeed - one for waiter, one direct
41+
post_create_ds_resp = {
42+
"Stacks": [
43+
{
44+
"StackName": SAM_CLI_STACK_NAME,
45+
"CreationTime": "2019-11-13",
46+
"StackStatus": "CREATE_COMPLETE",
47+
"Tags": [{"Key": "ManagedStackSource", "Value": "AwsSamCli"}],
48+
"Outputs": [{"OutputKey": "SourceBucket", "OutputValue": "generated-src-bucket"}],
49+
}
50+
]
51+
}
52+
stubber.add_response("describe_stacks", post_create_ds_resp, ds_params)
53+
stubber.add_response("describe_stacks", post_create_ds_resp, ds_params)
54+
stubber.activate()
55+
_create_or_get_stack(stub_cf)
56+
stubber.assert_no_pending_responses()
57+
stubber.deactivate()
58+
59+
def test_stack_exists(self):
60+
stub_cf, stubber = self._stubbed_cf_client()
61+
ds_resp = {
62+
"Stacks": [
63+
{
64+
"StackName": SAM_CLI_STACK_NAME,
65+
"CreationTime": "2019-11-13",
66+
"StackStatus": "CREATE_COMPLETE",
67+
"Tags": [{"Key": "ManagedStackSource", "Value": "AwsSamCli"}],
68+
"Outputs": [{"OutputKey": "SourceBucket", "OutputValue": "generated-src-bucket"}],
69+
}
70+
]
71+
}
72+
ds_params = {"StackName": SAM_CLI_STACK_NAME}
73+
stubber.add_response("describe_stacks", ds_resp, ds_params)
74+
stubber.activate()
75+
_create_or_get_stack(stub_cf)
76+
stubber.assert_no_pending_responses()
77+
stubber.deactivate()
78+
79+
def test_stack_missing_bucket(self):
80+
stub_cf, stubber = self._stubbed_cf_client()
81+
ds_resp = {
82+
"Stacks": [
83+
{
84+
"StackName": SAM_CLI_STACK_NAME,
85+
"CreationTime": "2019-11-13",
86+
"StackStatus": "CREATE_COMPLETE",
87+
"Tags": [{"Key": "ManagedStackSource", "Value": "AwsSamCli"}],
88+
"Outputs": [],
89+
}
90+
]
91+
}
92+
ds_params = {"StackName": SAM_CLI_STACK_NAME}
93+
stubber.add_response("describe_stacks", ds_resp, ds_params)
94+
stubber.activate()
95+
with self.assertRaises(UserException):
96+
_create_or_get_stack(stub_cf)
97+
stubber.assert_no_pending_responses()
98+
stubber.deactivate()
99+
100+
def test_stack_missing_tag(self):
101+
stub_cf, stubber = self._stubbed_cf_client()
102+
ds_resp = {
103+
"Stacks": [
104+
{
105+
"StackName": SAM_CLI_STACK_NAME,
106+
"CreationTime": "2019-11-13",
107+
"StackStatus": "CREATE_COMPLETE",
108+
"Tags": [],
109+
"Outputs": [{"OutputKey": "SourceBucket", "OutputValue": "generated-src-bucket"}],
110+
}
111+
]
112+
}
113+
ds_params = {"StackName": SAM_CLI_STACK_NAME}
114+
stubber.add_response("describe_stacks", ds_resp, ds_params)
115+
stubber.activate()
116+
with self.assertRaises(UserException):
117+
_create_or_get_stack(stub_cf)
118+
stubber.assert_no_pending_responses()
119+
stubber.deactivate()
120+
121+
def test_stack_wrong_tag(self):
122+
stub_cf, stubber = self._stubbed_cf_client()
123+
ds_resp = {
124+
"Stacks": [
125+
{
126+
"StackName": SAM_CLI_STACK_NAME,
127+
"CreationTime": "2019-11-13",
128+
"StackStatus": "CREATE_COMPLETE",
129+
"Tags": [{"Key": "ManagedStackSource", "Value": "WHY WOULD YOU EVEN DO THIS"}],
130+
"Outputs": [{"OutputKey": "SourceBucket", "OutputValue": "generated-src-bucket"}],
131+
}
132+
]
133+
}
134+
ds_params = {"StackName": SAM_CLI_STACK_NAME}
135+
stubber.add_response("describe_stacks", ds_resp, ds_params)
136+
stubber.activate()
137+
with self.assertRaises(UserException):
138+
_create_or_get_stack(stub_cf)
139+
stubber.assert_no_pending_responses()
140+
stubber.deactivate()
141+
142+
def test_change_set_creation_fails(self):
143+
stub_cf, stubber = self._stubbed_cf_client()
144+
# first describe_stacks call will fail
145+
ds_params = {"StackName": SAM_CLI_STACK_NAME}
146+
stubber.add_client_error("describe_stacks", service_error_code="ClientError", expected_params=ds_params)
147+
# creating change set - fails
148+
ccs_params = {
149+
"StackName": SAM_CLI_STACK_NAME,
150+
"TemplateBody": _get_stack_template(),
151+
"Tags": [{"Key": "ManagedStackSource", "Value": "AwsSamCli"}],
152+
"ChangeSetType": "CREATE",
153+
"ChangeSetName": "InitialCreation",
154+
}
155+
stubber.add_client_error("create_change_set", service_error_code="ClientError", expected_params=ccs_params)
156+
stubber.activate()
157+
with self.assertRaises(ClientError):
158+
_create_or_get_stack(stub_cf)
159+
stubber.assert_no_pending_responses()
160+
stubber.deactivate()
161+
162+
def test_change_set_execution_fails(self):
163+
stub_cf, stubber = self._stubbed_cf_client()
164+
# first describe_stacks call will fail
165+
ds_params = {"StackName": SAM_CLI_STACK_NAME}
166+
stubber.add_client_error("describe_stacks", service_error_code="ClientError", expected_params=ds_params)
167+
# creating change set
168+
ccs_params = {
169+
"StackName": SAM_CLI_STACK_NAME,
170+
"TemplateBody": _get_stack_template(),
171+
"Tags": [{"Key": "ManagedStackSource", "Value": "AwsSamCli"}],
172+
"ChangeSetType": "CREATE",
173+
"ChangeSetName": "InitialCreation",
174+
}
175+
ccs_resp = {"Id": "id", "StackId": "aws-sam-cli-managed-stack"}
176+
stubber.add_response("create_change_set", ccs_resp, ccs_params)
177+
# describe change set creation status for waiter
178+
dcs_params = {"ChangeSetName": "InitialCreation", "StackName": SAM_CLI_STACK_NAME}
179+
dcs_resp = {"Status": "CREATE_COMPLETE"}
180+
stubber.add_response("describe_change_set", dcs_resp, dcs_params)
181+
# executing change set - fails
182+
ecs_params = {"ChangeSetName": "InitialCreation", "StackName": SAM_CLI_STACK_NAME}
183+
stubber.add_client_error(
184+
"execute_change_set", service_error_code="InsufficientCapabilities", expected_params=ecs_params
185+
)
186+
stubber.activate()
187+
with self.assertRaises(ClientError):
188+
_create_or_get_stack(stub_cf)
189+
stubber.assert_no_pending_responses()
190+
stubber.deactivate()

0 commit comments

Comments
 (0)