Skip to content

Commit 0c6c50e

Browse files
viksrivatjfuss
viksrivat
authored and
jfuss
committed
design: CloudFormation ApiGateway support design (#1234)
1 parent 421d15e commit 0c6c50e

File tree

1 file changed

+354
-0
lines changed

1 file changed

+354
-0
lines changed

‎designs/cloud_formation_api_gateway_support.md

+354
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,354 @@
1+
# CloudFormation ApiGateway Support Design Doc
2+
3+
## The Problem
4+
5+
Customers use SAM CLI to run/test their applications by defining their resources in a SAM template. The raw CloudFormation ApiGateway resources are not currently supported to run and test locally in SAM CLI. The resource types include AWS::ApiGateway::* while ```sam local start-api``` only supports the AWS::Serverless::Api type. This prevents customers who have built/deployed services using the raw ApiGateway Resources or who have used tools to generate CloudFormation, like AWS CDK, from testing locally through SAM CLI. Specifically, Customers that generate their CloudFormation template using tools and have hand written CloudFormation templates have AWS::ApiGateway::* types preventing them from running locally.
6+
7+
Customers that are able to test and run locally can find errors early, reduce development time, etc. If customers are not able to test, the development cycles will be very long with a process similar to write code, deploy, deploy failed, investigate, fix and repeat.
8+
9+
## Who are the Customers?
10+
11+
* People who work with tools such as AWS CDK, Terraform, and others to generate their CloudFormation templates
12+
* People that have hand written CloudFormation templates with ApiGateway Resources
13+
14+
## Success criteria for the change
15+
16+
* Customers would be able to author SAM templates with Api Gateway Resources and test locally through start-api
17+
* Feature parity with what is currently supported with start-api on the AWS::Serverless::Api Resource
18+
* SAM CLI should support swagger and vanilla CloudFormation Api Gateway resources
19+
20+
Overall, SAM CLI should be able to seamlessly support local development and testing with the Api Gateway CloudFormation Resources.
21+
22+
## What will be changed?
23+
24+
When customers run ```sam local start-api``` with a template that uses raw CloudFormation AWS::ApiGateway::RestApi and AWS::ApiGateway::Stage resources, they will be able to interact and test their lambda functions as if they were using AWS::Serverless::Api.
25+
26+
## Out-of-Scope
27+
28+
Anything that SAM CLI doesn't currently support in SAM
29+
* ApiGateway Authorization such as resource policies, IAM roles/tags/policies, Lambda Authorizers, etc.
30+
* ApiGateway CORS support
31+
* Proper validation of the CloudFormation templates so that it does smart validation and not just yaml parsing.
32+
33+
## User Experience Walkthrough
34+
35+
There are two main types of users who are going to benefit from this change.
36+
37+
* Customers can use tools such as AWS CDK to generate a template. The customer can create their AWS CDK project with `cdk init app` and then generate their CloudFormation code using `cdk synth.`They can input their CloudFormation code to test it locally using the SAM CLI command.
38+
* Customers can author CloudFormation resources and test them locally by inputting their templates into sam local start-api.
39+
40+
For both cases, The code can be run locally if they have CodeUri's pointing to valid local paths.
41+
42+
Once the user has their CloudFormation code, they will be running `sam local start-api --template /path/to/template.yaml`
43+
44+
# Implementation
45+
46+
## **Design**
47+
48+
There are a few approaches to supporting the new CloudFormation ApiGateway types such as ApiGateway::RestApi.
49+
50+
### *Approach #1*: Parsing both CloudFormation and SAM Resources
51+
52+
Appending to the current Sam Api code and to have dual support of both the CloudFormation Template and SAM template
53+
54+
Pros:
55+
* This approach is something we do for lambda functions such as AWS::Lambda::Function and will provide consistency with our current implementation.
56+
57+
Cons:
58+
* Managing two different systems/templates require more work to resolve bugs. For example, supporting ApiGatewayV2, which supports web sockets with ApiGateway, the parsing of the template will need to be reimplemented in two places in order to support new functionality. One where it was defined using CloudFormation and the other where it was defined using the SAM template. This will slowly start to incur more and more technical debt as there is now more area to cover and duplication of work and resources. In the short term, it may be easier to implement, but in the long term there may be issues when dealing with support.
59+
* Another issue is that there may be escape hatches that are implemented in SAM causing additional effort to maintain differing parsing parts of the codebase.
60+
61+
### *Approach #2*: Process Once
62+
63+
Convert the SAM template into CloudFormation code and processes it once.
64+
65+
Pros:
66+
* This simplifies a lot of the issues with approach #1. This will make it much easier to add and extend the system such as adding ApiGatewayV2, web sockets. The feature would only need to be added once instead of duplicated effort.
67+
68+
Cons:
69+
* This will require more work in order to restructure the application such that the template is processed once after it processed to CloudFormation.
70+
* This could also produce bugs with issues if the SAM to CloudFormation transformation isn't correct in local testing. However, this may not matter as much since there is a direct translation of SAM resources to CloudFormation Resources and only a single point where the bug needs to be fixed.
71+
* Possible imperfections of the SAM to CloudFormation conversion. In the past, some version of the transformation caused incompatible changes with the SAM CLI. If the conversion fails, it could cause users that are currently running their code locally to break.
72+
* The SAM Translator also requires credentials, which would now cause users to login before running a command. Credentials are currently required in sam validate because of this. A local flavor of the translator needs to be created in order to avoid the process.
73+
74+
### *Approach #3*: Abstraction
75+
76+
Extending the current code to a CloudFormationApiProvider object overriding the CloudFormation processing in certain methods. This will be doing two passes through the code. SAM code can be processed first and the CloudFormation types second.
77+
78+
Pros:
79+
* This provides better abstraction, separating the CloudFormation code and the SamApi code. This can also be easily implemented.
80+
81+
Cons:
82+
* This falls through the same pitfalls as approach #1. The abstraction for the same types may also be inconsistent with the way we are currently processing CloudFormation resources such as AWS::Lambda::Function.
83+
84+
### Approach RECOMMENDATION
85+
86+
Although other parts of the project are using approach #1 currently, I recommend using **approach #2**. It seems to cause the least technical debt in the long run and reduces our duplication of code. It may require some work to port some of the code from AWS::Lambda::Function to follow a similar format, but it may be worth in the long term. Since the act of translating the SAM to CloudFormation may be imperfect. I plan to first implement approach #1 and gradually move it to approach #2 once all the items are parsed.
87+
88+
## **Design: Api Details**
89+
90+
### **AWS::ApiGateway::RestApi**
91+
92+
AWS::ApiGateway::RestApi will be one of the main resources defined in the CloudFormation template. It acts as the main definition for the Api. There are two approaches to consider for defining the RestApi template.
93+
94+
***Feature #1*: Swagger Method**
95+
The swagger method is very similar to the support in our current code base. Swagger has many advantageous as it can be exported, do validation, etc. Customers can also link to the other files containing their swagger documentation.
96+
97+
```yaml
98+
ServerlessRestApi:
99+
Type: 'AWS::ApiGateway::RestApi'
100+
Properties:
101+
Body:
102+
basePath : /pete
103+
paths:
104+
/hello:
105+
get:
106+
x-amazon-apigateway-integration:
107+
httpMethod: POST
108+
type: aws_proxy
109+
uri: !Sub >-
110+
arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorldFunction.Arn}/invocations
111+
responses: {}
112+
```
113+
114+
Swagger can also be inlined using the FN::Include Macro. This should also be supported while defining the CloudFormation template.
115+
****Feature* #2*: Using a combination of AWS::ApiGateway::Resource with ApiGateway::Methods.**
116+
117+
Although this approach is less common and more verbose, it is still used by some people while defining their resources. Tools
118+
such as aws-cdk currently generate Resource and Methods ApiGateway CloudFormation Resources in their yaml instead of swagger.
119+
120+
```yaml
121+
UsersResource:
122+
Type: AWS::ApiGateway::Resource
123+
Properties:
124+
RestApiId:!Ref RestApi
125+
ParentId:
126+
Fn::GetAtt:
127+
- RestApi
128+
- RootResourceId
129+
PathPart: user
130+
UsersGet:
131+
Type: AWS::ApiGateway::Method
132+
Properties:
133+
ResourceId: !Ref ApiGatewayResource
134+
RestApiId: !Ref ApiGatewayRestApi
135+
Integration:
136+
Type: AWS
137+
IntegrationHttpMethod: POST
138+
Uri:
139+
Fn::Join:
140+
- ''
141+
- - 'arn:aws:apigateway:'
142+
- Ref: AWS::Region
143+
- ":lambda:path/2015-03-31/functions/"
144+
- Fn::GetAtt:
145+
- Lambda
146+
- Arn
147+
- "/invocations"
148+
IntegrationResponses: []
149+
MethodResponses:
150+
- ResponseModels:
151+
application/json:
152+
Ref: UsersModel
153+
ResponseParameters:
154+
method.response.header.Link: true
155+
StatusCode: 200
156+
ApiGatewayMethod:
157+
Type: AWS::ApiGateway::Method
158+
Properties:
159+
HttpMethod: POST
160+
Integration:
161+
ConnectionType: INTERNET
162+
IntegrationResponses:
163+
- ResponseTemplates:
164+
application/json: "{\"message\": \"OK\"}"
165+
StatusCode: 200
166+
RequestTemplates:
167+
applicationjson: "{\"statusCode\": $input.json('$.statusCode'), \"message\": $input.json('$.message')}"
168+
169+
```
170+
171+
The SAM template requires people to use only a single RestApi to define all of their resources.
172+
173+
### **AWS::ApiGateway::Stages**
174+
175+
The stage allows for defining environments for different parts of the pipeline in CloudFormation. Since a single stage is attached to a deployment, the behavior of multiple deployment/stages locally is currently undefined.
176+
177+
Currently, the SAM template supports only one stage/one deployment and does not /<Stage>/<Serivce Name> as customers could want. One approach is to support additional flags in environments and stage names so they can test the apis locally in multiple environments. This would create a dict such as {“dev”: [models], “prod”: [models]}. However, this may be unnecessary.
178+
179+
The AWS::ApiGateway::Stage will initially support the schema:
180+
181+
```yaml
182+
Prod:
183+
Type: AWS::ApiGateway::Stage
184+
Properties:
185+
StageName: Prod
186+
Description: Prod Stage
187+
Variables:
188+
Stack: Prod
189+
RestApiId: RestApi`
190+
```
191+
192+
Since the stages can show up in any order as the code is iterating through the resources, There are two approaches to consider too quickly stitch the api and stages together.
193+
194+
*Approach #1:*
195+
Pick the last stage in the template that we find and use that on all the Apis
196+
Pros: Similar to the current approach
197+
Cons: The customer may not be able to view their environments in different stages.
198+
199+
*Approach: #2: *
200+
Create a dictionary of all the stages and display the url as localhost:3000/<Stage Name>/routeName
201+
Pros: This will allow people to view their code in different stages or environments.
202+
Cons: This will require refactoring parts of the SamApiProvider that handles the Api. Customers usually don't use different stages to test and it may not be worth the effort to test.
203+
204+
I recommend approach #1 as it follows the current standard and there is some data that very little people use the <Stage Name> when defining it.
205+
206+
**AWS::ApiGateway::Deployment **
207+
The deployment type defines which Apis objects should be available to the public. Normally, in the SAM template everything defined will have a deployment component, but the CloudFormation templates may have more bloat and information in them defining certain apis and deployments for Multi-stages.
208+
209+
One approach is to support multiple stages with multiple deployments and the code will filter out the stages that are not deployed or defined. One problem with this approach is that the RestApiId, a characteristic of the RestApi, for a deployment is only localhost and a single endpoint and the current schema doesn't allow deploying on multiple ports and multiple local domains. This will bring in unneeded complexity such as requiring the customer defining N valid ports and N valid domains.
210+
211+
Another approach, which I recommend, is to ignore the resource altogether as it always associated with a stage and to randomly pick a stage.
212+
213+
## **Updating the code**
214+
215+
First, the Apis in AWS::ApiGateway::RestApi will be parsed in the SAM Template in a similar way to the current AWS::Serverless:Api code. Stages and other Meta information in the Api will be parsed to support the CloudFormation template. The code will then be refactored to first translate the SAM templates inputted into the CloudFormation template using the SamTranslator. Once the Api code has been refactored, the function code can be refactored so that Serverless code is refactored out.
216+
217+
Some pseudo classes and function for the implementation
218+
219+
```python
220+
221+
class CloudFormationGatewayProvider(SamBaseProvider):
222+
# Same as serverless but update with ApiGateway Code
223+
_GATEWAY_REST_API = "AWS::ApiGateway::RestApi"
224+
_GATEWAY_STAGE_API = "AWS::ApiGateway::Stages"
225+
_GATEWAY_RESOURCE_API = "AWS::ApiGateway::Resource"
226+
_GATEWAY_METHODS_API = "AWS::ApiGateway::Method"
227+
228+
def _extrac_api_gateway_rest_api():
229+
for logical_id, resource in resources.items():
230+
resource_type = resource.get(SamApiProvider._TYPE)
231+
# call method based on matching type
232+
233+
# Detect Swagger or Method/Resource approach
234+
# Parse code similar to previous approach and update api dict
235+
# If the data is not swagger add the unfinished Api
236+
def _extract_gateway_rest_api():
237+
pass
238+
239+
# set the method_name for the Api from AWS::ApiGateway::Methods
240+
def _extract_gateway_methods():
241+
pass
242+
243+
# Set the stage for that api and its resources using AWS::ApiGateway::Stage
244+
def _extract_gateway_stage():
245+
pass
246+
247+
# The alterate method for defining resources. This will parse the Api and update the path
248+
def _extract_gateway_resource():
249+
pass
250+
251+
# Update the Api Model with the method name that corressponds to it.
252+
def _extract_gateway_method():
253+
pass
254+
```
255+
256+
While updating the structure and flow of the code around Apis, parts of the flow about how the Api and Route is currently implemented can be updated to make updating future milestones like cors. Some examples include abstracting stage states and variables that exist in every route can be abstracted.
257+
258+
## CLI Changes
259+
260+
*Explain the changes to command line interface, including adding new commands, modifying arguments etc*
261+
None
262+
263+
### Breaking Change
264+
265+
*Are there any breaking changes to CLI interface? Explain*
266+
267+
None
268+
269+
## `.samrc` Changes
270+
271+
*Explain the new configuration entries, if any, you want to add to .samrc*
272+
273+
None
274+
275+
## Security
276+
277+
*Tip: How does this change impact security? Answer the following questions to help answer this question better:*
278+
**What new dependencies (libraries/cli) does this change require?**
279+
None
280+
281+
**What other Docker container images are you using?**
282+
None
283+
284+
**Are you creating a new HTTP endpoint? If so explain how it will be created & used**
285+
This will be used for local development.
286+
287+
**Are you connecting to a remote API? If so explain how is this connection secured**
288+
No.
289+
290+
**Are you reading/writing to a temporary folder? If so, what is this used for and when do you clean up?**
291+
The setup is not written for local development.
292+
293+
# What is your Testing Plan (QA)?
294+
295+
## Goal
296+
297+
## Pre-Requisites
298+
299+
## Test Scenarios/Cases
300+
301+
* Basic Templates and unit tests to check that CloudFormation templates are covering the data
302+
* Go through https://github.com/awslabs/serverless-application-model with tests/sample inputs and outputs.
303+
* Create an example AWS Lambda Project with AWS CDK and test the generated CloudFormation code
304+
305+
## Expected Results
306+
307+
## Pass/Fail
308+
309+
# Documentation Changes
310+
311+
The main documentation change will be telling users that they are now allowed to pass SAM templates with ApiGateway resources.
312+
313+
# Open Issues
314+
315+
No open issues
316+
317+
# Task Breakdown
318+
319+
### Milestones:
320+
321+
**Milestone 1 Goal: Support Swagger and Stage Name with CloudFormation ApiGateway Resources**
322+
Milestone #1A
323+
324+
* Swagger Definition for AWS::ApiGateway works for RestApi
325+
326+
Milestone #1B:
327+
328+
* Support Stage Names/Variables, binary_media_types with AWS::ApiGatway
329+
330+
Milestone #1C:
331+
332+
* Support Non-inline swagger
333+
334+
Milestone #1D:
335+
336+
* Refactor Code to convert SAM into CloudFormationResource
337+
338+
**Milestone #2: Support Resource and Methods with CloudFormation ApiGateway Resources**
339+
340+
* Add Support for Resource and Methods parsing
341+
342+
### Time Breakdown
343+
344+
Milestone 1A ~ 1 Week
345+
346+
Milestone 1B ~ 1 Week
347+
348+
Milestone 1C ~ 2 day
349+
350+
Milestone 1D ~ 1 Week
351+
352+
Milestone 2 ~ 1 Week
353+
354+

0 commit comments

Comments
 (0)