Skip to content

Commit 8994a87

Browse files
bgroffbgroff
committed
Update the documentation to reflect the new STS Assume Role auth.
1 parent 0870cd1 commit 8994a87

File tree

1 file changed

+125
-35
lines changed
  • docs/integrations/destinations

1 file changed

+125
-35
lines changed

‎docs/integrations/destinations/s3.md

+125-35
Original file line numberDiff line numberDiff line change
@@ -6,24 +6,35 @@ This page guides you through the process of setting up the S3 destination connec
66

77
List of required fields:
88

9-
- **Access Key ID**
10-
- **Secret Access Key**
119
- **S3 Bucket Name**
1210
- **S3 Bucket Path**
1311
- **S3 Bucket Region**
1412

13+
If you are using STS Assume Role, you must provide the following:
14+
15+
- **Role ARN**
16+
17+
Otherwise, if you are using AWS credentials you must provide the following:
18+
19+
- **Access Key ID**
20+
- **Secret Access Key**
21+
22+
If you are using an Instance Profile, you may omit the Access Key ID and Secret Access Key,
23+
as well as, the Role ARN.
24+
25+
Additionally the following prerequisites are required:
26+
1527
1. Allow connections from Airbyte server to your AWS S3/ Minio S3 cluster \(if they exist in
1628
separate VPCs\).
17-
2. An S3 bucket with credentials or an instance profile with read/write permissions configured for
29+
2. An S3 bucket with credentials, a Role ARN, or an instance profile with read/write permissions configured for
1830
the host (ec2, eks).
1931
3. [Enforce encryption of data in transit](https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html#transit)
2032

2133
## Setup guide
2234

2335
### Step 1: Set up S3
2436

25-
[Sign in](https://console.aws.amazon.com/iam/) to your AWS account. Use an existing or create new
26-
[Access Key ID and Secret Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#:~:text=IAM%20User%20Guide.-,Programmatic%20access,-You%20must%20provide).
37+
[Sign in](https://console.aws.amazon.com/iam/) to your AWS account.
2738

2839
Prepare S3 bucket that will be used as destination, see
2940
[this](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to create
@@ -34,6 +45,109 @@ to an unencrypted connection. Airbyte recommends all connections be configured t
3445
support for AWS's
3546
[shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/)
3647

48+
#### Create bucket a Policy
49+
1. Open the [IAM console](https://console.aws.amazon.com/iam/home#home).
50+
2. In the IAM dashboard, select **Policies**, then click **Create Policy**.
51+
3. Select the **JSON** tab, then paste the following JSON into the Policy editor (be sure to substitute in your bucket name):
52+
```json
53+
{
54+
"Version": "2012-10-17",
55+
"Statement": [
56+
{
57+
"Effect": "Allow",
58+
"Action": [
59+
"s3:PutObject",
60+
"s3:GetObject",
61+
"s3:DeleteObject",
62+
"s3:PutObjectAcl",
63+
"s3:ListBucket",
64+
"s3:ListBucketMultipartUploads",
65+
"s3:AbortMultipartUpload",
66+
"s3:GetBucketLocation"
67+
],
68+
"Resource": [
69+
"arn:aws:s3:::YOUR_BUCKET_NAME/*",
70+
"arn:aws:s3:::YOUR_BUCKET_NAME"
71+
]
72+
}
73+
]
74+
}
75+
```
76+
77+
:::note
78+
At this time, object-level permissions alone are not sufficient to successfully authenticate the connection. Please ensure you include the **bucket-level** permissions as provided in the example above.
79+
:::
80+
81+
4. Give your policy a descriptive name, then click **Create policy**.
82+
83+
#### Authentication Option 1: Using an IAM Role (Most secure)
84+
85+
<!-- env:cloud -->
86+
:::note
87+
This authentication method is currently in the testing phase. To enable it for your workspace, please contact our Support Team.
88+
:::
89+
<!-- /env:cloud -->
90+
91+
1. In the IAM dashboard, click **Roles**, then **Create role**. <!-- env:oss -->
92+
2. Choose the appropriate trust entity and attach the policy you created.
93+
3. Set up a trust relationship for the role. For example for **AWS account** trusted entity use default AWS account on your instance (it will be used to assume role). To use **External ID** set it to environment variables as `export AWS_ASSUME_ROLE_EXTERNAL_ID="{your-external-id}"`. Edit the trust relationship policy to reflect this:
94+
```
95+
{
96+
"Version": "2012-10-17",
97+
"Statement": [
98+
{
99+
"Effect": "Allow",
100+
"Principal": {
101+
"AWS": "arn:aws:iam::{your-aws-account-id}:user/{your-username}"
102+
},
103+
"Action": "sts:AssumeRole",
104+
"Condition": {
105+
"StringEquals": {
106+
"sts:ExternalId": "{your-external-id}"
107+
}
108+
}
109+
}
110+
]
111+
}
112+
```
113+
<!-- /env:oss -->
114+
<!-- env:cloud -->
115+
2. Choose the **AWS account** trusted entity type.
116+
3. Set up a trust relationship for the role. This allows the Airbyte instance's AWS account to assume this role. You will also need to specify an external ID, which is a secret key that the trusting service (Airbyte) and the trusted role (the role you're creating) both know. This ID is used to prevent the "confused deputy" problem. The External ID should be your Airbyte workspace ID, which can be found in the URL of your workspace page. Edit the trust relationship policy to include the external ID:
117+
```
118+
{
119+
"Version": "2012-10-17",
120+
"Statement": [
121+
{
122+
"Effect": "Allow",
123+
"Principal": {
124+
"AWS": "arn:aws:iam::094410056844:user/delegated_access_user"
125+
},
126+
"Action": "sts:AssumeRole",
127+
"Condition": {
128+
"StringEquals": {
129+
"sts:ExternalId": "{your-airbyte-workspace-id}"
130+
}
131+
}
132+
}
133+
]
134+
}
135+
```
136+
<!-- /env:cloud -->
137+
4. Complete the role creation and note the Role ARN.
138+
5. Select **Attach policies directly**, then find and check the box for your new policy. Click **Next**, then **Add permissions**.
139+
140+
141+
##### Authentication Option 2: Using an IAM User
142+
143+
Use an existing or create new
144+
[Access Key ID and Secret Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#:~:text=IAM%20User%20Guide.-,Programmatic%20access,-You%20must%20provide).
145+
146+
1. In the IAM dashboard, click **Users**. Select an existing IAM user or create a new one by clicking **Add users**.
147+
2. If you are using an _existing_ IAM user, click the **Add permissions** dropdown menu and select **Add permissions**. If you are creating a _new_ user, you will be taken to the Permissions screen after selecting a name.
148+
3. Select **Attach policies directly**, then find and check the box for your new policy. Click **Next**, then **Add permissions**.
149+
4. After successfully creating your user, select the **Security credentials** tab and click **Create access key**. You will be prompted to select a use case and add optional tags to your access key. Click **Create access key** to generate the keys.
150+
37151
### Step 2: Set up the S3 destination connector in Airbyte
38152

39153
<!-- env:cloud -->
@@ -55,6 +169,10 @@ support for AWS's
55169
to objects in the bucket.
56170
- **Secret Access Key**
57171
- Corresponding key to the above key id.
172+
- **Role ARN**
173+
- See
174+
[this](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html)
175+
on how to create a role.
58176
- **S3 Bucket Name**
59177
- See [this](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html)
60178
to create an S3 bucket.
@@ -95,7 +213,8 @@ support for AWS's
95213
will require
96214
[read and write permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html)
97215
to objects in the staging bucket. _ If the Access Key and Secret Access Key are not provided, the
98-
authentication will rely on the instanceprofile. _ **Secret Access Key** _ Corresponding key to
216+
authentication will rely either on the Role ARN using STS Assume Role or on the instanceprofile.
217+
5. _ **Secret Access Key** _ Corresponding key to
99218
the above key id. _ Make sure your S3 bucket is accessible from the machine running Airbyte. _
100219
This depends on your networking setup. _ You can check AWS S3 documentation with a tutorial on
101220
how to properly configure your S3's access
@@ -120,35 +239,6 @@ support for AWS's
120239

121240
5. Click `Set up destination`.
122241

123-
In order for everything to work correctly, it is also necessary that the user whose "S3 Key Id" and
124-
"S3 Access Key" are used have access to both the bucket and its contents. Minimum required Policies
125-
to use:
126-
127-
```json
128-
{
129-
"Version": "2012-10-17",
130-
"Statement": [
131-
{
132-
"Effect": "Allow",
133-
"Action": [
134-
"s3:PutObject",
135-
"s3:GetObject",
136-
"s3:DeleteObject",
137-
"s3:PutObjectAcl",
138-
"s3:ListBucket",
139-
"s3:ListBucketMultipartUploads",
140-
"s3:AbortMultipartUpload",
141-
"s3:GetBucketLocation"
142-
],
143-
"Resource": [
144-
"arn:aws:s3:::YOUR_BUCKET_NAME/*",
145-
"arn:aws:s3:::YOUR_BUCKET_NAME"
146-
]
147-
}
148-
]
149-
}
150-
```
151-
152242
The full path of the output data with the default S3 Path Format
153243
`${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_` is:
154244

0 commit comments

Comments
 (0)