Skip to content

Commit 99865d7

Browse files
rwasksajarin
authored andcommitted
Update on-kubernetes-via-helm.md - Add GCS Logging steps (airbytehq#35455)
Co-authored-by: Sajarin <[email protected]>
1 parent 4c1aa4d commit 99865d7

File tree

1 file changed

+75
-5
lines changed

1 file changed

+75
-5
lines changed

docs/deploying-airbyte/on-kubernetes-via-helm.md

+75-5
Original file line numberDiff line numberDiff line change
@@ -167,11 +167,10 @@ Before upgrading the chart update values.yaml as stated above and then run:
167167
- Perform upgrade of chart by running `helm upgrade %release_name% airbyte/airbyte --set auth.rootPassword=$ROOT_PASSWORD`
168168
- If you get an error about setting the auth.rootPassword, then you forgot to update the `values.yaml` file
169169

170-
### External Logs
170+
### External Logs with S3
171171

172172
::info
173-
This was tested using [Airbyte Helm Chart Version 0.50.13](https://artifacthub.io/packages/helm/airbyte/airbyte/0.50.13) and S3 logs only.
174-
Previous or newer version can change how to setup the external logs.
173+
S3 logging was tested on [Airbyte Helm Chart Version 0.50.13](https://artifacthub.io/packages/helm/airbyte/airbyte/0.50.13)
175174
:::
176175

177176
Create a file called `airbyte-logs-secrets.yaml` to store the AWS Keys and other informations:
@@ -219,7 +218,7 @@ global:
219218
tolerations: []
220219
affinity: {}
221220
```
222-
You can try to use `GCS` or `External Minio` but both weren't tested yet. Feel free to run tests and update the documentation.
221+
GCS Logging information is below but you can try to use `External Minio` as well but it was not tested yet. Feel free to run tests and update the documentation.
223222

224223
Add extra env variables to the following blocks:
225224
```yaml
@@ -297,14 +296,85 @@ server:
297296
Than run:
298297
`helm upgrade --install %RELEASE_NAME% airbyte/airbyte -n <NAMESPACE> --values /path/to/values.yaml --version 0.50.13`
299298

299+
### External Logs with GCS
300+
301+
302+
:::Info
303+
GCS Logging is similar to the approach taken for S3 above, with a few small differences
304+
GCS logging was tested on [Airbyte Helm Chart Version 0.53.178](https://artifacthub.io/packages/helm/airbyte/airbyte/0.53.178)
305+
:::
306+
307+
#### Create Google Cloud Storage Bucket
308+
309+
1. **Access Google Cloud Console**: Go to the Google Cloud Console and select or create a project where you want to create the bucket.
310+
2. **Open Cloud Storage**: Navigate to "Storage" > "Browser" in the left-side menu.
311+
3. **Create Bucket**: Click on "Create bucket". Give your bucket a unique name, select a region for the bucket, and configure other settings such as storage class and access control according to your requirements. Finally, click "Create".
312+
313+
#### Create Google Cloud Service Account
314+
315+
1. **Open IAM & Admin**: In the Cloud Console, navigate to "IAM & Admin" > "Service Accounts".
316+
2. **Create Service Account**: Click "Create Service Account", enter a name, description, and then click "Create".
317+
3. **Grant Permissions**: Assign the role of "Storage Object Admin" to the service account by selecting it from the role list.
318+
4. **Create Key**: After creating the service account, click on it, go to the "Keys" tab, and then click "Add Key" > "Create new key". Choose JSON as the key type and click "Create". The key file will be downloaded automatically to your computer.
319+
320+
#### Create a Kubernetes Secret
321+
322+
- Use the **`kubectl create secret`** command to create a Kubernetes secret from the JSON key file. Replace **`<secret-name>`** with the desired name for your secret, **`<path-to-json-key-file>`** with the path to the JSON key file you downloaded, and **`<namespace>`** with the namespace where your deployment will be running.
323+
324+
```kubectl create secret generic <mysecret> --from-file=gcp.json=</location/to/secret.json> --namespace=<namespace>```
325+
326+
#### Create an extra Volume where the GCSFS secret will be added in the values.yaml inside of the worker section
327+
```
328+
worker:
329+
extraVolumes:
330+
- name: gcsfs-creds
331+
secret:
332+
secretName: <secret name>
333+
extraVolumeMounts:
334+
- name: gcsfs-creds
335+
mountPath: "/etc/secrets"
336+
readOnly: true
337+
```
338+
339+
#### Update the values.yaml with the GCS Logging Information below
340+
Update the following Environment Variables in the global section:
341+
```
342+
global:
343+
state:
344+
storage:
345+
type: "GCS"
346+
347+
logs:
348+
storage:
349+
type: "GCS"
350+
gcs:
351+
bucket: "<bucket name>"
352+
credentials: "/etc/secrets/gcp.json"
353+
354+
extraEnv:
355+
- name: STATE_STORAGE_GCS_BUCKET_NAME
356+
value: <bucket name>
357+
- name: STATE_STORAGE_GCS_APPLICATION_CREDENTIALS
358+
value: /etc/secrets/gcp.json
359+
- name: CONTAINER_ORCHESTRATOR_SECRET_NAME
360+
value: <name of secret>
361+
- name: CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH
362+
value: /etc/secrets/
363+
```
364+
365+
Than run:
366+
`helm upgrade --install %RELEASE_NAME% airbyte/airbyte -n <NAMESPACE> --values /path/to/values.yaml --version 0.53.178`
367+
300368
### External Airbyte Database
301369
302370
303-
::info
371+
372+
:::info
304373
This was tested using [Airbyte Helm Chart Version 0.50.13](https://artifacthub.io/packages/helm/airbyte/airbyte/0.50.13).
305374
Previous or newer version can change how the external database can be configured.
306375
:::
307376
377+
308378
The Airbyte Database only works with Postgres 13.
309379
Make sure the database is accessible inside the cluster using `busy-box` service using `telnet` or `ping` command.
310380

0 commit comments

Comments
 (0)