🎉 Create a Helm Chart For Airbyte #5891
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| name: Helm | ||
| on: | ||
| push: | ||
| paths: | ||
| - '.github/workflows/helm.yaml' | ||
| - 'charts/**' | ||
| pull_request: | ||
| paths: | ||
| - '.github/workflows/helm.yaml' | ||
| - 'charts/**' | ||
| jobs: | ||
| lint: | ||
| name: Lint | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Setup Kubectl | ||
| uses: azure/setup-kubectl@v1 | ||
| - name: Setup Helm | ||
| uses: azure/setup-helm@v1 | ||
| with: | ||
| version: '3.6.3' | ||
| - name: Lint Chart | ||
| working-directory: ./charts/airbyte | ||
| run: ./ci.sh lint | ||
|
|
||
| generate-docs: | ||
| name: Generate Docs Parameters | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Checkout bitnami-labs/readme-generator-for-helm | ||
| uses: actions/checkout@v2 | ||
| with: | ||
| repository: 'bitnami-labs/readme-generator-for-helm' | ||
| ref: '55cab5dd2191c4ffa7245cfefa428d4d9bb12730' | ||
| path: readme-generator-for-helm | ||
| - name: Install readme-generator-for-helm dependencies | ||
| working-directory: readme-generator-for-helm | ||
| run: npm install -g | ||
| - name: Test can update README with generated parameters | ||
| working-directory: charts/airbyte | ||
| run: ./ci.sh check-docs-updated | ||
|
|
||
| install: | ||
|
There was a problem hiding this comment. @davinchia We'll want to run acceptance tests at the end here and rename to test. |
||
| name: Install | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Setup Kubectl | ||
| uses: azure/setup-kubectl@v1 | ||
| - name: Setup Helm | ||
| uses: azure/setup-helm@v1 | ||
| with: | ||
| version: '3.6.3' | ||
| - name: Setup Kind Cluster | ||
|
There was a problem hiding this comment. @davinchia we'll need to test how this works for the Github AMIs + set up running this step on external runners. Not sure what configuration this will need vs There was a problem hiding this comment. I ran this helm action in my fork and can try to move to |
||
| uses: helm/[email protected] | ||
| with: | ||
| version: "v0.11.1" | ||
| image: "kindest/node:v1.21.1" | ||
| - name: Install airbyte chart | ||
| working-directory: ./charts/airbyte | ||
| run: ./ci.sh install | ||
| - if: always() | ||
| name: Print diagnostics | ||
| working-directory: ./charts/airbyte | ||
| run: ./ci.sh diagnostics | ||
| - if: success() | ||
| name: Test airbyte chart | ||
| working-directory: ./charts/airbyte | ||
| run: ./ci.sh test | ||
|
There was a problem hiding this comment. @jonstacks curious, what is this testing today? There was a problem hiding this comment. @davinchia, This job in the workflow:
You can find the an example and output of the action here(https://github.com/jonstacks/airbyte/runs/3613634997) |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # Charts are downloaded at install time with `helm dep build`. | ||
| charts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Patterns to ignore when building packages. | ||
| # This supports shell glob matching, relative path matching, and | ||
| # negation (prefixed with !). Only one pattern per line. | ||
| .DS_Store | ||
| # Common VCS dirs | ||
| .git/ | ||
| .gitignore | ||
| .bzr/ | ||
| .bzrignore | ||
| .hg/ | ||
| .hgignore | ||
| .svn/ | ||
| # Common backup files | ||
| *.swp | ||
| *.bak | ||
| *.tmp | ||
| *.orig | ||
| *~ | ||
| # Various IDEs | ||
| .project | ||
| .idea/ | ||
| *.tmproj | ||
| .vscode/ | ||
|
|
||
| ci.sh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| dependencies: | ||
| - name: common | ||
| repository: https://charts.bitnami.com/bitnami | ||
| version: 1.8.0 | ||
| - name: postgresql | ||
| repository: https://charts.bitnami.com/bitnami | ||
| version: 10.9.4 | ||
| - name: minio | ||
| repository: https://charts.bitnami.com/bitnami | ||
| version: 7.2.0 | ||
| digest: sha256:6b5428c4bffa53a16e1015635d712f28b1c1991eea4d048928e4fbb88974cf4f | ||
| generated: "2021-08-31T22:58:47.835303672-05:00" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| apiVersion: v2 | ||
| name: airbyte | ||
| description: Helm chart to deploy airbyte | ||
|
|
||
| # A chart can be either an 'application' or a 'library' chart. | ||
| # | ||
| # Application charts are a collection of templates that can be packaged into versioned archives | ||
| # to be deployed. | ||
| # | ||
| # Library charts provide useful utilities or functions for the chart developer. They're included as | ||
| # a dependency of application charts to inject those utilities and functions into the rendering | ||
| # pipeline. Library charts do not define any templates and therefore cannot be deployed. | ||
| type: application | ||
|
|
||
| # This is the chart version. This version number should be incremented each time you make changes | ||
| # to the chart and its templates, including the app version. | ||
| # Versions are expected to follow Semantic Versioning (https://semver.org/) | ||
| version: 0.1.0 | ||
|
There was a problem hiding this comment. @davinchia any opinions on how we should handle chart versioning? Presumably we want to update this both manually and when we bump There was a problem hiding this comment. yeah I wonder if we can configure bumpversion to do this for us as well |
||
|
|
||
| # This is the version number of the application being deployed. This version number should be | ||
| # incremented each time you make changes to the application. Versions are not expected to | ||
| # follow Semantic Versioning. They should reflect the version the application is using. | ||
| # It is recommended to use it with quotes. | ||
| appVersion: "0.29.13-alpha" | ||
|
There was a problem hiding this comment. We need to add this file to |
||
|
|
||
| dependencies: | ||
| - name: common | ||
| repository: https://charts.bitnami.com/bitnami | ||
| tags: | ||
| - bitnami-common | ||
| version: 1.x.x | ||
| - condition: postgresql.enabled | ||
| name: postgresql | ||
| version: 10.x.x | ||
| repository: https://charts.bitnami.com/bitnami | ||
| - condition: minio.enabled | ||
| name: minio | ||
| version: 7.x.x | ||
| repository: https://charts.bitnami.com/bitnami | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| # airbyte | ||
|
|
||
| ## Parameters | ||
|
|
||
| ### Global Parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | --------------------- | -------------------------------------------- | ----- | | ||
| | `global.storageClass` | Global StorageClass for Persistent Volume(s) | `""` | | ||
|
|
||
|
|
||
| ### Common Parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | ---------------------------- | ------------------------------------------------------------------------------------------------------------------- | --------------- | | ||
| | `nameOverride` | String to partially override airbyte.fullname template with a string (will prepend the release name) | `""` | | ||
| | `fullnameOverride` | String to fully override airbyte.fullname template with a string | `""` | | ||
| | `serviceAccount.annotations` | Annotations for service account. Evaluated as a template. Only used if `create` is `true`. | `{}` | | ||
| | `serviceAccount.create` | Specifies whether a ServiceAccount should be created | `true` | | ||
| | `serviceAccount.name` | Name of the service account to use. If not set and create is true, a name is generated using the fullname template. | `airbyte-admin` | | ||
|
|
||
|
|
||
| ### Webapp Parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | ---------------------------- | ---------------------------------------------------------------- | ---------------- | | ||
| | `webapp.replicaCount` | Number of webapp replicas | `1` | | ||
| | `webapp.image.repository` | The repository to use for the airbyte webapp image. | `airbyte/webapp` | | ||
| | `webapp.image.pullPolicy` | the pull policy to use for the airbyte webapp image | `IfNotPresent` | | ||
| | `webapp.image.tag` | The airbyte webapp image tag. Defaults to the chart's AppVersion | `0.29.13-alpha` | | ||
| | `webapp.podAnnotations` | Add extra annotations to the scheduler pod | `{}` | | ||
| | `webapp.service.type` | The service type to use for the webapp service | `ClusterIP` | | ||
| | `webapp.service.port` | The service port to expose the webapp on | `80` | | ||
| | `webapp.resources.limits` | The resources limits for the Web container | `{}` | | ||
| | `webapp.resources.requests` | The requested resources for the Web container | `{}` | | ||
| | `webapp.nodeSelector` | Node labels for pod assignment | `{}` | | ||
| | `webapp.tolerations` | Tolerations for webapp pod assignment. | `[]` | | ||
| | `webapp.ingress.enabled` | Set to true to enable ingress record generation | `false` | | ||
| | `webapp.ingress.className` | Specifies ingressClassName for clusters >= 1.18+ | `""` | | ||
| | `webapp.ingress.hosts` | Ingress Hosts configuration | `[]` | | ||
| | `webapp.ingress.annotations` | Ingress annotations done as key:value pairs | `{}` | | ||
| | `webapp.ingress.hosts` | The list of hostnames to be covered with this ingress record. | `[]` | | ||
| | `webapp.ingress.tls` | Custom ingress TLS configuration | `[]` | | ||
|
|
||
|
|
||
| ### Scheduler Parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | ------------------------------ | ------------------------------------------------------------------- | ------------------- | | ||
| | `scheduler.replicaCount` | Number of scheduler replicas | `1` | | ||
| | `scheduler.image.repository` | The repository to use for the airbyte scheduler image. | `airbyte/scheduler` | | ||
| | `scheduler.image.pullPolicy` | the pull policy to use for the airbyte scheduler image | `IfNotPresent` | | ||
| | `scheduler.image.tag` | The airbyte scheduler image tag. Defaults to the chart's AppVersion | `0.29.13-alpha` | | ||
| | `scheduler.podAnnotations` | Add extra annotations to the scheduler pod | `{}` | | ||
| | `scheduler.resources.limits` | The resources limits for the scheduler container | `{}` | | ||
| | `scheduler.resources.requests` | The requested resources for the scheduler container | `{}` | | ||
| | `scheduler.nodeSelector` | Node labels for pod assignment | `{}` | | ||
| | `scheduler.tolerations` | Tolerations for scheduler pod assignment. | `[]` | | ||
|
|
||
|
|
||
| ### Pod Sweeper parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | ------------------------------- | ---------------------------------------------------- | ----------------- | | ||
| | `podSweeper.image.repository` | The image repository to use for the pod sweeper | `bitnami/kubectl` | | ||
| | `podSweeper.image.pullPolicy` | The pull policy for the pod sweeper image | `IfNotPresent` | | ||
| | `podSweeper.image.tag` | The pod sweeper image tag to use | `latest` | | ||
| | `podSweeper.podAnnotations` | Add extra annotations to the podSweeper pod | `{}` | | ||
| | `podSweeper.resources.limits` | The resources limits for the podSweeper container | `{}` | | ||
| | `podSweeper.resources.requests` | The requested resources for the podSweeper container | `{}` | | ||
| | `podSweeper.nodeSelector` | Node labels for pod assignment | `{}` | | ||
| | `podSweeper.tolerations` | Tolerations for podSweeper pod assignment. | `[]` | | ||
|
|
||
|
|
||
| ### Server parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | ------------------------------------------- | ---------------------------------------------------------------- | ---------------- | | ||
| | `server.replicaCount` | Number of server replicas | `1` | | ||
| | `server.image.repository` | The repository to use for the airbyte server image. | `airbyte/server` | | ||
| | `server.image.pullPolicy` | the pull policy to use for the airbyte server image | `IfNotPresent` | | ||
| | `server.image.tag` | The airbyte server image tag. Defaults to the chart's AppVersion | `0.29.13-alpha` | | ||
| | `server.podAnnotations` | Add extra annotations to the server pod | `{}` | | ||
| | `server.livenessProbe.enabled` | Enable livenessProbe on the server | `true` | | ||
| | `server.livenessProbe.initialDelaySeconds` | Initial delay seconds for livenessProbe | `30` | | ||
| | `server.livenessProbe.periodSeconds` | Period seconds for livenessProbe | `10` | | ||
| | `server.livenessProbe.timeoutSeconds` | Timeout seconds for livenessProbe | `1` | | ||
| | `server.livenessProbe.failureThreshold` | Failure threshold for livenessProbe | `3` | | ||
| | `server.livenessProbe.successThreshold` | Success threshold for livenessProbe | `1` | | ||
| | `server.readinessProbe.enabled` | Enable readinessProbe on the server | `true` | | ||
| | `server.readinessProbe.initialDelaySeconds` | Initial delay seconds for readinessProbe | `10` | | ||
| | `server.readinessProbe.periodSeconds` | Period seconds for readinessProbe | `10` | | ||
| | `server.readinessProbe.timeoutSeconds` | Timeout seconds for readinessProbe | `1` | | ||
| | `server.readinessProbe.failureThreshold` | Failure threshold for readinessProbe | `3` | | ||
| | `server.readinessProbe.successThreshold` | Success threshold for readinessProbe | `1` | | ||
| | `server.resources.limits` | The resources limits for the server container | `{}` | | ||
| | `server.resources.requests` | The requested resources for the server container | `{}` | | ||
| | `server.service.type` | The service type to use for the API server | `ClusterIP` | | ||
| | `server.service.port` | The service port to expose the API server on | `8001` | | ||
| | `server.persistence.accessMode` | The access mode for the airbyte server pvc | `ReadWriteOnce` | | ||
| | `server.persistence.size` | The size of the pvc to use for the airbyte server pvc | `1Gi` | | ||
| | `server.persistence.storageClass` | The storage class to use for the airbyte server pvc | `""` | | ||
| | `server.nodeSelector` | Node labels for pod assignment | `{}` | | ||
| | `server.tolerations` | Tolerations for server pod assignment. | `[]` | | ||
|
|
||
|
|
||
| ### Temporal parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | --------------------------- | --------------------------------------------- | ----------------------- | | ||
| | `temporal.replicaCount` | The number of temporal replicas to deploy | `1` | | ||
| | `temporal.image.repository` | The temporal image repository to use | `temporalio/auto-setup` | | ||
| | `temporal.image.pullPolicy` | The pull policy for the temporal image | `IfNotPresent` | | ||
| | `temporal.image.tag` | The temporal image tag to use | `1.7.0` | | ||
| | `temporal.service.type` | The Kubernetes Service Type | `ClusterIP` | | ||
| | `temporal.service.port` | The temporal port and exposed kubernetes port | `7233` | | ||
| | `temporal.nodeSelector` | Node labels for pod assignment | `{}` | | ||
| | `temporal.tolerations` | Tolerations for pod assignment. | `[]` | | ||
|
|
||
|
|
||
| ### Airbyte Database parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | -------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------ | | ||
| | `postgresql.enabled` | Switch to enable or disable the PostgreSQL helm chart | `true` | | ||
| | `postgresql.postgresqlUsername` | Airbyte Postgresql username | `airbyte` | | ||
| | `postgresql.postgresqlPassword` | Airbyte Postgresql password | `airbyte` | | ||
| | `postgresql.postgresqlDatabase` | Airbyte Postgresql database | `db-airbyte` | | ||
| | `postgresql.existingSecret` | Name of an existing secret containing the PostgreSQL password ('postgresql-password' key) | `""` | | ||
| | `externalDatabase.host` | Database host | `localhost` | | ||
| | `externalDatabase.user` | non-root Username for Airbyte Database | `airbyte` | | ||
| | `externalDatabase.password` | Database password | `""` | | ||
| | `externalDatabase.existingSecret` | Name of an existing secret resource containing the DB password | `""` | | ||
| | `externalDatabase.existingSecretPasswordKey` | Name of an existing secret key containing the DB password | `""` | | ||
| | `externalDatabase.database` | Database name | `db-airbyte` | | ||
| | `externalDatabase.port` | Database port number | `5432` | | ||
|
|
||
|
|
||
| ### Minio parameters | ||
|
|
||
| | Name | Description | Value | | ||
| | -------------------------- | ------------------------------------------------ | ----------- | | ||
| | `minio.enabled` | Switch to enable or disable the Minio helm chart | `true` | | ||
| | `minio.accessKey.password` | Minio Access Key | `minio` | | ||
| | `minio.secretKey.password` | Minio Secret Key | `minio123` | | ||
| | `externalMinio.host` | Minio Host | `localhost` | | ||
| | `externalMinio.port` | Minio Port | `9000` | | ||
|
|
||
|
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This step should check for docs that aren't in sync with the generated.