Improve kube deploy process. #13397
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,40 +1,51 @@ | ||
| # We would prefer to use Pod instead of Job here for exactly-once execution guarantee, however | ||
| # Pods have a problem of sticking around after completion, causing errors upon upgrade if they | ||
| # are not manually deleted first. | ||
| # Using generateName would solve this by giving each bootloader pod a unique name, but Kustomize | ||
| # does not currently support generateName. | ||
| # Therefore, using Job here with a ttl is required in order to have a smooth upgrade process for kube. | ||
| # See discussion about this on the PR: https://github.com/airbytehq/airbyte/pull/13397#discussion_r887600449 | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: airbyte-bootloader | ||
| spec: | ||
| # This ttl is necessary to prevent errors when upgrading airbyte | ||
| ttlSecondsAfterFinished: 5 | ||
|
There was a problem hiding this comment. As mentioned in the PR description, this is necessary to avoid an error being about the airbyte-bootloader job being immutable. This isn't an ideal solution, because this means that the bootloader pod is deleted after it completes, making its logs inaccessible through kube. This was the only solution I could come up with that allowed us to still use Definitely open to feedback here if there are any other options I haven't considered. There was a problem hiding this comment. can you add a comment explaining why we do this, please? |
||
| template: | ||
| spec: | ||
| restartPolicy: Never | ||
| containers: | ||
| - name: airbyte-bootloader-container | ||
| image: airbyte/bootloader | ||
| env: | ||
| - name: AIRBYTE_VERSION | ||
| valueFrom: | ||
| configMapKeyRef: | ||
| name: airbyte-env | ||
| key: AIRBYTE_VERSION | ||
| - name: DATABASE_HOST | ||
| valueFrom: | ||
| configMapKeyRef: | ||
| name: airbyte-env | ||
| key: DATABASE_HOST | ||
| - name: DATABASE_PORT | ||
| valueFrom: | ||
| configMapKeyRef: | ||
| name: airbyte-env | ||
| key: DATABASE_PORT | ||
| - name: DATABASE_PASSWORD | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: airbyte-secrets | ||
| key: DATABASE_PASSWORD | ||
| - name: DATABASE_URL | ||
| valueFrom: | ||
| configMapKeyRef: | ||
| name: airbyte-env | ||
| key: DATABASE_URL | ||
| - name: DATABASE_USER | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: airbyte-secrets | ||
| key: DATABASE_USER | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Kube jobs only have best effort parallelism guarantees, which is why I don't really like using them for crucial workflows. Took a look and confirmed this is probably the best way of doing this with Kustomize. Can we add a comment here that we generally want to use
Pod(our Helm charts use Pod) for the best exactly-once execution guarantees and cannot do so because Kustomize does not supportgenerateName? Want to prevent confusion in the future.If Kustomize did support
generateName, we should be able to instruct users to runkubectl createon initial create andreplaceon subsequent runs.This happens relatively infrequently so risk is low. In the long term, I think we'll consolidate the Kube deploys into Helm so I think this is fine for now.