Skip to content

feat: Update oss charts #15719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Aug 19, 2022
Merged

feat: Update oss charts #15719

merged 14 commits into from
Aug 19, 2022

Conversation

xpuska513
Copy link
Contributor

@xpuska513 xpuska513 commented Aug 17, 2022

What

  • Include disable minio env if minios is not enabled #15685 changes in this PR (Disable minio from deployment)
  • Update airbyte-worker, webapp,bootloader, server to be able to be consumed by cloud deployment
  • Update way of defining secrets, make it more user friendly

Recommended reading order

  1. charts/x/deployment.yaml
  2. charts/x/secrets.yaml
  3. charts/x/pdb.yaml
  4. charts/x/hpa.yaml
  5. charts/x/values.yaml

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

@xpuska513 xpuska513 changed the title Update oss charts feat: Update oss charts Aug 17, 2022
@xpuska513 xpuska513 requested review from supertopher and git-phu and removed request for supertopher August 17, 2022 11:45
@github-actions github-actions bot added area/platform issues related to the platform kubernetes labels Aug 17, 2022
@xpuska513 xpuska513 requested a review from supertopher August 17, 2022 11:45
@xpuska513 xpuska513 requested review from a team and davinchia August 17, 2022 12:29
@xpuska513
Copy link
Contributor Author

image

Copy link
Contributor

@supertopher supertopher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good captain

app.airbyte.io/fullname: {{ include "airbyte.fullname" . }}
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-weight: "-1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

secretKeyRef:
name: webapp-secrets
key: {{ $k }}
{{- end }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the spacing for this end block seems inconsistent with likes 97-98. I don't know what helm standards are.

does this tabbing match helm standard practice?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tabbing for the gotmpl stuff is not required in general, unless you just want the code to look more beautified

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I always want the code to look pretty.

I don't have enough depth here to have strong opinions though so as long as we are consistent ¯\(ツ)

Copy link
Contributor

@davinchia davinchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

Two major comments:

  1. Can we think of a better way to not leak the GSM abstraction? Or at least make it clear this is Airbyte Cloud only/a way for users to inject secret maps?
  2. We don't need a worker HPA in OSS. Is this only for Cloud? Can we override it on inheritance?

@xpuska513 xpuska513 requested a review from davinchia August 18, 2022 06:53
@xpuska513
Copy link
Contributor Author

Hi @davinchia ,

  1. I'll try an another approach of handling cloud based attributes today(after this PR gets merged if no other questions present). Already removed everything related to cloud except the ordinary secrets(since thought it'll give more flexebility in terms of deploying each component individually)
  2. Already removed it from oss and moved it into cloud alongside with PDB(Pod Distribution Budget)

Copy link
Contributor

@git-phu git-phu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Have a few questions but otherwise it looks good!

Copy link
Contributor

@git-phu git-phu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

Copy link
Contributor

@supertopher supertopher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. ty @xpuska513

secretKeyRef:
name: webapp-secrets
key: {{ $k }}
{{- end }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I always want the code to look pretty.

I don't have enough depth here to have strong opinions though so as long as we are consistent ¯\(ツ)

app.airbyte.io/fullname: {{ include "airbyte.fullname" . }}
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-weight: "-1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why is this -1?

extraEnv: []
extraEnv: {}
secrets: {}
gsm: {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we get rid of all the gsm values?

- name: gcs-log-creds-volume
secret:
secretName: {{ ternary (printf "%s-gcs-log-creds" ( .Release.Name )) (.Values.global.credVolumeOverride) (eq .Values.global.deploymentMode "oss") }}
{{- end }}
{{- if .Values.extraVolumes }}
{{ toYaml .Values.extraVolumes | nindent 6 }}
{{- end }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: newline


secrets: {}
gsm: {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here on removing this.

There are a few other places too.

@@ -656,6 +656,9 @@ worker:
##
extraVolumes: []

hpa:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can also get rid of this

Copy link
Contributor

@davinchia davinchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xpuska513 !

Looks like there is still minor leftover values - please remove them before merging.

There are also some Cloud related deployment bits here with the volume mounts. At least it's clear what these are for - so not a big issue.

Is there documentation to clarify how secrets and the extra env blocks are handled? We definitely need those. Can do in a separate PR. Thanks for the good work!

@xpuska513
Copy link
Contributor Author

@davinchia i'll do a separate pr for docs update, right now working on that one

@xpuska513 xpuska513 merged commit 626f51f into master Aug 19, 2022
@xpuska513 xpuska513 deleted the update-oss-charts branch August 19, 2022 12:09
rodireich pushed a commit that referenced this pull request Aug 25, 2022
* fix: revert extraEnv delition in values.yaml for bootloader

* add newline

* feat: Update bootloader,webapp,server. Add way of defining secrets, update extraEnv usage. Add PodDistributionBudget into all deployments

* feat: Update oss charts, make them able to be ingested in cloud deployment

* fix: include #15685 changes

* fix: Update Chart.yaml. fix minio deployment conditional operator

* fix: fix EOF in worker, update worker HPA conditional

* fix: remove cloud related stuff

* fix: add conditional for hooks

* fix: remove hooks for worker

* fix: update nit, remove gsm

* fix: fix nits

* fix: remove gsm and hpa from values.yaml
@@ -2,7 +2,7 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "common.names.fullname" . }}
name: airbyte-server-svc

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wonder why we hard-coded the service names. This kinda messed up the branch deployments :|

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform kubernetes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants