Support helm release instead of helm template

[eugene-chernyshenko]

Summary

ArgoCD applies yaml rendered with helm template.
It makes the typical workflow with helm difficult.
We are unable to use helm features like listing releases and showing the diff

Proposal

It's could be better to use helm upgrade --install instead of helm template with apply

[eugene-chernyshenko]

May be it's could be possible to do with configManagementPlugins
but there are only init and generate commands available

[gadiener]

Related to #1672 and #7759. That would be a really great enhancement!

[rainest]

From the chart design end of things, use of helm template makes our lives much more complicated, since it breaks expectations of how the release will be managed and breaks important pieces of Helm functionality (#3640 in addition to the items mentioned above) that we need to ensure correct functionality when install/upgrade are used.

Simply using install and upgrade/allowing Helm to manage the releases would remove that problem, though I don't know if it would conflict with ArgoCD's own release management tooling. If it's not possible to support Helm releases, would it be possible to publish an ArgoCD chart developer guide, with explanations of how ArgoCD is replacing parts of Helm functionality, known limitations, and typical workarounds?

[Eugst]

+1

[jhemarcos]

This behavior of not using helm install/update discourages people who would like to use the Argo CD but have been using Helm for a long time. It's an odd decision, as Helm adoption is huge.

[neoakris]

Hi I want to point out a workaround and play a bit of devil's advocate to point out a reason not to implement your feature / why the current implementation / doing nothing might be better than what you're suggesting:

Workaround:

1. You can install Flux v2 which is an alternative to ArgoCD that does use the helm cli. So deploying helm charts with Flux v2 is one option.
2. Another option is to use ArgoCD to wrap Flux v2. Since Flux v2 uses CRDs & CRs (which are just kube yaml manifests), You could use ArgoCD to deploy Flux v2 HelmRelease Custom Resources.
3. My personal preference is use a CICD pipeline, but if I was on a project where the decision was made to use ArgoCD or FluxCD, I'd suggest using both* with the idea of using pure ArgoCD ~95% of the time. And then using FluxCD for edge cases where ArgoCD driven helm isn't a good fit. (Edge cases I'm aware of are that Argo Hooks aren't as good / accurate as Helm Hooks, and there's actually a lot more helm repo formats than you might think and FluxCD has best support for uncommon helm repo formats.)

(Disclaimer: I like ArgoCD more than Flux v2 to the point where I've written a blog post, https://www.doit-intl.com/avoid-fluxcd-for-gitops/, actively encouraging people to consider avoiding Flux v2. That being said I'm aware that ArgoCD isn't perfect / universally better than Flux v2. My primary reason for liking ArgoCD > FluxCD is purely from a reliability standpoint / SRE(site reliability engineer) perspective. And I mean SRE in terms of simpler software solutions are less buggy/more reliable, vs complex things like istio.)

Devils Advocate of why the current approach could be considered best:
Let me start by saying I know the current approach isn't 100% compatible with helm charts. I think that lack of compatibility actually increases reliability.

I've run into rare edge cases when using the Helm CLI (v2 and v3) and FluxCD(Flux v2).

• With Flux I've seen a Flux triggered HelmRelease get into a stuck state, where trying to change the git repo using GitOps, does nothing. And you need to either do flux stop, flux start, flux reconcile helmrelease --with-source, or delete and recreate it in order to get it unstuck / out of the confused state and accepting Git triggered / GitOps changes again.
• With Helm if you successfully install, then you do an upgrade and the upgrade fails or had malformed input parameters passed into the helm values files that caused errors it can sometimes do a helm rollback. I've run into scenarios where when using helm if the initial install failed due to bad input data, helm got into a stuck state where it had nothing to roll back to, and it'd block future updates, so you'd have to delete the helm release and reinstall the helm release before it'd work.

Why I'm not a fan of the helm cli approach:
Helm cli adds just enough complexity for interesting behavior to emerge and things to go wrong / interesting failure mode scenarios to arise.
Why I like ArgoCD's approach:
ArgoCD's just use helm as a dumb templating engine & kubectl apply -f the results is so KISS that there's little that can go wrong, and because of that in my experience ArgoCD has always had rock solid reliability.

[Eugst]

It would be nice to have feature in the tool than switch the tool. If people would like to use another tool, they won’t ask here for the feature.
reminds me apple-pear discussion. 🤷‍♂️

[shinebayar-g]

Crossposting my comment from the other thread.

+1, IIRC Helm v3 is client side only and have no server components. In theory ArgoCD could support Helm deployments (still could keep using helm template and implement correct behaviors) more than just templating. How much gap is there between helm install vs helm template? Feels like main helm command could be supported. For example, helm list , helm history, helm status etc are useful commands. As many others said it's similar to kubectl. Even if helm install or helm upgrade is not supported that's fine. It's no different than manual kubectl edits, and should be responsibility of the user, not argocd limitation.

Edit: Helm has a native Golang SDK which would allow ArgoCD to integrate with Helm natively.

[Almenon]

Another thing to note here is this is intuitively what a beginner would expect argocd to be doing. I was surprised when I didn't see a helm release in my namespace.

[AlexisDucastel]

I do totally agree, beginners expect argo to use Helm lifecycle. It may be a choice to the user to want to sync with helm upgrade/install or helm template. That would be even more clear for understanding what to expect.

[davidham]

I actually prefer it that Argo CD only uses helm template. I think a big failure of Helm is that it tries to do too much, and it locks people into its CLI and practices when there are better alternatives available. Argo can work with your Helm chart, and it doesn't require chart repos, or version history stored in Secrets. Keep Argo agnostic on templating tools!

[rowi1de]

I actually prefer it that Argo CD only uses helm template. I think a big failure of Helm is that it tries to do too much, and it locks people into its CLI and practices when there are better alternatives available. Argo can work with your Helm chart, and it doesn't require chart repos, or version history stored in Secrets. Keep Argo agnostic on templating tools!

The surprises you get with unsupported features for a lot for 3rd part charts is huge (of course questionable if random values & lookup comply with gitops) #14944 #14943

Using helm template to deploy or install helm charts in the first place is questionable IMO.

Render chart templates locally and display the output. Any values that would normally be looked up or retrieved in-cluster will be faked locally. Additionally, none of the server-side testing of chart validity (e.g. whether an API is supported) is done.

While I agree on the “too much part”
a lot of charts are out there and in production use which won’t be changed (unless someone proposes a Helm 4 with “pure” functions)
Alternatively people can spend a lot of time on adding workarounds for those charts or use fluxCD 😇

Picking up the “agnostic” idea: Does ArgoCD support plugins for Sync & Deploy? Having “helm template” in the core code already ties it to helm

[davidham]

It’s not that Argo CD “doesn’t support” those functions—Argo CD will render your template and have it generate random data just fine—it’s that these functions, when working as intended, conflict with Argo’s GitOps model. But it’s not Argo’s job to protect you from weird things that happen in 3rd party Helm charts.

As for Argo usinghelm template, I think it’s a great solution. All of the tools Argo supports work the same way, they render the Helm chart/JSonnet/kustomize into YAML, and then apply that to the cluster.

Helm is a templating tool (with the chart model) but it’s also very opinionated on how apps are deployed, with its Releases and rollbacks and whatever. I think Argo is correct to support Helm’s template features and not its deployment features.

[TheBritz]

Has anyone explored the possibility of using a helm post-processor to render the template rather than just helm template? I know this wouldn't create a helm release (which some folks here are asking for) but it would at least allow for functions like lookup to work which aren't possible with just a helm template.

[rainest]

Since it wasn't mentioned here yet, there is now Helm functionality to enable lookup in helm template, but Argo would need additional functionality to use it: #5202 (comment)

[lucasfcnunes]

Not using helm install breaks helm hooks (post-install, post-upgrade, ...) possibly making some installations buggy.

e.g.

• https://github.com/apache/airflow/blob/a64bb40a0f4fbe18939c479adae6a346734e5d35/chart/templates/jobs/create-user-job.yaml#L45-L48
• https://github.com/apache/airflow/blob/a64bb40a0f4fbe18939c479adae6a346734e5d35/chart/templates/jobs/migrate-database-job.yaml#L45-L49

--

I'm actually trying to find a solution with helmfile. All plugins use helmfile template instead of helmfile apply (actually uses helm install in the end).
A friend of mine, @gabriel-suela, found a guy that uses helmfile apply on Argo CD but I have no ideia how he did it.

Related

• How to integrate with Argo CD without losing native helmfile features? mumoshu/helmfile-operator#7

[cogentist-yann]

A question is:
how is it working when uninstalling the application. Deleting the ArgoCD application seems to just uninstall the ArgoCD application but keep the resources in K8s.
Is it a way to ensure deleting the ArgoCD application remove the deployed helm template applied yaml ?

[nwsparks]

A major thing that is missing is the ability to helm diff against a deployed app when making changes to the chart. You can kind of get this with helm template | kubectl diff -f -, but it is littered with metadata that does not appear in helm diff making it more difficult to work with.

[crenshaw-dev]

Not using helm install breaks helm hooks (post-install, post-upgrade, ...) possibly making some installations buggy.

Argo CD has partial support for Helm hooks: https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#helm-hooks

how is it working when uninstalling the application. Deleting the ArgoCD application seems to just uninstall the ArgoCD application but keep the resources in K8s.

Argo CD can do cascading deletion: https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/#about-the-deletion-finalizer

A major thing that is missing is the ability to helm diff

Argo CD has some diffing capability with argocd app diff --local. It's admittedly not as good a UX as it should be. I believe adding a first-class PR diffing bot would significantly improve this experience (related ArgoCon talk: https://www.youtube.com/watch?v=p5EPKY3vM-E).

[nwsparks]

Argo CD has some diffing capability

Yah, it's still not ideal when developing charts in Helm though. It suffers from the same problems as I stated already except now I need to download and configure yet another tool on top of things.

I think the root of the problem is that it is just clunky, not intuitive and generally does not work well when having to jump between tools to get things done that the original tool is already providing.

Given how ubiquitous Helm is, it would make sense to provide the option to support it natively.

[crenshaw-dev]

Given how ubiquitous Helm is, it would make sense to provide the option to support it natively.

The problem is that Argo CD and Helm are just different tools with different approaches to the problems of development, deployment, secret management, etc. If Argo CD were to provide all of Helms features exactly as Helm provides them, then Argo CD would be a very different product (with probably a lot more code and complexity). By taking the Helm features that fit within Argo CD's existing approach and leaving the parts that don't fit, we are able to provide a coherent experience across multiple config management tools instead of just one.

[voidlily]

One thing I miss from argocd not having helm install is that it's harder to track out of date helm charts with tools like nova. Is anyone aware of equivalent tools that work with how argocd manages helm charts and can check for helm chart updates?

[davidham]

This might be a good feature request for Nova, if they could read Argo CD Application objects as well as scan for HelmReleases. Argo CD is more focused on deploying K8s applications, some of which may be Helm charts and some not, and monitoring their deployed state for changes against the Application spec.

[danieljkemp]

One thing I miss from argocd not having helm install is that it's harder to track out of date helm charts with tools like nova. Is anyone aware of equivalent tools that work with how argocd manages helm charts and can check for helm chart updates?

I've found that renovate works perfectly for this use case

[crenshaw-dev]

Closing for now as there are no active plans to support helm release and there seems to be maintainer consensus that it would be the wrong direction for the project anyway. (Not to say that cons described in this thread are invalid, just that we don't currently believe they outweigh the pros of sticking with helm template.)

[clayrisser]

I wonder if the project could at least support showing the resources a Flux helm release created. When I create Flux HelmRelease resources I am no longer able to track the resources in argo.

[davidham]

Are you using Flux and Argo CD together?

[crenshaw-dev]

@clayrisser Argo CD currently only uses owner references to build the resource tree. People have proposed alternative tree-construction methods (for example, by setting a label), but none of those proposals currently have traction.

[clayrisser]

@davidham yes I am. I have to use flux because argocd doesn't correctly support the {{ lookup }} function.