Description
Describe the bug
In Kubeflow, we are using Kustomize v3.2.0
and want to upgrade to v4.0.5
: kubeflow/manifests#1797
However, our deployment failed for v4.0.5
, while it succeeded for v3.2.0
.
This is what happened:
- In later kustomize versions, it orders Admission Webhooks last. Before, they were first.
- Pods are created before the istio injection webhook, thus they don't get a sidecar.
- Our apps fail because they haven't been mutated appropriately.
Regression Background
For reference, here is the original issue and PRs that made these changes:
#821
#1104
#2459
Issue #821 presents the following scenario, which led to PR #1104:
- User builds cert-manager kustomization, containing a webhook, a deployment and a CR (simplified).
- Apply webhook and deployment.
- Apply CR.
Step (3) fails because the Deployment has not become ready yet.
The solution SHOULD be to retry the apply.
PR #1104 solution was to order the webhook last, so that it doesn't mutate/validate the CR.
This is false, as it circumvents logic that the application has explicitly declared should be applied to all relevant resources.
Files that can reproduce the issue
Please see: https://github.com/kubeflow/manifests/blob/v1.3-branch/README.md
which includes the example
kustomization we use for Kubeflow components.
- Build the
example
kustomization with kustomizev3.2.0
, as per the README. - Build the
example
kustomization with kustomizev4.0.5
. You will see the WebhookConfigurations ordered last, which causes the issues.
Proposed Solution
Restore the order of Mutating / Validating Webhooks as it was before PR #1104
Kustomize version
v4.0.5
cc'ing authors of the referenced issues and PRs: @donbowman @mgoltzsche @asadali
cc @monopole @Shell32-Natsu @pwittrock