Skip to content

Kustomize resource ordering regression #3794

Closed
@yanniszark

Description

@yanniszark

Describe the bug

In Kubeflow, we are using Kustomize v3.2.0 and want to upgrade to v4.0.5: kubeflow/manifests#1797
However, our deployment failed for v4.0.5, while it succeeded for v3.2.0.

This is what happened:

  1. In later kustomize versions, it orders Admission Webhooks last. Before, they were first.
  2. Pods are created before the istio injection webhook, thus they don't get a sidecar.
  3. Our apps fail because they haven't been mutated appropriately.

Regression Background

For reference, here is the original issue and PRs that made these changes:
#821
#1104
#2459

Issue #821 presents the following scenario, which led to PR #1104:

  1. User builds cert-manager kustomization, containing a webhook, a deployment and a CR (simplified).
  2. Apply webhook and deployment.
  3. Apply CR.

Step (3) fails because the Deployment has not become ready yet.
The solution SHOULD be to retry the apply.
PR #1104 solution was to order the webhook last, so that it doesn't mutate/validate the CR.
This is false, as it circumvents logic that the application has explicitly declared should be applied to all relevant resources.

Files that can reproduce the issue

Please see: https://github.com/kubeflow/manifests/blob/v1.3-branch/README.md
which includes the example kustomization we use for Kubeflow components.

  • Build the example kustomization with kustomize v3.2.0, as per the README.
  • Build the example kustomization with kustomize v4.0.5. You will see the WebhookConfigurations ordered last, which causes the issues.

Proposed Solution

Restore the order of Mutating / Validating Webhooks as it was before PR #1104

Kustomize version

v4.0.5

cc'ing authors of the referenced issues and PRs: @donbowman @mgoltzsche @asadali
cc @monopole @Shell32-Natsu @pwittrock

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions