Skip to content

[RFC-0009] Add CEL custom healthchecks #1344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 31, 2025
Merged

[RFC-0009] Add CEL custom healthchecks #1344

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions api/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ module github.com/fluxcd/kustomize-controller/api
go 1.23.0

require (
github.com/fluxcd/pkg/apis/kustomize v1.8.0
github.com/fluxcd/pkg/apis/meta v1.9.0
k8s.io/apiextensions-apiserver v0.32.0
k8s.io/apimachinery v0.32.0
github.com/fluxcd/pkg/apis/kustomize v1.9.0
github.com/fluxcd/pkg/apis/meta v1.10.0
k8s.io/apiextensions-apiserver v0.32.1
k8s.io/apimachinery v0.32.1
sigs.k8s.io/controller-runtime v0.19.3
)

Expand All @@ -23,8 +23,8 @@ require (
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/x448/float16 v0.8.4 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/text v0.19.0 // indirect
golang.org/x/net v0.34.0 // indirect
golang.org/x/text v0.21.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 // indirect
Expand Down
32 changes: 16 additions & 16 deletions api/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/fluxcd/pkg/apis/kustomize v1.8.0 h1:HH6YRa3SMS72KK4cUyb9m5sK/dZH+Eti1qhjWDCgwKg=
github.com/fluxcd/pkg/apis/kustomize v1.8.0/go.mod h1:QCKIFj1ocdndaWSkrLs5JKvdGNYyTzQX1ZB3lYTwma0=
github.com/fluxcd/pkg/apis/meta v1.9.0 h1:wPgm7bWNJZ/ImS5GqikOxt362IgLPFBG73dZ27uWRiQ=
github.com/fluxcd/pkg/apis/meta v1.9.0/go.mod h1:pMea8eEZcsFSI7ngRnTHFtDZk2CEZGgtrueNgI6Iu70=
github.com/fluxcd/pkg/apis/kustomize v1.9.0 h1:SJpT1CK58AnTvCpDKeGfMNA0Xud/4VReZNvPe8XkTxo=
github.com/fluxcd/pkg/apis/kustomize v1.9.0/go.mod h1:AZl2GU03oPVue6SUivdiIYd/3mvF94j7t1G2JO26d4s=
github.com/fluxcd/pkg/apis/meta v1.10.0 h1:rqbAuyl5ug7A5jjRf/rNwBXmNl6tJ9wG2iIsriwnQUk=
github.com/fluxcd/pkg/apis/meta v1.10.0/go.mod h1:n7NstXHDaleAUMajcXTVkhz0MYkvEXy1C/eLI/t1xoI=
github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E=
github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
Expand Down Expand Up @@ -65,20 +65,20 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.30.0 h1:AcW1SDZMkb8IpzCdQUaIq2sP4sZ4zw+55h6ynffypl4=
golang.org/x/net v0.30.0/go.mod h1:2wGyMJ5iFasEhkwi13ChkO/t1ECNC4X4eBKkVFyYFlU=
golang.org/x/net v0.34.0 h1:Mb7Mrk043xzHgnRM88suvJFwzVrRfHEHJEl5/71CKw0=
golang.org/x/net v0.34.0/go.mod h1:di0qlW3YNM5oh6GqDGQr92MyTozJPmybPK4Ev/Gm31k=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.29.0 h1:TPYlXGxvx1MGTn2GiZDhnjPA9wZzZeGKHHmKhHYvgaU=
golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.19.0 h1:kTxAhCbGbxhK0IwgSKiMO5awPoDQ0RpfiVYBfK860YM=
golang.org/x/text v0.19.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
Expand All @@ -96,12 +96,12 @@ gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
k8s.io/api v0.32.0 h1:OL9JpbvAU5ny9ga2fb24X8H6xQlVp+aJMFlgtQjR9CE=
k8s.io/api v0.32.0/go.mod h1:4LEwHZEf6Q/cG96F3dqR965sYOfmPM7rq81BLgsE0p0=
k8s.io/apiextensions-apiserver v0.32.0 h1:S0Xlqt51qzzqjKPxfgX1xh4HBZE+p8KKBq+k2SWNOE0=
k8s.io/apiextensions-apiserver v0.32.0/go.mod h1:86hblMvN5yxMvZrZFX2OhIHAuFIMJIZ19bTvzkP+Fmw=
k8s.io/apimachinery v0.32.0 h1:cFSE7N3rmEEtv4ei5X6DaJPHHX0C+upp+v5lVPiEwpg=
k8s.io/apimachinery v0.32.0/go.mod h1:GpHVgxoKlTxClKcteaeuF1Ul/lDVb74KpZcxcmLDElE=
k8s.io/api v0.32.1 h1:f562zw9cy+GvXzXf0CKlVQ7yHJVYzLfL6JAS4kOAaOc=
k8s.io/api v0.32.1/go.mod h1:/Yi/BqkuueW1BgpoePYBRdDYfjPF5sgTr5+YqDZra5k=
k8s.io/apiextensions-apiserver v0.32.1 h1:hjkALhRUeCariC8DiVmb5jj0VjIc1N0DREP32+6UXZw=
k8s.io/apiextensions-apiserver v0.32.1/go.mod h1:sxWIGuGiYov7Io1fAS2X06NjMIk5CbRHc2StSmbaQto=
k8s.io/apimachinery v0.32.1 h1:683ENpaCBjma4CYqsmZyhEzrGz6cjn1MY/X2jB2hkZs=
k8s.io/apimachinery v0.32.1/go.mod h1:GpHVgxoKlTxClKcteaeuF1Ul/lDVb74KpZcxcmLDElE=
k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 h1:M3sRQVHv7vB20Xc2ybTt7ODCeFj6JSWYFzOFnYeS6Ro=
Expand Down
6 changes: 6 additions & 0 deletions api/v1/kustomization_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,12 @@ type KustomizationSpec struct {
// Components specifies relative paths to specifications of other Components.
// +optional
Components []string `json:"components,omitempty"`

// HealthCheckExprs is a list of healthcheck expressions for evaluating the
// health of custom resources using Common Expression Language (CEL).
// The expressions are evaluated only when Wait or HealthChecks are specified.
// +optional
HealthCheckExprs []kustomize.CustomHealthCheck `json:"healthCheckExprs,omitempty"`
}

// CommonMetadata defines the common labels and annotations.
Expand Down
5 changes: 5 additions & 0 deletions api/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

36 changes: 36 additions & 0 deletions config/crd/bases/kustomize.toolkit.fluxcd.io_kustomizations.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,42 @@ spec:
Force instructs the controller to recreate resources
when patching fails due to an immutable field change.
type: boolean
healthCheckExprs:
description: |-
HealthCheckExprs is a list of healthcheck expressions for evaluating the
health of custom resources using Common Expression Language (CEL).
The expressions are evaluated only when Wait or HealthChecks are specified.
items:
description: CustomHealthCheck defines the health check for custom
resources.
properties:
apiVersion:
description: APIVersion of the custom resource under evaluation.
type: string
current:
description: |-
Current is the CEL expression that determines if the status
of the custom resource has reached the desired state.
type: string
failed:
description: |-
Failed is the CEL expression that determines if the status
of the custom resource has failed to reach the desired state.
type: string
inProgress:
description: |-
InProgress is the CEL expression that determines if the status
of the custom resource has not yet reached the desired state.
type: string
kind:
description: Kind of the custom resource under evaluation.
type: string
required:
- apiVersion
- current
- kind
type: object
type: array
healthChecks:
description: A list of resources to be included in the health assessment.
items:
Expand Down
32 changes: 32 additions & 0 deletions docs/api/v1/kustomize.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,22 @@ resources. When enabled, the HealthChecks are ignored. Defaults to false.</p>
<p>Components specifies relative paths to specifications of other Components.</p>
</td>
</tr>
<tr>
<td>
<code>healthCheckExprs</code><br>
<em>
<a href="https://godoc.org/github.com/fluxcd/pkg/apis/kustomize#CustomHealthCheck">
[]github.com/fluxcd/pkg/apis/kustomize.CustomHealthCheck
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>HealthCheckExprs is a list of healthcheck expressions for evaluating the
health of custom resources using Common Expression Language (CEL).
The expressions are evaluated only when Wait or HealthChecks are specified.</p>
</td>
</tr>
</table>
</td>
</tr>
Expand Down Expand Up @@ -918,6 +934,22 @@ resources. When enabled, the HealthChecks are ignored. Defaults to false.</p>
<p>Components specifies relative paths to specifications of other Components.</p>
</td>
</tr>
<tr>
<td>
<code>healthCheckExprs</code><br>
<em>
<a href="https://godoc.org/github.com/fluxcd/pkg/apis/kustomize#CustomHealthCheck">
[]github.com/fluxcd/pkg/apis/kustomize.CustomHealthCheck
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>HealthCheckExprs is a list of healthcheck expressions for evaluating the
health of custom resources using Common Expression Language (CEL).
The expressions are evaluated only when Wait or HealthChecks are specified.</p>
</td>
</tr>
</tbody>
</table>
</div>
Expand Down
60 changes: 60 additions & 0 deletions docs/spec/v1/kustomizations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,66 @@ spec:
If all the HelmRelease objects are successfully installed or upgraded, then
the Kustomization will be marked as ready.

### Health check expressions

`.spec.healthCheckExprs` can be used to define custom logic for performing
health checks on custom resources. This is done through Common Expression
Language (CEL) expressions. This field accepts a list of objects with the
following fields:

- `apiVersion`: The API version of the custom resource. Required.
- `kind`: The kind of the custom resource. Required.
- `current`: A required CEL expression that returns `true` if the resource is ready.
- `inProgress`: An optional CEL expression that returns `true` if the resource
is still being reconciled.
- `failed`: An optional CEL expression that returns `true` if the resource
failed to reconcile.

The controller will evaluate the expressions in the following order:

1. `inProgress` if specified
2. `failed` if specified
3. `current`

The first expression that evaluates to `true` will determine the health
status of the custom resource.

For example, to define a set of health check expressions for the `SealedSecret`
custom resource:

```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: sealed-secrets
namespace: flux-system
spec:
interval: 5m
path: ./path/to/sealed/secrets
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 1m
wait: true # Tells the controller to wait for all resources to be ready by performing health checks.
healthCheckExprs:
- apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
failed: status.conditions.filter(e, e.type == 'Synced').all(e, e.status == 'False')
current: status.conditions.filter(e, e.type == 'Synced').all(e, e.status == 'True')
```

A common error is writing expressions that reference fields that do not
exist in the custom resource. This will cause the controller to wait
for the resource to be ready until the timeout is reached. To avoid this,
make sure your CEL expressions are correct. The
[CEL Playground](https://playcel.undistro.io/) is a useful resource for
this task. The input passed to each expression is the custom resource
object itself.

It's worth checking if [the library](/flux/cheatsheets/cel-healthchecks/)
has expressions for the custom resources you are using.

### Wait

`.spec.wait` is an optional boolean field to perform health checks for __all__
Expand Down
Loading
Loading