Skip to content

Fix Istio sidecar injection by moving from annotations to labels #2527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 9, 2025
Merged

Fix Istio sidecar injection by moving from annotations to labels #2527

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d9a0803
Fix Istio sidecar injection by moving from annotations to labels
madmecodes Mar 14, 2025
dabb6ca
Update Istio sidecar injection from annotations to labels across the …
madmecodes Mar 19, 2025
d0ca2ee
fix: Update SuggestionLabels function and composer implementation for…
madmecodes Mar 20, 2025
6759bb1
Fix linting issues in mpi-job-horovod.py
madmecodes Mar 20, 2025
1b94f1e
update: function moved from annotations to labels
madmecodes May 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion conformance/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Run conformance test and generate test report.
python test/e2e/v1beta1/scripts/gh-actions/run-e2e-experiment.py --experiment-path examples/v1beta1/hp-tuning/random.yaml --namespace kf-conformance \
--trial-pod-annotations '{"sidecar.istio.io/inject": "false"}' | tee /tmp/katib-conformance.log
--trial-pod-labels '{"sidecar.istio.io/inject": "false"}' | tee /tmp/katib-conformance.log


# Create the done file.
Expand Down
2 changes: 1 addition & 1 deletion docs/proposals/1214-custom-crd-in-trial/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ SucceededCondition: Succeeded
Previously, we had problems with Istio sidecar containers,
check [kubeflow/issue#1081](https://github.com/kubeflow/kubeflow/issues/4742).
In some cases, it is unable to properly download datasets in training pod.
It was fixed by adding annotation `sidecar.istio.io/inject: false` to appropriate Trial job in Katib controller.
It was fixed by adding label `sidecar.istio.io/inject: false` to appropriate Trial job in Katib controller.

Various CRD can have unified design and it is hard to understand where annotation must be specified
to disable Istio injection for the running pods.
Expand Down
2 changes: 1 addition & 1 deletion examples/v1beta1/kubeflow-pipelines/early-stopping.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@
" \"spec\": {\n",
" \"template\": {\n",
" \"metadata\": {\n",
" \"annotations\": {\n",
" \"labels\": {\n",
" \"sidecar.istio.io/inject\": \"false\"\n",
" }\n",
" },\n",
Expand Down
10 changes: 5 additions & 5 deletions examples/v1beta1/kubeflow-pipelines/kubeflow-e2e-mnist.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@
" \"restartPolicy\": \"OnFailure\",\n",
" \"template\": {\n",
" \"metadata\": {\n",
" \"annotations\": {\n",
" \"labels\": {\n",
" \"sidecar.istio.io/inject\": \"false\"\n",
" }\n",
" },\n",
Expand All @@ -236,7 +236,7 @@
" \"restartPolicy\": \"OnFailure\",\n",
" \"template\": {\n",
" \"metadata\": {\n",
" \"annotations\": {\n",
" \"labels\": {\n",
" \"sidecar.istio.io/inject\": \"false\"\n",
" }\n",
" },\n",
Expand Down Expand Up @@ -360,7 +360,7 @@
" \"restartPolicy\": \"OnFailure\",\n",
" \"template\": {\n",
" \"metadata\": {\n",
" \"annotations\": {\n",
" \"labels\": {\n",
" \"sidecar.istio.io/inject\": \"false\"\n",
" }\n",
" },\n",
Expand Down Expand Up @@ -401,7 +401,7 @@
" \"restartPolicy\": \"OnFailure\",\n",
" \"template\": {\n",
" \"metadata\": {\n",
" \"annotations\": {\n",
" \"labels\": {\n",
" \"sidecar.istio.io/inject\": \"false\"\n",
" }\n",
" },\n",
Expand Down Expand Up @@ -600,7 +600,7 @@
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA1ElEQVR4nN3QPwtBYRQG8EMU0e0uZLIw+QKXRZlMGC0GX8CglE0pk0VxPwQmE5YrJYPVIjYMlImSwXNiMOi97319AM/6O6fzh+g/Y5hr5mrRNByseAZba4D7EnlSN8wy3uAYXJOwDEw0ohKwD9mtxehqRLQBCnZr8GPkJ/Ll79y0m37GiIjiK2AQsGMYiIbryyvjmZO20U9gAIcjTg43GhfethOROToO+En6xRUlZhnSjd+I6BY7xVIRY79w4XapR9IOSTWWYSWUqE0xlH771R7UrULefm5U2pxVCt0AAAAASUVORK5CYII=\n",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA1ElEQVR4nN3QPwtBYRQG8EMU0e0uZLIw+QKXRZlMGC0GX8CglE0pk0VxPwQmE5YrJYPVIjYMlImSwXNiMOi97319AM/6O6fzh+g/Y5hr5mrRNByseAZba4D7EnlSN8wy3uAYXJOwDEw0ohKwD9mtxehqRLQBCnZr8GPkJ/Ll79y0m37GiIjiK2AQsGMYiIbryyvjmZO20U9gAIcjTg43GhfethOROToO+En6xRUlZhnSjd+I6BY7xVIRY79w4XapR9IOSTWWYSWUqE0xlH771R7UrULefm5U2pxVCt0AAAAASUVORK5CYII=",
"text/plain": [
"<PIL.BmpImagePlugin.BmpImageFile image mode=L size=28x28 at 0x7F9A711F33A0>"
]
Expand Down
8 changes: 2 additions & 6 deletions examples/v1beta1/kubeflow-pipelines/mpi-job-horovod.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,7 @@ def horovod_mnist_hpo(
"Launcher": {
"replicas": 1,
"template": {
"metadata": {
"annotations": {"sidecar.istio.io/inject": "false"}
},
"metadata": {"labels": {"sidecar.istio.io/inject": "false"}},
"spec": {
"containers": [
{
Expand Down Expand Up @@ -141,9 +139,7 @@ def horovod_mnist_hpo(
"Worker": {
"replicas": 2,
"template": {
"metadata": {
"annotations": {"sidecar.istio.io/inject": "false"}
},
"metadata": {"labels": {"sidecar.istio.io/inject": "false"}},
"spec": {
"containers": [
{
Expand Down
2 changes: 1 addition & 1 deletion examples/v1beta1/sdk/cmaes-and-resume-policies.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@
" \"spec\": {\n",
" \"template\": {\n",
" \"metadata\": {\n",
" \"annotations\": {\n",
" \"labels\": {\n",
" \"sidecar.istio.io/inject\": \"false\"\n",
" }\n",
" },\n",
Expand Down
2 changes: 1 addition & 1 deletion examples/v1beta1/sdk/nas-with-darts.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@
" \"spec\": {\n",
" \"template\": {\n",
" \"metadata\": {\n",
" \"annotations\": {\n",
" \"labels\": {\n",
" \"sidecar.istio.io/inject\": \"false\"\n",
" }\n",
" },\n",
Expand Down
2 changes: 1 addition & 1 deletion manifests/v1beta1/components/controller/controller.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ spec:
metadata:
labels:
katib.kubeflow.org/component: controller
sidecar.istio.io/inject: "false"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
sidecar.istio.io/inject: "false"
spec:
serviceAccountName: katib-controller
containers:
Expand Down
1 change: 0 additions & 1 deletion manifests/v1beta1/components/db-manager/db-manager.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ spec:
metadata:
labels:
katib.kubeflow.org/component: db-manager
annotations:
sidecar.istio.io/inject: "false"
spec:
containers:
Expand Down
1 change: 0 additions & 1 deletion manifests/v1beta1/components/mysql/mysql.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ spec:
metadata:
labels:
katib.kubeflow.org/component: mysql
annotations:
sidecar.istio.io/inject: "false"
spec:
containers:
Expand Down
2 changes: 1 addition & 1 deletion manifests/v1beta1/components/postgres/postgres.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ spec:
metadata:
labels:
katib.kubeflow.org/component: postgres
annotations:
sidecar.istio.io/inject: "false"
annotations:
spec:
containers:
- name: katib-postgres
Expand Down
1 change: 0 additions & 1 deletion manifests/v1beta1/components/ui/ui.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ spec:
metadata:
labels:
katib.kubeflow.org/component: ui
annotations:
sidecar.istio.io/inject: "false"
spec:
containers:
Expand Down
2 changes: 1 addition & 1 deletion manifests/v1beta1/installs/katib-with-kubeflow/patches/istio-sidecar-injection.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ metadata:
spec:
template:
metadata:
annotations:
labels:
sidecar.istio.io/inject: "true"
8 changes: 4 additions & 4 deletions pkg/controller.v1beta1/consts/const.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,11 +119,11 @@ const (
// JobKindJob is the kind of the Kubernetes Job.
JobKindJob = "Job"

// AnnotationIstioSidecarInjectName is the annotation of Istio Sidecar
AnnotationIstioSidecarInjectName = "sidecar.istio.io/inject"
// LabelIstioSidecarInjectName is the label of Istio Sidecar
LabelIstioSidecarInjectName = "sidecar.istio.io/inject"

// AnnotationIstioSidecarInjectValue is the value of Istio Sidecar annotation
AnnotationIstioSidecarInjectValue = "false"
// LabelIstioSidecarInjectValue is the value of Istio Sidecar label
LabelIstioSidecarInjectValue = "false"

// LabelTrialTemplateConfigMapName is the label name for the Trial templates configMap
LabelTrialTemplateConfigMapName = "katib.kubeflow.org/component"
Expand Down
2 changes: 1 addition & 1 deletion pkg/controller.v1beta1/suggestion/composer/composer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ func (g *General) DesiredDeployment(s *suggestionsv1beta1.Suggestion) (*appsv1.D
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: util.SuggestionLabels(s),
Annotations: util.SuggestionAnnotations(s),
Annotations: s.Annotations,
},
Spec: corev1.PodSpec{
Containers: g.desiredContainers(s, suggestionConfigData, earlyStoppingConfigData),
Expand Down
12 changes: 6 additions & 6 deletions pkg/controller.v1beta1/suggestion/composer/composer_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,15 +72,15 @@ var (
}

deploymentLabels = map[string]string{
"custom-label": "test",
consts.LabelDeploymentName: suggestionName + "-" + suggestionAlgorithm,
consts.LabelExperimentName: suggestionName,
consts.LabelSuggestionName: suggestionName,
"custom-label": "test",
consts.LabelDeploymentName: suggestionName + "-" + suggestionAlgorithm,
consts.LabelExperimentName: suggestionName,
consts.LabelSuggestionName: suggestionName,
consts.LabelIstioSidecarInjectName: consts.LabelIstioSidecarInjectValue,
}

podAnnotations = map[string]string{
"custom-annotation": "test",
"sidecar.istio.io/inject": "false",
"custom-annotation": "test",
}

namespace = "kubeflow"
Expand Down
40 changes: 0 additions & 40 deletions pkg/controller.v1beta1/util/annotations.go
View file
Open in desktop

This file was deleted.

13 changes: 12 additions & 1 deletion pkg/controller.v1beta1/util/labels.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"github.com/kubeflow/katib/pkg/controller.v1beta1/consts"
)

// SuggestionLabels returns the expected suggestion labels.
// SuggestionLabels returns the expected suggestion labels.
func SuggestionLabels(instance *suggestionsv1beta1.Suggestion) map[string]string {
res := make(map[string]string)
Expand All @@ -31,7 +32,7 @@ func SuggestionLabels(instance *suggestionsv1beta1.Suggestion) map[string]string
res[consts.LabelDeploymentName] = GetSuggestionDeploymentName(instance)
res[consts.LabelExperimentName] = instance.Name
res[consts.LabelSuggestionName] = instance.Name

res[consts.LabelIstioSidecarInjectName] = consts.LabelIstioSidecarInjectValue
return res
}

Expand All @@ -45,3 +46,13 @@ func TrialLabels(instance *experimentsv1beta1.Experiment) map[string]string {

return res
}

// AppendIstioSidecarLabel adds the Istio sidecar injection label to a labels map
func AppendIstioSidecarLabel(labels map[string]string) map[string]string {
res := make(map[string]string)
for k, v := range labels {
res[k] = v
}
res[consts.LabelIstioSidecarInjectName] = consts.LabelIstioSidecarInjectValue
return res
}
2 changes: 1 addition & 1 deletion sdk/python/v1beta1/kubeflow/katib/api/katib_client_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ def generate_trial_template() -> V1beta1TrialTemplate:
"kind": "Job",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need to also update this one: https://github.com/kubeflow/trainer/blob/release-1.9/sdk/python/kubeflow/training/utils/utils.py#L259

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need to also update this one: https://github.com/kubeflow/trainer/blob/release-1.9/sdk/python/kubeflow/training/utils/utils.py#L259

Made the respective Pull Request in the trainer/release-1.9 kubeflow/trainer#2637

"spec": {
"template": {
"metadata": {"annotations": {"sidecar.istio.io/inject": "false"}},
"metadata": {"labels": {"sidecar.istio.io/inject": "false"}},
"spec": {
"containers": [
{
Expand Down
Loading