Skip to content

knative serving not getting ready due to post install job storage-version-migration-serving-serving-1.17.0 #2008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
alexanderwanyoike opened this issue Mar 1, 2025 · 5 comments
Open

knative serving not getting ready due to post install job storage-version-migration-serving-serving-1.17.0 #2008

alexanderwanyoike opened this issue Mar 1, 2025 · 5 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@alexanderwanyoike
Copy link

alexanderwanyoike commented Mar 1, 2025 •
edited
Loading

Describe the bug

Kubernetes running on a GKE cluster

Server Version: v1.31.5-gke.1169000

Having issues when tring to setup knative serving via the operator

Installation was done via Helm charts with fluxcd using the following guide

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: knative-operator
  namespace: knative
spec:
  interval: 10m
  chart:
    spec:
      chart: knative-operator
      version: 1.17.3
      sourceRef:
        kind: HelmRepository
        name: knative-operator
---

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: knative-operator
  namespace: knative
spec:
  interval: 1m0s
  url: https://knative.github.io/operator

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving

Expected behavior
After installation of knative serving running

kubectl get KnativeServing knative-serving -n knative-serving

should result in a ready KnativeServing resource

Actual Behaviour

After installation of knative serving resource returns the following error KnativeServing

kubectl get KnativeServing -n knative-serving -o yaml                     

apiVersion: v1
items:
- apiVersion: operator.knative.dev/v1beta1
  kind: KnativeServing
  metadata:
    creationTimestamp: "2025-03-01T16:14:26Z"
    finalizers:
    - knativeservings.operator.knative.dev
    generation: 1
    labels:
      kustomize.toolkit.fluxcd.io/name: flux-system
      kustomize.toolkit.fluxcd.io/namespace: flux-system
    name: knative-serving
    namespace: knative-serving
    resourceVersion: "10052980"
    uid: c3c3ea44-ed6d-4e70-9a76-82e08169ea3f
  status:
    conditions:
    - lastTransitionTime: "2025-03-01T16:14:40Z"
      status: "True"
      type: DependenciesInstalled
    - lastTransitionTime: "2025-03-01T16:36:47Z"
      status: "True"
      type: DeploymentsAvailable
    - lastTransitionTime: "2025-03-01T18:05:01Z"
      message: 'Install failed with message: Job.batch "storage-version-migration-serving-serving-1.17.0"
        is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"",
        GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0,
        CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>,
        DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"storage-version-migration-serving",
        "app.kubernetes.io/component":"storage-version-migration-job", "app.kubernetes.io/name":"knative-serving",
        "app.kubernetes.io/version":"1.17.0", "batch.kubernetes.io/controller-uid":"2ec2d1ee-d0df-45d9-b84a-da158441a069",
        "batch.kubernetes.io/job-name":"storage-version-migration-serving-serving-1.17.0",
        "controller-uid":"2ec2d1ee-d0df-45d9-b84a-da158441a069", "job-name":"storage-version-migration-serving-serving-1.17.0",
        "sidecar.istio.io/inject":"false"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil),
        Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil),
        InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"migrate",
        Image:"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:c7ae2bf09e8f4236fad16705aa2ec69ab69e0f31ebd53159a6b0d734039538b4",
        Command:[]string(nil), Args:[]string{"services.serving.knative.dev", "configurations.serving.knative.dev",
        "revisions.serving.knative.dev", "routes.serving.knative.dev", "domainmappings.serving.knative.dev"},
        WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil),
        Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:1,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1", Format:"DecimalSI"},
        "ephemeral-storage":resource.Quantity{i:resource.int64Amount{value:1073741824,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"},
        "memory":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0},
        d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100,
        scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"},
        "ephemeral-storage":resource.Quantity{i:resource.int64Amount{value:1073741824,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"},
        "memory":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0},
        d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100Mi", Format:"BinarySI"}},
        Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil),
        RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil),
        VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil),
        ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil),
        TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File",
        ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc02004d500),
        Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil),
        RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc03811cac0),
        ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil),
        ServiceAccountName:"controller", AutomountServiceAccountToken:(*bool)(nil),
        NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc014e47540), ImagePullSecrets:[]core.LocalObjectReference(nil),
        Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil),
        SchedulerName:"default-scheduler", Tolerations:[]core.Toleration{core.Toleration{Key:"kubernetes.io/arch",
        Operator:"Equal", Value:"amd64", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}},
        HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil),
        PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil),
        ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil),
        Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil),
        OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}:
        field is immutable'
      reason: Error
      status: "False"
      type: InstallSucceeded
    - lastTransitionTime: "2025-03-01T18:05:01Z"
      message: 'Install failed with message: Job.batch "storage-version-migration-serving-serving-1.17.0"
        is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"",
        GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0,
        CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>,
        DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"storage-version-migration-serving",
        "app.kubernetes.io/component":"storage-version-migration-job", "app.kubernetes.io/name":"knative-serving",
        "app.kubernetes.io/version":"1.17.0", "batch.kubernetes.io/controller-uid":"2ec2d1ee-d0df-45d9-b84a-da158441a069",
        "batch.kubernetes.io/job-name":"storage-version-migration-serving-serving-1.17.0",
        "controller-uid":"2ec2d1ee-d0df-45d9-b84a-da158441a069", "job-name":"storage-version-migration-serving-serving-1.17.0",
        "sidecar.istio.io/inject":"false"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil),
        Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil),
        InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"migrate",
        Image:"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:c7ae2bf09e8f4236fad16705aa2ec69ab69e0f31ebd53159a6b0d734039538b4",
        Command:[]string(nil), Args:[]string{"services.serving.knative.dev", "configurations.serving.knative.dev",
        "revisions.serving.knative.dev", "routes.serving.knative.dev", "domainmappings.serving.knative.dev"},
        WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil),
        Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:1,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1", Format:"DecimalSI"},
        "ephemeral-storage":resource.Quantity{i:resource.int64Amount{value:1073741824,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"},
        "memory":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0},
        d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100,
        scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"},
        "ephemeral-storage":resource.Quantity{i:resource.int64Amount{value:1073741824,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"},
        "memory":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0},
        d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100Mi", Format:"BinarySI"}},
        Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil),
        RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil),
        VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil),
        ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil),
        TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File",
        ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc02004d500),
        Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil),
        RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc03811cac0),
        ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil),
        ServiceAccountName:"controller", AutomountServiceAccountToken:(*bool)(nil),
        NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc014e47540), ImagePullSecrets:[]core.LocalObjectReference(nil),
        Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil),
        SchedulerName:"default-scheduler", Tolerations:[]core.Toleration{core.Toleration{Key:"kubernetes.io/arch",
        Operator:"Equal", Value:"amd64", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}},
        HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil),
        PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil),
        ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil),
        Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil),
        OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}:
        field is immutable'
      reason: Error
      status: "False"
      type: Ready
    - lastTransitionTime: "2025-03-01T16:14:26Z"
      status: "True"
      type: VersionMigrationEligible
    manifests:
    - /var/run/ko/knative-serving/1.17.0
    - /var/run/ko/ingress/1.17/istio
    observedGeneration: 1
    version: 1.17.0
kind: List
metadata:
  resourceVersion: ""

Knative release version

knative-operator version: v1.17
knative-serving version: latest (v1.17.0)
storage-version-migration-serving-serving version: 1.17.0

Additional context

  • Networking layer set up with istio and istio installed separately
@alexanderwanyoike alexanderwanyoike added the kind/bug Categorizes issue or PR as related to a bug. label Mar 1, 2025
@alexanderwanyoike
Copy link
Author

will move this to serving as its serving specific

@alexanderwanyoike
Copy link
Author

knative/serving#15799

@alexanderwanyoike
Copy link
Author

Actually been digging through the charts and seems that this could issue with the operator will re-open

@alexanderwanyoike
Copy link
Author

For all who are struggling I just excluded the post install job not ideal but its a workaround 😢

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  version: "1.17.0"
  manifests:
  - URL: https://raw.githubusercontent.com/knative/operator/2f4a10ff2fc7c03afa449116df23c754905a9eda/cmd/operator/kodata/knative-serving/1.17.0/1-serving-crds.yaml
  - URL: https://raw.githubusercontent.com/knative/operator/2f4a10ff2fc7c03afa449116df23c754905a9eda/cmd/operator/kodata/knative-serving/1.17.0/2-serving-core.yaml
  - URL: https://raw.githubusercontent.com/knative/operator/2f4a10ff2fc7c03afa449116df23c754905a9eda/cmd/operator/kodata/knative-serving/1.17.0/3-serving-hpa.yaml

@alexanderwanyoike alexanderwanyoike mentioned this issue Mar 2, 2025
Closed
@dprotaso
Copy link
Member

cc @houshengbo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dprotaso @alexanderwanyoike