FAQs and Behavior description for Config Connector and GCP labels

[yuwenma]

Checklist

• I did not find a related open enhancement request.
• I understand that enhancement requests filed in the GitHub repository are by default low priority.
• If this request is time-sensitive, I have submitted a corresponding issue with GCP support.

Describe the feature or resource

Overview

GCP labels helps you to group the GCP resources for a variety of use cases.

Config Connector supports the GCP labels in two ways:

• Kubernetes labels (.metadata.labels)

• Kubernetes Specification (.spec.labels)

• For both methods, Config Connector also adds a managed-by-cnrm label to the GCP resource.

FAQs and troubleshooting

1. Why are my Kubernetes labels not added to some GCP resources?

Reason 1: Every Config Connector resource has the Kubernetes labels (the .metadata.labels field), but not every GCP resources support labels. When you use kustomize, helm or any others GitOps tools to add Kubernetes labels to your Config Connector resources, it will only add to the GCP resource if 1.) the GCP server supports labels, and 2.) the Config Connector controller pass the label to the server.

Reason 2: Some Config Connector resources only support labels via the Specification (.spec.labels), some only support via Kubernetes labels, and some support both with a condition (migration with API backward compatibility).

2. Why does some labels fail in Config Connector , but work well in Cloud SDK and Cloud Console?

Kubernetes label has a different format requirements than the GCP label format. For example, if you has dot (.) in the key or value, it is valid in K8s but not in GCP.

Note: Some Config Connector resources filter out the labels that contain dot before applied to the GCP labels. But this also cause some confusions (see below).

3. Why does the same label can only apply to some GCP resources but not others?

Just like not every Google service support labels, some service support labels with a different format requirements. You may hit different problems depending on the actual service requirements under the hood.

4. Why does Config Connector have both Kubernetes labels and Specification labels? It is so confusing ...

Config Connector attempts to provide users a coherent experience between Kubernetes labels and GCP labels. So that you can apply the Kubernetes convention about resource grouping to your GCP resources, and use the Kubernetes tools like kubectl, kustomize to manage the labels.

However, due to the GCP differentiations on label availability and supportability, it confuses Config Connector customers, and sometimes even block the normal use of Kubernetes labels. For example, a valid Kubernetes labels cannot be applied to the CR object because it is also treated as GCP labels and break that GCP service label rules. In other cases, the Kubernetes labels are silently filtered out even if it is expected to apply to a specific GCP service.

Therefore, Config Connector introduces the spec.labels which uses the Kubernetes Specification to reflect the real user intent.

Best practices

• If you haven't encountered any of the above problems, there's no reason to change.

• If you only care about your Kubernetes labels, you should use the Config Connector metadata labels and expect it should always work.

• If you want to use Kubernetes tools like (kubectl, kustomize, helm or others) to manage you GCP labels, you can use the Kubernetes labels as the default option. But you may hit the above problems.

• If you hit any of the problems as described above, you can consider switching to the Specification approach.

  • Move the labels you want to apply to GCP from .metadata.labels to .spec.labels, and add alpha.cnrm.cloud.google.com/reconciler: "direct" annotation.
  • If the resource does not have spec.labels field, please file a GitHub issue and we will take care of the rest.

Additional information

No response

Importance

No response

[anhdle-sso]

Here are some example for SpannerInstance resource.
Example labels with alpha.cnrm.cloud.google.com/reconciler: "direct" annotation. The labels that are specific to Kubernetes are in .metadata.labels. The labels for GCP object are in .spec.labels.

apiVersion: spanner.cnrm.cloud.google.com/v1beta1
kind: SpannerInstance
metadata:
  name: spannerinstance-sample-${uniqueId}
  labels:
    app.kubernetes.io/managed-by: configmanagement.gke.io
  annotations:
    alpha.cnrm.cloud.google.com/reconciler: "direct"
spec:
  config: regional-us-west1
  displayName: Instance Sample
  labels:
    label-one: "value-one"
  autoscalingConfig:
    autoscalingLimits:
      maxNodes: 4
      minNodes: 2
    autoscalingTargets:
      highPriorityCpuUtilizationPercent: 50
      storageUtilizationPercent: 50
  edition: ENTERPRISE
  defaultBackupScheduleType: NONE

Example labels without alpha.cnrm.cloud.google.com/reconciler: "direct" annotation. The labels for both Kubernetes and GCP are in .metadata.labels. The labels in .spec.labels are ignored for SpannerInstance; this might not be true for other resources.

apiVersion: spanner.cnrm.cloud.google.com/v1beta1
kind: SpannerInstance
metadata:
  name: spannerinstance-sample-${uniqueId}
  labels:
    app.kubernetes.io/managed-by: configmanagement.gke.io
    label-one: "value-one"
spec:
  config: regional-us-west1
  displayName: Instance Sample
  autoscalingConfig:
    autoscalingLimits:
      maxNodes: 4
      minNodes: 2
    autoscalingTargets:
      highPriorityCpuUtilizationPercent: 50
      storageUtilizationPercent: 50
  edition: ENTERPRISE
  defaultBackupScheduleType: NONE

[gemmahou]

Seems like system labels are removed here during TF and DCL based controller reconciliation. So we don't see issues before.