Node re-registration required to update the node labels #29019
Conversation
k8s-ci-robot
added
sig/node
k8s-ci-robot
added
sig/docs
|
✔️ Deploy Preview for kubernetes-io-main-staging ready! 🔨 Explore the source changes: 40b013b 🔍 Inspect the deploy log: https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/617b85ee16e7ab000740c845 😎 Browse the preview: https://deploy-preview-29019--kubernetes-io-main-staging.netlify.app |
| As mentioned in the [Node name uniqueness](#node-name-uniqueness) section, | ||
| when Node configuration needs to be updated, it is a good practice to re-register | ||
| the node with the API server. For example, if the kubelet being restarted with | ||
| the new set of `--node-labels`, but the same Node name is used, the change will | ||
| not take an effect, as labels are being set on the Node registration. |
There was a problem hiding this comment.
Is there a reason for this limitation (eg information security)? I'm wondering how come a running kubelet isn't able to relabel the Node that represents it.
There was a problem hiding this comment.
I don't know the history. I think it is to avoid situation when pods get incorrectly-scheduled due to taints on the kubelet restart. Draining and re-adding node works around this issue
There was a problem hiding this comment.
do you think adding the reasoning in the text will be helpful for readers?
|
@sftim I added a reason why re-registration is needed. PTAL if you have time |
| the node with the API server. For example, if the kubelet being restarted with | ||
| the new set of `--node-labels`, but the same Node name is used, the change will | ||
| not take an effect, as labels are being set on the Node registration. One of | ||
| the reasons for this behavior is that changing labels on Kublet restart may |
There was a problem hiding this comment.
nit: “Kublet” should be “kubelet”.
This'll need a tech review I think.
| the node with the API server. For example, if the kubelet being restarted with | ||
| the new set of `--node-labels`, but the same Node name is used, the change will | ||
| not take an effect, as labels are being set on the Node registration. One of | ||
| the reasons for this behavior is that changing labels on Kublet restart may |
There was a problem hiding this comment.
| the reasons for this behavior is that changing labels on Kublet restart may | |
| the reasons for this behavior is that changing labels on kubelet restart may |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tengqm The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
| not take an effect, as labels are being set on the Node registration. One of | ||
| the reasons for this behavior is that changing labels on kubelet restart may | ||
| lead to situation when Pods are already scheduled on the Node, while they | ||
| have taints to the modified labels set preventing them from being on that node. | ||
| {{< /note >}} |
There was a problem hiding this comment.
One of the reasons for this behavior is that changing labels on kubelet restart may lead to situation when Pods are already scheduled on the Node, while they have taints to the modified labels set preventing them from being on that node.
I'd omit this. A full discussion probably wants to mention things like the descheduler. Better (IMO) not to touch on the topic at all.
There's probably an opportunity for anyone with the appetite for it, to improve how we manage label changes for kubelets that do stay in service.
There was a problem hiding this comment.
Is the good practice here to delete the Node and then register it again? Or to change the name that the kubelet would register as? (I wasn't sure).
There was a problem hiding this comment.
I'd omit this.
I started with this. And it raised questions. It may be helpful to explain some reasoning behind it, this is where the language "one of the reasons" come from. Do you feel strongly about removing it?
Is the good practice here to delete the Node and then register it again? Or to change the name that the kubelet would register as? (I wasn't sure).
If one desires to match the node name to machine name for easier discoverability, it would make sense to re-use the name. Different name would be easier to see the change and not confuse things like logs or metrics. So I don't think there is a clear cut best practice.
There was a problem hiding this comment.
OK, let's not remove that text. But I would want to reword it - rather than framing this as explaining the motivation, explain the problem that a cluster operator could encounter if they were unaware.
(do we have an example of how it could go wrong?)
If it's really tricky to explain, we could track a new issue about making a Task page, eg “Change the Labels That Apply to a Node”. I'm leaning towards that but not convinced it's needed.
There was a problem hiding this comment.
rephrased. Good idea!
|
This looks right to me. Thanks @SergeyKanzhelev and @tengqm |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 118fb33d26f880ab991d174fdef163dac5bea753
|
Added a note about the node labels.
/sig node
/kind cleanup