Move Control Planes taint to kubelet config instead of markcontrolplane phase

[yagonobre]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

What happened?

Due we apply the control plane taint after the control plane comes up, on a multi-control plane case we can have pod scheduled to this control plane.

What you expected to happen?

Use the kubelet --register-with-taints config instead of handler it on a separate phase.

How to reproduce it (as minimally and precisely as possible)?

1. Kubeadm init
2. Create a daemonset
3. Join another control plane

Anything else we need to know?

For now, I just use this config but would be nice if kubeadm can handler it.

apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
nodeRegistration:
  # Remove the default control-plane taint so we taint it manually with KUBELET_EXTRA_ARGS
  taints: []
```

[neolit123]

i guess this can be a real problem. thanks for the report @yagonobre

does the kubelet allow self-taints with node roles such as node-role.kubernetes.io/master=:NoSchedule?
(it certainly does not for labels kubernetes/kubernetes#45361)

[yagonobre]

does the kubelet allow self-taints with node roles such as node-role.kubernetes.io/master=:NoSchedule?

Yes, probably we can keep the phase to add the label

[madhukar32]

@neolit123 : Can I take a crack at this?

[neolit123]

@madhukar32
hi, this needs discussion before sending the PR.
if we remove the taint from markcontrolplane phase it will break existing users.

[yagonobre]

I was away, but now I'll have enough time to do this.
@neolit123 what you think about add the taint to the kubelet config, and keep it on the markcontrolplane phase just for backward compatibility?

[neolit123]

we might have to keep it in both places with a deprecation notice.