I changed the LAN IP address and now the api server doesn't work

[alexgleason]

I was running MicroK8s on a server which had a 192.168.1.158 IP. I later moved the server to a different LAN and changed its IP to 10.0.0.10. Afterwards, MicroK8s stopped working:

tribes@tribes-doge:~$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
tribes@tribes-doge:~$ microk8s inspect
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-control-plane-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting juju
  Inspect Juju
Inspecting kubeflow
  Inspect Kubeflow

Building the report tarball
  Report tarball is at /var/snap/microk8s/1910/inspection-report-20210131_020755.tar.gz

It seems the API server is crashing. From journalctl -u snap.microk8s.daemon-apiserver:

Jan 31 01:57:29 tribes-doge systemd[1]: snap.microk8s.daemon-apiserver.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 01:57:29 tribes-doge systemd[1]: snap.microk8s.daemon-apiserver.service: Failed with result 'exit-code'.
Jan 31 01:57:29 tribes-doge systemd[1]: snap.microk8s.daemon-apiserver.service: Scheduled restart job, restart counter is at 4.
Jan 31 01:57:29 tribes-doge systemd[1]: Stopped Service for snap application microk8s.daemon-apiserver.
Jan 31 01:57:29 tribes-doge systemd[1]: Started Service for snap application microk8s.daemon-apiserver.

(I see a lot of these logs repeated)

So I tried grep -R '192.168.1.158' /var/snap/microk8s/ and replaced every occurrence of it I could find with my new IP address, but even after rebooting the problem persists.

Any ideas?

[alexgleason]

After editing /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml and rebooting, the file reverts to its old IPs. Any way to make that permanent?

[balchua]

@alexgleason do you see /var/snap/microk8s/current/var/kubernetes/backend/info.yaml? I may contain the IP address too.

[alexgleason]

Yep I changed it there too... I eventually came to the conclusion the IPs are probably stored in dqlite, and I couldn't figure out how to query it, so I ended up just nuking the cluster and starting over. It was painful and I lost some data.

I think it would make sense for there to be a microk8s set-ip subcommand or something, but I learned my lesson about changing IPs and will never do it again.

[alexgleason]

Could someone point me in the right direction for how to query the dqlite database?

I'm running microk8s in production and worried I don't know how to recover the cluster if something were to go wrong.

Thank you :)

[balchua]

I normally use velero to backup my kubernetes resources. Instead of backing up the datastores directly.

[ss22219]

I also has this problem

[JesperBerggren]

You can query the dqlite database like this:

$ dbdir="/var/snap/microk8s/current/var/kubernetes/backend"
$ sudo /snap/microk8s/current/bin/dqlite -c ${dbdir}/cluster.crt -k ${dbdir}/cluster.key -s localhost:19001 k8s

Unfortunately I have to change the IP adresses of my Linux servers on which the cluster nodes are running.
I too have not found a way to get the cluster up and running on the new IP-addresses.
Changing IP-address in files on the filesystem does not seem to fix the problem and I can't seem to find any key/value in dqlite that holds the IP-address.

Coming to the same conclusion as @alexgleason I'm nuking our (production) cluster as well :-(
And I agree, there really should be a microk8s.set-ip option but I guess this might not have such a high priority compared to other issues.

Regards
Jesper Berggren

[joes]

I guess I will just have to nuke my development environment and start over. It is a shame that microk8s running on Multipass on Windows does not seem to give you a stable static IP-address.

[GBrawl]

If anyone is still running into this, I followed the following steps: https://microk8s.io/docs/restore-quorum
After changing the ip address in cluster.yaml and reconfiguring the changes everything seemed to be working fine

[cjchand]

@GBrawl's post got me most of the way there. I also had to find and update the IP address in these files on each node (while microk8s was restarted) with that machine's externally-facing IP:

/var/snap/microk8s/current/var/kubernetes/backend/info.yaml
/var/snap/microk8s/current/var/kubernetes/backend/localnode.yaml

Once done, I had to run sudo microk8s refresh-certs -e ca.crt on each node (Source: https://microk8s.io/docs/command-reference#heading--microk8s-refresh-certs). Failing to do this will cause kubectl to reject the cert, as it will reference the old IP.

Hope that helps!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kepakiano]

After an IP change, info.yaml and localnode.yaml both contained only localhost as IP. As @cjchand suggested I replaced the Address with the new external IP, refreshed the certs and now my one-node cluster is working again. Thank you!