Microk8s snap breaks on custom Ubuntu Core image on AirGapped environments

[gustavosr98]

Summary

Microk8s snap breaks on custom Ubuntu Core image on AirGapped environments
It fails to create SSL certificates because of the lack of an IP

Versions

I have a custom Ubuntu Core image with Microk8s pre-installed that I deploy on Air Gapped machines

snap	channel	rev
core	latest/stable	16928
core18	latest/stable	2823
core20	latest/stable	2318
core22	latest/stable	1380
snapd	latest/stable	21465
pc-kernel	22/stable	1833
microk8s	1.29-strict/stable	6529
And some other snaps

The same image works when booting on an environment with Internet
But fails when AirGapped

Reproduction Steps

Build a custom ubuntu core image with Microk8s
And boot on an airgapped environment

Logs

# On custom ubuntu core it even breaks snapd
# ubuntu@ubuntu:~$ snap list
-bash: snap: command not found

# ubuntu@ubuntu:~$ systemctl list-units | grep -v active
  UNIT                                                                                                     LOAD   ACTIVE SUB       DESCRIPTION
● snapd.failure.service                                                                                    loaded failed failed    Failure handling of the snapd snap
● snapd.seeded.service                                                                                     loaded failed failed    Wait until snapd is fully seeded
● snapd.service                                                                                            loaded failed failed    Snap Daemon
● systemd-networkd-wait-online.service                                                                     loaded failed failed    Wait for Network to be Configured

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
To show all installed unit files use 'systemctl list-unit-files'.

# ubuntu@ubuntu:~$ sudo journalctl -u snapd.service
May 22 21:58:45 ubuntu systemd[1]: Starting Snap Daemon...
May 22 21:58:49 ubuntu snapd[2075]: overlord.go:271: Acquiring state lock file
May 22 21:58:49 ubuntu snapd[2075]: overlord.go:276: Acquired state lock file
May 22 21:58:49 ubuntu snapd[2075]: patch.go:64: Patching system state level 6 to sublevel 1...
May 22 21:58:49 ubuntu snapd[2075]: patch.go:64: Patching system state level 6 to sublevel 2...
May 22 21:58:49 ubuntu snapd[2075]: patch.go:64: Patching system state level 6 to sublevel 3...
May 22 21:58:49 ubuntu snapd[2075]: daemon.go:247: started snapd/2.62 (series 16) ubuntu-core/22 (amd64) linux/5.15.0-107-generic.
May 22 21:58:49 ubuntu snapd[2075]: daemon.go:340: adjusting startup timeout by 35s (pessimistic estimate of 30s plus 5s per snap)
May 22 21:58:49 ubuntu snapd[2075]: backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
May 22 21:58:49 ubuntu snapd[2075]: devicemgr.go:340: save already mounted under /var/lib/snapd/save
May 22 21:58:49 ubuntu systemd[1]: Started Snap Daemon.
May 22 21:58:55 ubuntu snapd[2075]: devicestate.go:219: installing unasserted gadget "ipi-airgapped-gadget"
May 22 22:00:00 ubuntu snapd[2075]: taskrunner.go:299: [change 1 "Run install hook of \"microk8s\" snap if present" task] failed: run hook "install":
May 22 22:00:00 ubuntu snapd[2075]: -----
May 22 22:00:00 ubuntu snapd[2075]: + force=true
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/bin/cp /var/snap/microk8s/6529/certs/csr.conf.rendered /var/snap/microk8s/6529/certs/csr.conf
May 22 22:00:00 ubuntu snapd[2075]: + true
May 22 22:00:00 ubuntu snapd[2075]: + gen_server_cert
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper req -new -sha256 -key /var/snap/microk8s/6529/certs/server.key -out /var/snap/microk8s/6529/certs/server.csr -config /var/snap/microk8s/6529/certs/csr.conf
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper x509 -req -sha256 -in /var/snap/microk8s/6529/certs/server.csr -CA /var/snap/microk8s/6529/certs/ca.crt -CAkey /var/snap/microk8s/6529/certs/ca.key -CAcreateserial -out /var/snap/microk8s/6529/certs/server.crt -days 365 -extensions v3_ext -extfile /var/snap/microk8s/6529/certs/csr.conf
May 22 22:00:00 ubuntu snapd[2075]: Signature ok
May 22 22:00:00 ubuntu snapd[2075]: subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
May 22 22:00:00 ubuntu snapd[2075]: Getting CA Private Key
May 22 22:00:00 ubuntu snapd[2075]: + gen_proxy_client_cert
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper req -new -sha256 -key /var/snap/microk8s/6529/certs/front-proxy-client.key -out /var/snap/microk8s/6529/certs/front-proxy-client.csr -config /dev/fd/63 -subj /CN=front-proxy-client
May 22 22:00:00 ubuntu snapd[2075]: ++ sed '/^prompt = no/d' /var/snap/microk8s/6529/certs/csr.conf
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper x509 -req -sha256 -in /var/snap/microk8s/6529/certs/front-proxy-client.csr -CA /var/snap/microk8s/6529/certs/front-proxy-ca.crt -CAkey /var/snap/microk8s/6529/certs/front-proxy-ca.key -CAcreateserial -out /var/snap/microk8s/6529/certs/front-proxy-client.crt -days 365 -extensions v3_ext -extfile /var/snap/microk8s/6529/certs/csr.conf
May 22 22:00:00 ubuntu snapd[2075]: Signature ok
May 22 22:00:00 ubuntu snapd[2075]: subject=CN = front-proxy-client
May 22 22:00:00 ubuntu snapd[2075]: Getting CA Private Key
May 22 22:00:00 ubuntu snapd[2075]: + echo 1
May 22 22:00:00 ubuntu snapd[2075]: 1
May 22 22:00:00 ubuntu snapd[2075]: + rm -rf .srl
May 22 22:00:00 ubuntu snapd[2075]: + create_user_certs_and_configs
May 22 22:00:00 ubuntu snapd[2075]: + create_user_certificates
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/hostname
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/usr/bin/tr '[:upper:]' '[:lower:]'
May 22 22:00:00 ubuntu snapd[2075]: + hostname=ubuntu
May 22 22:00:00 ubuntu snapd[2075]: + generate_csr_with_sans /CN=system:node:ubuntu/O=system:nodes /var/snap/microk8s/6529/certs/kubelet.key
May 22 22:00:00 ubuntu snapd[2075]: + sign_certificate
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/cat
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/hostname
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/usr/bin/tr '[:upper:]' '[:lower:]'
May 22 22:00:00 ubuntu snapd[2075]: + hostname=ubuntu
May 22 22:00:00 ubuntu snapd[2075]: + subjectAltName=DNS:ubuntu
May 22 22:00:00 ubuntu snapd[2075]: ++ get_ips
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/hostname -I
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/sed 's/169\.254\.[0-9]\{1,3\}\.[0-9]\{1,3\}//g'
May 22 22:00:00 ubuntu snapd[2075]: ++ local IP_ADDR=
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IPS=
May 22 22:00:00 ubuntu snapd[2075]: ++ for CNI_INTERFACE in vxlan.calico flannel.1 cni0 ovn0
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/sbin/ip -o -4 addr list vxlan.calico
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/grep -v 'inet 169.254'
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/gawk '{print $4}'
May 22 22:00:00 ubuntu snapd[2075]: +++ head -1
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/cut -d/ -f1
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IP=
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IPS=//
May 22 22:00:00 ubuntu snapd[2075]: ++ for CNI_INTERFACE in vxlan.calico flannel.1 cni0 ovn0
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/sbin/ip -o -4 addr list flannel.1
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/grep -v 'inet 169.254'
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/cut -d/ -f1
May 22 22:00:00 ubuntu snapd[2075]: +++ head -1
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/gawk '{print $4}'
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IP=
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IPS=////
May 22 22:00:00 ubuntu snapd[2075]: ++ for CNI_INTERFACE in vxlan.calico flannel.1 cni0 ovn0
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/sbin/ip -o -4 addr list cni0
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/grep -v 'inet 169.254'
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/gawk '{print $4}'
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/cut -d/ -f1
May 22 22:00:00 ubuntu snapd[2075]: +++ head -1
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IP=
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IPS=//////
May 22 22:00:00 ubuntu snapd[2075]: ++ for CNI_INTERFACE in vxlan.calico flannel.1 cni0 ovn0
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/sbin/ip -o -4 addr list ovn0
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/grep -v 'inet 169.254'
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/gawk '{print $4}'
May 22 22:00:00 ubuntu snapd[2075]: +++ head -1
May 22 22:00:00 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/cut -d/ -f1
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IP=
May 22 22:00:00 ubuntu snapd[2075]: ++ CNI_IPS=////////
May 22 22:00:00 ubuntu snapd[2075]: ++ [[ -z '' ]]
May 22 22:00:00 ubuntu snapd[2075]: ++ echo none
May 22 22:00:00 ubuntu snapd[2075]: + for ip in $(get_ips)
May 22 22:00:00 ubuntu snapd[2075]: + subjectAltName='DNS:ubuntu, IP:none'
May 22 22:00:00 ubuntu snapd[2075]: + '[' '!' -f /var/snap/microk8s/6529/certs/kubelet.key ']'
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper genrsa -out /var/snap/microk8s/6529/certs/kubelet.key 2048
May 22 22:00:00 ubuntu snapd[2075]: Generating RSA private key, 2048 bit long modulus (2 primes)
May 22 22:00:00 ubuntu snapd[2075]: ......+++++
May 22 22:00:00 ubuntu snapd[2075]: ..............................................................................................+++++
May 22 22:00:00 ubuntu snapd[2075]: e is 65537 (0x010001)
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/bin/chown 0:0 /var/snap/microk8s/6529/certs/kubelet.key
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/bin/chmod 0600 /var/snap/microk8s/6529/certs/kubelet.key
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper req -new -sha256 -subj /CN=system:node:ubuntu/O=system:nodes -key /var/snap/microk8s/6529/certs/kubelet.key -addext 'subjectAltName = DNS:ubuntu, IP:none'
May 22 22:00:00 ubuntu snapd[2075]: Error Loading command line extensions
May 22 22:00:00 ubuntu snapd[2075]: 139763439187776:error:220A4076:X509 V3 routines:a2i_GENERAL_NAME:bad ip address:../crypto/x509v3/v3_alt.c:477:value=none
May 22 22:00:00 ubuntu snapd[2075]: 139763439187776:error:22098080:X509 V3 routines:X509V3_EXT_nconf:error in extension:../crypto/x509v3/v3_conf.c:47:name=subjectAltName, value=DNS:ubuntu, IP:none
May 22 22:00:00 ubuntu snapd[2075]: + csr=
May 22 22:00:00 ubuntu snapd[2075]: + extensions=
May 22 22:00:00 ubuntu snapd[2075]: ++ echo ''
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/openssl.wrapper req -text
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/grep 'X509v3 Subject Alternative Name:' -A1
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/usr/bin/tail -n 1
May 22 22:00:00 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/sed 's,IP Address:,IP:,g'
May 22 22:00:00 ubuntu snapd[2075]: unable to load X509 request
May 22 22:00:00 ubuntu snapd[2075]: 140020165695296:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: CERTIFICATE REQUEST
May 22 22:00:00 ubuntu snapd[2075]: + alt_names=
May 22 22:00:00 ubuntu snapd[2075]: + test x '!=' x
May 22 22:00:00 ubuntu snapd[2075]: + echo ''
May 22 22:00:00 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper x509 -req -sha256 -CA /var/snap/microk8s/6529/certs/ca.crt -CAkey /var/snap/microk8s/6529/certs/ca.key -CAcreateserial -days 3650 -extfile /dev/fd/63
May 22 22:00:00 ubuntu snapd[2075]: ++ echo ''
May 22 22:00:00 ubuntu snapd[2075]: 139665341499200:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: CERTIFICATE REQUEST
May 22 22:00:00 ubuntu snapd[2075]: -----
May 22 22:00:06 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-flanneld.service
May 22 22:00:06 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-k8s-dqlite.service
May 22 22:00:06 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-etcd.service
May 22 22:00:06 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-apiserver-kicker.service
May 22 22:00:06 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-cluster-agent.service
May 22 22:00:06 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-containerd.service
May 22 22:00:06 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-apiserver-proxy.service
May 22 22:00:06 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-kubelite.service
May 22 22:00:10 ubuntu snapd[2075]: services.go:1047: RemoveSnapServices - socket snap.lxd.user-daemon.unix.socket
May 22 22:00:10 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.lxd.user-daemon.service
May 22 22:00:10 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.lxd.activate.service
May 22 22:00:10 ubuntu snapd[2075]: services.go:1047: RemoveSnapServices - socket snap.lxd.daemon.unix.socket
May 22 22:00:10 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.lxd.daemon.service
May 22 22:00:15 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.mon.service
May 22 22:00:15 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.osd.service
May 22 22:00:15 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.mgr.service
May 22 22:00:15 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.daemon.service
May 22 22:00:15 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.mds.service
May 22 22:00:15 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.rgw.service
May 22 22:00:29 ubuntu snapd[2075]: main.go:151: Exiting on terminated signal.
May 22 22:00:29 ubuntu systemd[1]: Stopping Snap Daemon...
May 22 22:01:59 ubuntu systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.
May 22 22:01:59 ubuntu systemd[1]: snapd.service: Killing process 2075 (snapd) with signal SIGKILL.
May 22 22:01:59 ubuntu systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL
May 22 22:01:59 ubuntu systemd[1]: snapd.service: Failed with result 'timeout'.
May 22 22:01:59 ubuntu systemd[1]: snapd.service: Unit process 8459 (systemctl) remains running after unit stopped.
May 22 22:01:59 ubuntu systemd[1]: Stopped Snap Daemon.
May 22 22:01:59 ubuntu systemd[1]: snapd.service: Triggering OnFailure= dependencies.
May 22 22:01:59 ubuntu systemd[1]: snapd.service: Consumed 1min 35.501s CPU time.

[gustavosr98]

Also, related to this dicussion https://chat.canonical.com/canonical/pl/uwcgyfgt4tfxpcs8yag1hfmu1r

[gustavosr98]

Tried adding an initial IP with clout-init and netplan but seems to be ignored by Microk8s

# sudo grep netplan /var/log/cloud-init.log 
2024-05-23 02:13:08,927 - __init__.py[DEBUG]: Selected renderer 'netplan' from priority list: ['netplan']
2024-05-23 02:13:08,927 - network_state.py[DEBUG]: Passthrough netplan v2 config
2024-05-23 02:13:08,928 - netplan.py[DEBUG]: V2 to V2 passthrough
2024-05-23 02:13:08,929 - util.py[DEBUG]: Writing to /etc/netplan/50-cloud-init.yaml - wb: [644] 485 bytes
2024-05-23 02:13:08,930 - util.py[DEBUG]: Reading from /etc/netplan/00-snapd-config.yaml (quiet=False)
2024-05-23 02:13:08,930 - util.py[DEBUG]: Read 308 bytes from /etc/netplan/00-snapd-config.yaml
2024-05-23 02:13:08,930 - netplan.py[DEBUG]: removing known config '/etc/netplan/00-snapd-config.yaml' and derived existing files: ['/run/systemd/network/10-netplan-all-en.network', '/run/systemd/network/10-netplan-all-eth.network', '/run/systemd/generator/netplan.stamp']
2024-05-23 02:13:08,930 - subp.py[DEBUG]: Running command ['netplan', 'generate'] with allowed return codes [0] (shell=False, capture=True)
2024-05-23 02:15:10,455 - __init__.py[DEBUG]: Selected renderer 'netplan' from priority list: ['netplan']
2024-05-23 02:15:10,455 - network_state.py[DEBUG]: Passthrough netplan v2 config
2024-05-23 02:15:10,456 - netplan.py[DEBUG]: V2 to V2 passthrough
2024-05-23 02:15:10,457 - util.py[DEBUG]: Writing to /etc/netplan/50-cloud-init.yaml - wb: [644] 485 bytes
2024-05-23 02:15:10,457 - netplan.py[DEBUG]: skipping call to `netplan generate`. reason: identical netplan config
2024-05-23 02:15:10,494 - activators.py[DEBUG]: Using selected activator: <class 'cloudinit.net.activators.NetplanActivator'> from priority: ['netplan', 'eni', 'network-manager', 'networkd']
2024-05-23 02:15:10,494 - activators.py[DEBUG]: Attempting command ['netplan', 'apply'] for device all
2024-05-23 02:15:10,494 - subp.py[DEBUG]: Running command ['netplan', 'apply'] with allowed return codes [0] (shell=False, capture=True)
2024-05-23 02:15:12,055 - activators.py[WARNING]: Running ['netplan', 'apply'] resulted in stderr output: 
** (generate:897): WARNING **: 02:15:10.742: Permissions for /etc/netplan/50-cloud-init.yaml are too open. Netplan configuration should NOT be accessible by others.
Warning: The unit file, source configuration file or drop-ins of netplan-ovs-cleanup.service changed on disk. Run 'systemctl daemon-reload' to reload units.
** (process:895): WARNING **: 02:15:11.357: Permissions for /etc/netplan/50-cloud-init.yaml are too open. Netplan configuration should NOT be accessible by others.
** (process:895): WARNING **: 02:15:11.731: Permissions for /etc/netplan/50-cloud-init.yaml are too open. Netplan configuration should NOT be accessible by others.
** (process:895): WARNING **: 02:15:11.731: Permissions for /etc/netplan/50-cloud-init.yaml are too open. Netplan configuration should NOT be accessible by others.
2024-05-23 02:15:12,239 - util.py[DEBUG]: Writing to /etc/netplan/01-dummy.yaml - wb: [600] 130 bytes
2024-05-23 02:15:12,239 - util.py[DEBUG]: Changing the ownership of /etc/netplan/01-dummy.yaml to 0:0

At 02:15:12 the IP is already assinged on the machine
Snapd daemon starts at 02:15:28 but still seems to ignore the IP

#  sudo journalctl -u snapd
[..]
May 23 02:16:42 ubuntu snapd[2075]: 140103165663040:error:220A4076:X509 V3 routines:a2i_GENERAL_NAME:bad ip address:../crypto/x509v3/v3_alt.c:477:value=none
[..]

#  sudo journalctl -u snapd
May 23 02:15:28 ubuntu systemd[1]: Starting Snap Daemon...
May 23 02:15:32 ubuntu snapd[2075]: overlord.go:271: Acquiring state lock file
May 23 02:15:32 ubuntu snapd[2075]: overlord.go:276: Acquired state lock file
May 23 02:15:32 ubuntu snapd[2075]: patch.go:64: Patching system state level 6 to sublevel 1...
May 23 02:15:32 ubuntu snapd[2075]: patch.go:64: Patching system state level 6 to sublevel 2...
May 23 02:15:32 ubuntu snapd[2075]: patch.go:64: Patching system state level 6 to sublevel 3...
May 23 02:15:32 ubuntu snapd[2075]: daemon.go:247: started snapd/2.62 (series 16) ubuntu-core/22 (amd64) linux/5.15.0-107-generic.
May 23 02:15:32 ubuntu snapd[2075]: daemon.go:340: adjusting startup timeout by 35s (pessimistic estimate of 30s plus 5s per snap)
May 23 02:15:32 ubuntu snapd[2075]: backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
May 23 02:15:32 ubuntu snapd[2075]: devicemgr.go:340: save already mounted under /var/lib/snapd/save
May 23 02:15:32 ubuntu systemd[1]: Started Snap Daemon.
May 23 02:15:37 ubuntu snapd[2075]: devicestate.go:219: installing unasserted gadget "ipi-airgapped-gadget"
May 23 02:16:42 ubuntu snapd[2075]: taskrunner.go:299: [change 1 "Run install hook of \"microk8s\" snap if present" task] failed: run hook "install":
May 23 02:16:42 ubuntu snapd[2075]: -----
May 23 02:16:42 ubuntu snapd[2075]: + force=true
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/bin/cp /var/snap/microk8s/6529/certs/csr.conf.rendered /var/snap/microk8s/6529/certs/csr.conf
May 23 02:16:42 ubuntu snapd[2075]: + true
May 23 02:16:42 ubuntu snapd[2075]: + gen_server_cert
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper req -new -sha256 -key /var/snap/microk8s/6529/certs/server.key -out /var/snap/microk8s/6529/certs/server.csr -config /var/snap/microk8s/6529/certs/csr.conf
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper x509 -req -sha256 -in /var/snap/microk8s/6529/certs/server.csr -CA /var/snap/microk8s/6529/certs/ca.crt -CAkey /var/snap/microk8s/6529/certs/ca.key -CAcreateserial -out /var/snap/microk8s/6529/certs/server.crt -days 365 -extensions v3_ext -extfile /var/snap/microk8s/6529/certs/csr.conf
May 23 02:16:42 ubuntu snapd[2075]: Signature ok
May 23 02:16:42 ubuntu snapd[2075]: subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
May 23 02:16:42 ubuntu snapd[2075]: Getting CA Private Key
May 23 02:16:42 ubuntu snapd[2075]: + gen_proxy_client_cert
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper req -new -sha256 -key /var/snap/microk8s/6529/certs/front-proxy-client.key -out /var/snap/microk8s/6529/certs/front-proxy-client.csr -config /dev/fd/63 -subj /CN=front-proxy-client
May 23 02:16:42 ubuntu snapd[2075]: ++ sed '/^prompt = no/d' /var/snap/microk8s/6529/certs/csr.conf
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper x509 -req -sha256 -in /var/snap/microk8s/6529/certs/front-proxy-client.csr -CA /var/snap/microk8s/6529/certs/front-proxy-ca.crt -CAkey /var/snap/microk8s/6529/certs/front-proxy-ca.key -CAcreateserial -out /var/snap/microk8s/6529/certs/front-proxy-client.crt -days 365 -extensions v3_ext -extfile /var/snap/microk8s/6529/certs/csr.conf
May 23 02:16:42 ubuntu snapd[2075]: Signature ok
May 23 02:16:42 ubuntu snapd[2075]: subject=CN = front-proxy-client
May 23 02:16:42 ubuntu snapd[2075]: Getting CA Private Key
May 23 02:16:42 ubuntu snapd[2075]: + echo 1
May 23 02:16:42 ubuntu snapd[2075]: 1
May 23 02:16:42 ubuntu snapd[2075]: + rm -rf .srl
May 23 02:16:42 ubuntu snapd[2075]: + create_user_certs_and_configs
May 23 02:16:42 ubuntu snapd[2075]: + create_user_certificates
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/hostname
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/usr/bin/tr '[:upper:]' '[:lower:]'
May 23 02:16:42 ubuntu snapd[2075]: + hostname=ubuntu
May 23 02:16:42 ubuntu snapd[2075]: + generate_csr_with_sans /CN=system:node:ubuntu/O=system:nodes /var/snap/microk8s/6529/certs/kubelet.key
May 23 02:16:42 ubuntu snapd[2075]: + sign_certificate
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/cat
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/hostname
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/usr/bin/tr '[:upper:]' '[:lower:]'
May 23 02:16:42 ubuntu snapd[2075]: + hostname=ubuntu
May 23 02:16:42 ubuntu snapd[2075]: + subjectAltName=DNS:ubuntu
May 23 02:16:42 ubuntu snapd[2075]: ++ get_ips
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/hostname -I
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/sed 's/169\.254\.[0-9]\{1,3\}\.[0-9]\{1,3\}//g'
May 23 02:16:42 ubuntu snapd[2075]: ++ local IP_ADDR=
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IPS=
May 23 02:16:42 ubuntu snapd[2075]: ++ for CNI_INTERFACE in vxlan.calico flannel.1 cni0 ovn0
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/sbin/ip -o -4 addr list vxlan.calico
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/grep -v 'inet 169.254'
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/cut -d/ -f1
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/gawk '{print $4}'
May 23 02:16:42 ubuntu snapd[2075]: +++ head -1
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IP=
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IPS=//
May 23 02:16:42 ubuntu snapd[2075]: ++ for CNI_INTERFACE in vxlan.calico flannel.1 cni0 ovn0
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/sbin/ip -o -4 addr list flannel.1
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/grep -v 'inet 169.254'
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/cut -d/ -f1
May 23 02:16:42 ubuntu snapd[2075]: +++ head -1
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/gawk '{print $4}'
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IP=
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IPS=////
May 23 02:16:42 ubuntu snapd[2075]: ++ for CNI_INTERFACE in vxlan.calico flannel.1 cni0 ovn0
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/sbin/ip -o -4 addr list cni0
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/grep -v 'inet 169.254'
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/gawk '{print $4}'
May 23 02:16:42 ubuntu snapd[2075]: +++ head -1
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/cut -d/ -f1
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IP=
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IPS=//////
May 23 02:16:42 ubuntu snapd[2075]: ++ for CNI_INTERFACE in vxlan.calico flannel.1 cni0 ovn0
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/sbin/ip -o -4 addr list ovn0
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/bin/grep -v 'inet 169.254'
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/gawk '{print $4}'
May 23 02:16:42 ubuntu snapd[2075]: +++ /snap/microk8s/6529/usr/bin/cut -d/ -f1
May 23 02:16:42 ubuntu snapd[2075]: +++ head -1
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IP=
May 23 02:16:42 ubuntu snapd[2075]: ++ CNI_IPS=////////
May 23 02:16:42 ubuntu snapd[2075]: ++ [[ -z '' ]]
May 23 02:16:42 ubuntu snapd[2075]: ++ echo none
May 23 02:16:42 ubuntu snapd[2075]: + for ip in $(get_ips)
May 23 02:16:42 ubuntu snapd[2075]: + subjectAltName='DNS:ubuntu, IP:none'
May 23 02:16:42 ubuntu snapd[2075]: + '[' '!' -f /var/snap/microk8s/6529/certs/kubelet.key ']'
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper genrsa -out /var/snap/microk8s/6529/certs/kubelet.key 2048
May 23 02:16:42 ubuntu snapd[2075]: Generating RSA private key, 2048 bit long modulus (2 primes)
May 23 02:16:42 ubuntu snapd[2075]: ..................+++++
May 23 02:16:42 ubuntu snapd[2075]: ............................+++++
May 23 02:16:42 ubuntu snapd[2075]: e is 65537 (0x010001)
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/bin/chown 0:0 /var/snap/microk8s/6529/certs/kubelet.key
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/bin/chmod 0600 /var/snap/microk8s/6529/certs/kubelet.key
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper req -new -sha256 -subj /CN=system:node:ubuntu/O=system:nodes -key /var/snap/microk8s/6529/certs/kubelet.key -addext 'subjectAltName = DNS:ubuntu, IP:none'
May 23 02:16:42 ubuntu snapd[2075]: Error Loading command line extensions
May 23 02:16:42 ubuntu snapd[2075]: 140103165663040:error:220A4076:X509 V3 routines:a2i_GENERAL_NAME:bad ip address:../crypto/x509v3/v3_alt.c:477:value=none
May 23 02:16:42 ubuntu snapd[2075]: 140103165663040:error:22098080:X509 V3 routines:X509V3_EXT_nconf:error in extension:../crypto/x509v3/v3_conf.c:47:name=subjectAltName, value=DNS:ubuntu, IP:none
May 23 02:16:42 ubuntu snapd[2075]: + csr=
May 23 02:16:42 ubuntu snapd[2075]: + extensions=
May 23 02:16:42 ubuntu snapd[2075]: ++ echo ''
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/openssl.wrapper req -text
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/grep 'X509v3 Subject Alternative Name:' -A1
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/bin/sed 's,IP Address:,IP:,g'
May 23 02:16:42 ubuntu snapd[2075]: ++ /snap/microk8s/6529/usr/bin/tail -n 1
May 23 02:16:42 ubuntu snapd[2075]: unable to load X509 request
May 23 02:16:42 ubuntu snapd[2075]: 140243362359104:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: CERTIFICATE REQUEST
May 23 02:16:42 ubuntu snapd[2075]: + alt_names=
May 23 02:16:42 ubuntu snapd[2075]: + test x '!=' x
May 23 02:16:42 ubuntu snapd[2075]: + echo ''
May 23 02:16:42 ubuntu snapd[2075]: + /snap/microk8s/6529/openssl.wrapper x509 -req -sha256 -CA /var/snap/microk8s/6529/certs/ca.crt -CAkey /var/snap/microk8s/6529/certs/ca.key -CAcreateserial -days 3650 -extfile /dev/fd/63
May 23 02:16:42 ubuntu snapd[2075]: ++ echo ''
May 23 02:16:42 ubuntu snapd[2075]: 140512320833344:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: CERTIFICATE REQUEST
May 23 02:16:42 ubuntu snapd[2075]: -----
May 23 02:16:47 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-etcd.service
May 23 02:16:47 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-cluster-agent.service
May 23 02:16:47 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-containerd.service
May 23 02:16:47 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-flanneld.service
May 23 02:16:47 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-k8s-dqlite.service
May 23 02:16:47 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-kubelite.service
May 23 02:16:47 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-apiserver-proxy.service
May 23 02:16:47 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microk8s.daemon-apiserver-kicker.service
May 23 02:16:52 ubuntu snapd[2075]: services.go:1047: RemoveSnapServices - socket snap.lxd.user-daemon.unix.socket
May 23 02:16:52 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.lxd.user-daemon.service
May 23 02:16:52 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.lxd.activate.service
May 23 02:16:52 ubuntu snapd[2075]: services.go:1047: RemoveSnapServices - socket snap.lxd.daemon.unix.socket
May 23 02:16:52 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.lxd.daemon.service
May 23 02:16:56 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.rgw.service
May 23 02:16:56 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.mds.service
May 23 02:16:56 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.mgr.service
May 23 02:16:56 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.osd.service
May 23 02:16:56 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.daemon.service
May 23 02:16:56 ubuntu snapd[2075]: services.go:1071: RemoveSnapServices - disabling snap.microceph.mon.service
May 23 02:17:10 ubuntu snapd[2075]: main.go:151: Exiting on terminated signal.
May 23 02:17:10 ubuntu systemd[1]: Stopping Snap Daemon...
May 23 02:18:40 ubuntu systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.
May 23 02:18:40 ubuntu systemd[1]: snapd.service: Killing process 2075 (snapd) with signal SIGKILL.
May 23 02:18:40 ubuntu systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL
May 23 02:18:40 ubuntu systemd[1]: snapd.service: Failed with result 'timeout'.
May 23 02:18:40 ubuntu systemd[1]: snapd.service: Unit process 8415 (systemctl) remains running after unit stopped.
May 23 02:18:40 ubuntu systemd[1]: Stopped Snap Daemon.
May 23 02:18:40 ubuntu systemd[1]: snapd.service: Triggering OnFailure= dependencies.
May 23 02:18:40 ubuntu systemd[1]: snapd.service: Consumed 1min 34.231s CPU time.

[gustavosr98]

It seems like the issue is around these lines

https://github.com/canonical/microk8s/blob/master/microk8s-resources/actions/common/utils.sh#L575
https://github.com/canonical/microk8s/blob/master/microk8s-resources/actions/common/utils.sh#L1091

[gustavosr98]

Also tried pre-adding the certificates to /var/snap/microk8s/current/certs/* with cloud-init but cloud-init seems to run before /var/snap/<snap> gets mounted

[gustavosr98]

Workaround, have a cloud-init with a dummy IP that gets set on an early cycle of cloud-init (not on runcmd)
Something like this

#cloud-init
[..]
bootcmd:
- ip link add dev dummy1 type dummy
- ip link set dummy1 up
- ip addr add dev dummy1 192.0.2.10/32
- hostname -I

[gustavosr98]

The problem with my current approach of using a cloud-init file for this is that it comes with a burden
The need to mantain a custom gadget snap so you can provide the cloud-init file

[medicalwei]

I've also encountered the issue when preseeding microk8s in the installer image, and I can confirm the workaround above can make the snap seeding on microk8s finish.