|
| 1 | +# Create a Raspberry Pi 5 cluster |
| 2 | + |
| 3 | +## Prerequisites |
| 4 | + |
| 5 | +This guide assumes that you have a [Raspberry Pi 5] and a |
| 6 | +sufficiently large SD card of at least 32 GB. We will be using [Raspberry Pi OS] |
| 7 | +for this guide, though k0s should run just fine on other 64-bit Linux |
| 8 | +distributions for the Raspberry Pi 5 as well. Please [file a Bug] if you encounter |
| 9 | +any issues. |
| 10 | + |
| 11 | +[Raspberry Pi 5]: https://www.raspberrypi.com/products/raspberry-pi-5/ |
| 12 | +[Raspberry Pi OS]: https://www.raspberrypi.com/software/ |
| 13 | +[file a Bug]: https://github.com/k0sproject/k0s/issues/new?assignees=&labels=bug&template=BUG_REPORT.yml |
| 14 | + |
| 15 | +## Set up the system |
| 16 | + |
| 17 | +Follow the official Raspberry Pi instructions on [booting Pi OS from the USB drive](https://projects.raspberrypi.org/en/projects/install-raspberry-pi-desktop/3) or another method depending on your needs. |
| 18 | + |
| 19 | +### SSH Access |
| 20 | + |
| 21 | +1. Typically, you can enable SSH while creating the bootable drive for Raspberry Pi OS. If not, you can enable it later |
| 22 | + |
| 23 | +2. Add your public key to the server. |
| 24 | + From your host machine, run [ssh-copy-id] to copy your public SSH key to your Pi 5: |
| 25 | + |
| 26 | + ```bash |
| 27 | + ssh-copy-id -i ~/.ssh/id_rsa.pub <YOUR_USER_NAME>@<IP_ADDRESS_OF_THE_SERVER> |
| 28 | + ``` |
| 29 | + |
| 30 | + When prompted, enter the password for your user account for the Pi. Your public key should be copied at the appropriate folder on the remote Pi automatically. |
| 31 | + |
| 32 | + Note: `~/.ssh/id_rsa.pub` is the default location for the public ssh key. If your key is elsewhere, adjust accordingly. |
| 33 | + |
| 34 | +3. Verify SSH access |
| 35 | + |
| 36 | + ```shell |
| 37 | + ssh <YOUR_USER_NAME>@<IP_ADDRESS_OF_THE_SERVER> |
| 38 | + ``` |
| 39 | + |
| 40 | + If your key has a paraphrase, you’ll be prompted for it. |
| 41 | + |
| 42 | +[ssh-copy-id]: https://www.cyberciti.biz/faq/use-ssh-copy-id-with-an-openssh-server-listing-on-a-different-port/ |
| 43 | + |
| 44 | +### Enable Memory Cgroups |
| 45 | + |
| 46 | +By default, Raspberry Pi OS does not enable memory cgroups. Since k0s requires them to pass its pre-flight checks, enable them: |
| 47 | + |
| 48 | +1. Edit /boot/cmdline.txt: |
| 49 | + |
| 50 | + ```bash |
| 51 | + sudo nano /boot/cmdline.txt |
| 52 | + ``` |
| 53 | + |
| 54 | + Append (on the same single line): |
| 55 | + |
| 56 | + ```bash |
| 57 | + cgroup_enable=memory cgroup_memory=1 |
| 58 | + ``` |
| 59 | + |
| 60 | +2. Reboot |
| 61 | + |
| 62 | + ```bash |
| 63 | + sudo reboot |
| 64 | + ``` |
| 65 | + |
| 66 | +## Install k0s |
| 67 | + |
| 68 | +### Download k0s |
| 69 | + |
| 70 | +Download a [k0s release](https://github.com/k0sproject/k0s/releases/latest). For |
| 71 | +example: |
| 72 | + |
| 73 | + ```shell |
| 74 | + wget -O /tmp/k0s https://github.com/k0sproject/k0s/releases/download/v{{{ extra.k8s_version }}}+k0s.0/k0s-v{{{ extra.k8s_version }}}+k0s.0-arm64 # replace version number! |
| 75 | + sudo install /tmp/k0s /usr/local/bin/k0s |
| 76 | + ``` |
| 77 | + |
| 78 | +― or ― |
| 79 | + |
| 80 | +Use the k0s download script (as one command) to download the latest stable k0s |
| 81 | +and make it executable in `/usr/bin/k0s`. |
| 82 | + |
| 83 | + ```shell |
| 84 | + curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh |
| 85 | + ``` |
| 86 | + |
| 87 | +At this point you can run `k0s`: |
| 88 | + |
| 89 | + ```console |
| 90 | + $ sudo k0s version |
| 91 | + v{{{ extra.k8s_version }}}+k0s.0 |
| 92 | + ``` |
| 93 | + |
| 94 | +To check if k0s's [system requirements](system-requirements.md) and [external |
| 95 | +runtime dependencies](external-runtime-deps.md) are fulfilled by your current |
| 96 | +setup, you can invoke `k0s sysinfo`: |
| 97 | +
|
| 98 | + ```console |
| 99 | + ramesses-pi5@pi:~ $ sudo k0s sysinfo |
| 100 | + Total memory: 7.9 GiB (pass) |
| 101 | + File system of /var/lib/k0s: ext4 (pass) |
| 102 | + Disk space available for /var/lib/k0s: 44.3 GiB (pass) |
| 103 | + Relative disk space available for /var/lib/k0s: 79% (pass) |
| 104 | + Name resolution: localhost: [::1 127.0.0.1] (pass) |
| 105 | + Operating system: Linux (pass) |
| 106 | + Linux kernel release: 6.6.51+rpt-rpi-2712 (pass) |
| 107 | + Max. file descriptors per process: current: 1048576 / max: 1048576 (pass) |
| 108 | + AppArmor: unavailable (pass) |
| 109 | + Executable in PATH: modprobe: /usr/sbin/modprobe (pass) |
| 110 | + Executable in PATH: mount: /usr/bin/mount (pass) |
| 111 | + Executable in PATH: umount: /usr/bin/umount (pass) |
| 112 | + /proc file system: mounted (0x9fa0) (pass) |
| 113 | + Control Groups: version 2 (pass) |
| 114 | + cgroup controller "cpu": available (is a listed root controller) (pass) |
| 115 | + cgroup controller "cpuacct": available (via cpu in version 2) (pass) |
| 116 | + cgroup controller "cpuset": available (is a listed root controller) (pass) |
| 117 | + cgroup controller "memory": available (is a listed root controller) (pass) |
| 118 | + cgroup controller "devices": available (device filters attachable) (pass) |
| 119 | + cgroup controller "freezer": available (cgroup.freeze exists) (pass) |
| 120 | + cgroup controller "pids": available (is a listed root controller) (pass) |
| 121 | + cgroup controller "hugetlb": unavailable (warning) |
| 122 | + cgroup controller "blkio": available (via io in version 2) (pass) |
| 123 | + CONFIG_CGROUPS: Control Group support: built-in (pass) |
| 124 | + CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass) |
| 125 | + CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass) |
| 126 | + CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass) |
| 127 | + CONFIG_CPUSETS: Cpuset support: built-in (pass) |
| 128 | + CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass) |
| 129 | + CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass) |
| 130 | + CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: unknown (warning) |
| 131 | + CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass) |
| 132 | + CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass) |
| 133 | + CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass) |
| 134 | + CONFIG_BLK_CGROUP: Block IO controller: built-in (pass) |
| 135 | + CONFIG_NAMESPACES: Namespaces support: built-in (pass) |
| 136 | + CONFIG_UTS_NS: UTS namespace: built-in (pass) |
| 137 | + CONFIG_IPC_NS: IPC namespace: built-in (pass) |
| 138 | + CONFIG_PID_NS: PID namespace: built-in (pass) |
| 139 | + CONFIG_NET_NS: Network namespace: built-in (pass) |
| 140 | + CONFIG_NET: Networking support: built-in (pass) |
| 141 | + CONFIG_INET: TCP/IP networking: built-in (pass) |
| 142 | + CONFIG_IPV6: The IPv6 protocol: module (pass) |
| 143 | + CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass) |
| 144 | + CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass) |
| 145 | + CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass) |
| 146 | + CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass) |
| 147 | + CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass) |
| 148 | + CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass) |
| 149 | + CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass) |
| 150 | + CONFIG_NETFILTER_XT_SET: set target and match support: module (pass) |
| 151 | + CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass) |
| 152 | + CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass) |
| 153 | + CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass) |
| 154 | + CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass) |
| 155 | + CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass) |
| 156 | + CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass) |
| 157 | + CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass) |
| 158 | + CONFIG_NETFILTER_NETLINK: module (pass) |
| 159 | + CONFIG_NF_NAT: module (pass) |
| 160 | + CONFIG_IP_SET: IP set support: module (pass) |
| 161 | + CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass) |
| 162 | + CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass) |
| 163 | + CONFIG_IP_VS: IP virtual server support: module (pass) |
| 164 | + CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass) |
| 165 | + CONFIG_IP_VS_SH: Source hashing scheduling: module (pass) |
| 166 | + CONFIG_IP_VS_RR: Round-robin scheduling: module (pass) |
| 167 | + CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass) |
| 168 | + CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning) |
| 169 | + CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass) |
| 170 | + CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning) |
| 171 | + CONFIG_IP_NF_IPTABLES: IP tables support: module (pass) |
| 172 | + CONFIG_IP_NF_FILTER: Packet filtering: module (pass) |
| 173 | + CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass) |
| 174 | + CONFIG_IP_NF_NAT: iptables NAT support: module (pass) |
| 175 | + CONFIG_IP_NF_MANGLE: Packet mangling: module (pass) |
| 176 | + CONFIG_NF_DEFRAG_IPV4: module (pass) |
| 177 | + CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning) |
| 178 | + CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning) |
| 179 | + CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass) |
| 180 | + CONFIG_IP6_NF_FILTER: Packet filtering: module (pass) |
| 181 | + CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass) |
| 182 | + CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass) |
| 183 | + CONFIG_NF_DEFRAG_IPV6: module (pass) |
| 184 | + CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass) |
| 185 | + CONFIG_LLC: module (pass) |
| 186 | + CONFIG_STP: module (pass) |
| 187 | + CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass) |
| 188 | + CONFIG_PROC_FS: /proc file system support: built-in (pass) |
| 189 | + ``` |
| 190 | +
|
| 191 | +### Deploy a Node using k0s |
| 192 | +
|
| 193 | +If you want a more hands-on process for setting up your Pi 5 devices as Kubernetes nodes with the k0s binary, refer to [the guide for Pi devices](./raspberry-pi4.md). |
| 194 | +
|
| 195 | +## Deploy a Node using k0sctl |
| 196 | +
|
| 197 | +### Install k0sctl on Your Host |
| 198 | +
|
| 199 | +Follow [the k0sctl installation guide](https://github.com/k0sproject/k0sctl#installation) and install k0sctl on your host machine. |
| 200 | +
|
| 201 | +### Single node K0s cluster |
| 202 | +
|
| 203 | +For this example, we'll create a cluster.yaml that describes your known Pi 5 device and use it as a single node (controller & worker) cluster, for example: |
| 204 | + |
| 205 | + ```yaml |
| 206 | + apiVersion: k0sctl.k0sproject.io/v1beta1 |
| 207 | + kind: Cluster |
| 208 | + metadata: |
| 209 | + name: k0s-cluster |
| 210 | + user: admin |
| 211 | + spec: |
| 212 | + hosts: |
| 213 | + - ssh: |
| 214 | + address: <IP_ADDRESS_OF_THE_SERVER> |
| 215 | + user: <YOUR_USER_NAME> |
| 216 | + port: 22 |
| 217 | + keyPath: ~/.ssh/id_rsa |
| 218 | + role: controller+worker |
| 219 | + ``` |
| 220 | + |
| 221 | +### SSH agent |
| 222 | + |
| 223 | + By default, k0sctl doesn’t prompt you for passphrases, so the easiest solution is to load your key into an SSH agent before running k0sctl. Here’s how you can do it: |
| 224 | + |
| 225 | + 1. Start the SSH agent (if not already running) |
| 226 | + |
| 227 | + ```bash |
| 228 | + eval "$(ssh-agent -s)" |
| 229 | + ``` |
| 230 | + |
| 231 | + 2. Add your private key (you’ll be prompted for the passphrase) |
| 232 | + |
| 233 | + ```bash |
| 234 | + ssh-add ~/.ssh/id_rsa |
| 235 | + ``` |
| 236 | + |
| 237 | + 3. Verify the key is loaded |
| 238 | + |
| 239 | + ```bash |
| 240 | + ssh-add -l |
| 241 | + ``` |
| 242 | + |
| 243 | +### Deploy cluster |
| 244 | + |
| 245 | + 1. Apply the cluster.yaml using k0sctl on your local machine. |
| 246 | + |
| 247 | + ```console |
| 248 | + $ k0sctl apply --config cluster.yaml |
| 249 | +
|
| 250 | + ⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███ |
| 251 | + ⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███ ███ ███ |
| 252 | + ⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███ ███ ███ |
| 253 | + ⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███ ███ ███ |
| 254 | + ⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████ ███ ██████████ |
| 255 | + k0sctl v0.21.0 Copyright 2023, k0sctl authors. |
| 256 | + By continuing to use k0sctl you agree to these terms: |
| 257 | + https://k0sproject.io/licenses/eula |
| 258 | + INFO ==> Running phase: Set k0s version |
| 259 | + INFO Looking up latest stable k0s version |
| 260 | + INFO Using k0s version v{{{ extra.k8s_version }}}+k0s.0 |
| 261 | + INFO ==> Running phase: Connect to hosts |
| 262 | + INFO [ssh] 192.168.31.93:22: connected |
| 263 | + INFO ==> Running phase: Detect host operating systems |
| 264 | + INFO [ssh] 192.168.31.93:22: is running Debian GNU/Linux 12 (bookworm) |
| 265 | + INFO ==> Running phase: Acquire exclusive host lock |
| 266 | + INFO ==> Running phase: Prepare hosts |
| 267 | + INFO ==> Running phase: Gather host facts |
| 268 | + INFO [ssh] 192.168.31.93:22: using pi as hostname |
| 269 | + INFO [ssh] 192.168.31.93:22: discovered wlan0 as private interface |
| 270 | + INFO ==> Running phase: Validate hosts |
| 271 | + INFO ==> Running phase: Validate facts |
| 272 | + INFO ==> Running phase: Download k0s on hosts |
| 273 | + INFO [ssh] 192.168.31.93:22: downloading k0s v{{{ extra.k8s_version }}}+k0s.0 |
| 274 | + INFO ==> Running phase: Install k0s binaries on hosts |
| 275 | + INFO [ssh] 192.168.31.93:22: validating configuration |
| 276 | + INFO ==> Running phase: Configure k0s |
| 277 | + INFO [ssh] 192.168.31.93:22: installing new configuration |
| 278 | + INFO ==> Running phase: Initialize the k0s cluster |
| 279 | + INFO [ssh] 192.168.31.93:22: installing k0s controller |
| 280 | + INFO [ssh] 192.168.31.93:22: waiting for the k0s service to start |
| 281 | + INFO [ssh] 192.168.31.93:22: wait for kubernetes to reach ready state |
| 282 | + INFO ==> Running phase: Release exclusive host lock |
| 283 | + INFO ==> Running phase: Disconnect from hosts |
| 284 | + INFO ==> Finished in 4m14s |
| 285 | + INFO k0s cluster version v{{{ extra.k8s_version }}}+k0s.0 is now installed |
| 286 | + INFO Tip: To access the cluster you can now fetch the admin kubeconfig using: |
| 287 | + INFO k0sctl kubeconfig --config cluster.yaml |
| 288 | + ``` |
| 289 | + |
| 290 | + 2. Fetch the kubeconfig use k0sctl. |
| 291 | + |
| 292 | + ```bash |
| 293 | + k0sctl kubeconfig --config cluster.yaml > pi_cluster.kubeconfig |
| 294 | + ``` |
| 295 | + |
| 296 | + 3. Export KUBECONFIG and verify |
| 297 | + |
| 298 | + ```console |
| 299 | + $ export KUBECONFIG=pi_cluster.kubeconfig |
| 300 | + $ kubectl get nodes |
| 301 | + NAME STATUS ROLES AGE VERSION |
| 302 | + pi Ready control-plane 2m54s v{{{ extra.k8s_version }}}+k0s |
| 303 | + ``` |
| 304 | + |
| 305 | +## Tear down k0s on Pi 5 |
| 306 | + |
| 307 | +If you need to remove k0s entirely (for example, if you run into conflicts or just want a clean slate): |
| 308 | + |
| 309 | + 1. Stop existing processes |
| 310 | + |
| 311 | + ```bash |
| 312 | + sudo k0s stop |
| 313 | + ``` |
| 314 | + |
| 315 | + 2. Reset k0s |
| 316 | + |
| 317 | + ```bash |
| 318 | + sudo k0s reset |
| 319 | + ``` |
| 320 | + |
| 321 | + 3. Remove k0s binaries |
| 322 | + |
| 323 | + ```bash |
| 324 | + sudo rm -rf /usr/local/bin/k0s |
| 325 | + ``` |
| 326 | + |
| 327 | + 4. Reboot Pi 5. |
| 328 | + |
| 329 | + ```bash |
| 330 | + sudo reboot |
| 331 | + ``` |
0 commit comments