Skip to content

Commit 7cd13ef

Browse files
Michael Tripsmira
Michael Trip
authored and
smira
committed
docs: add documentation on using Multus with Talos
Short introduction into running Multus CNI. Signed-off-by: Michael Trip <[email protected]> Signed-off-by: Andrey Smirnov <[email protected]>
1 parent 4784da3 commit 7cd13ef

File tree

2 files changed

+286
-0
lines changed
  • website/content

2 files changed

+286
-0
lines changed

‎website/content/v1.7/kubernetes-guides/network/multus.md

+143
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
title: "Multus CNI"
3+
description: "A brief instruction on howto use Multus on Talos Linux"
4+
---
5+
6+
[Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods.
7+
Typically, in Kubernetes each pod only has one network interface (apart from a loopback) -- with Multus you can create a multi-homed pod that has multiple interfaces.
8+
This is accomplished by Multus acting as a "meta-plugin", a CNI plugin that can call multiple other CNI plugins.
9+
10+
## Installation
11+
12+
Multus can be deployed by simply applying the `thick` `DaemonSet` with `kubectl`.
13+
14+
```bash
15+
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml
16+
```
17+
18+
This will create a `DaemonSet` and a CRD: `NetworkAttachmentDefinition`.
19+
This can be used to specify your network configuration.
20+
21+
## Configuration
22+
23+
### Patching the `DaemonSet`
24+
25+
For Multus to properly work with Talos a change need to be made to the `DaemonSet`.
26+
Instead of of mounting the volume called `host-run-netns` on `/run/netns` it has to be mounted on `/var/run/netns`.
27+
28+
Edit the `DaemonSet` and change the volume `host-run-netns` from `/run/netns` to `/var/run/netns`.
29+
30+
```yaml
31+
...
32+
- name: host-run-netns
33+
hostPath:
34+
path: /var/run/netns/
35+
```
36+
37+
Failing to do so will leave your cluster crippled.
38+
Running pods will remain running but new pods and deployments will give you the following error in the events:
39+
40+
```text
41+
Normal Scheduled 3s default-scheduler Successfully assigned virtualmachines/samplepod to virt2
42+
Warning FailedCreatePodSandBox 3s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b" Netns:"/var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=virtualmachines;K8S_POD_NAME=samplepod;K8S_POD_INFRA_CONTAINER_ID=3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b;K8S_POD_UID=8304765e-fd7e-4968-9144-c42c53be04f4" Path:"" ERRORED: error configuring pod [virtualmachines/samplepod] networking: [virtualmachines/samplepod/8304765e-fd7e-4968-9144-c42c53be04f4:cbr0]: error adding container to network "cbr0": DelegateAdd: cannot set "" interface name to "eth0": validateIfName: no net namespace /var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293 found: failed to Statfs "/var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293": no such file or directory
43+
': StdinData: {"capabilities":{"portMappings":true},"clusterNetwork":"/host/etc/cni/net.d/10-flannel.conflist","cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","type":"multus-shim"}
44+
```
45+
46+
### Creating your `NetworkAttachmentDefinition`
47+
48+
The `NetworkAttachmentDefinition` configuration is used to define your bridge where your second pod interface needs to be attached to.
49+
50+
```yaml
51+
apiVersion: "k8s.cni.cncf.io/v1"
52+
kind: NetworkAttachmentDefinition
53+
metadata:
54+
name: macvlan-conf
55+
spec:
56+
config: '{
57+
"cniVersion": "0.3.0",
58+
"type": "macvlan",
59+
"master": "eth0",
60+
"mode": "bridge",
61+
"ipam": {
62+
"type": "host-local",
63+
"subnet": "192.168.1.0/24",
64+
"rangeStart": "192.168.1.200",
65+
"rangeEnd": "192.168.1.216",
66+
"routes": [
67+
{ "dst": "0.0.0.0/0" }
68+
],
69+
"gateway": "192.168.1.1"
70+
}
71+
}'
72+
```
73+
74+
In this example `macvlan` is used as a bridge type.
75+
There are 3 types of bridges: `bridge`, `macvlan` and `ipvlan`:
76+
77+
1. `bridge` is a way to connect two Ethernet segments together in a protocol-independent way.
78+
Packets are forwarded based on Ethernet address, rather than IP address (like a router).
79+
Since forwarding is done at Layer 2, all protocols can go transparently through a bridge.
80+
In terms of containers or virtual machines, a bridge can also be used to connect the virtual interfaces of each container/VM to the host network, allowing them to communicate.
81+
82+
2. `macvlan` is a driver that makes it possible to create virtual network interfaces that appear as distinct physical devices each with unique MAC addresses.
83+
The underlying interface can route traffic to each of these virtual interfaces separately, as if they were separate physical devices.
84+
This means that each macvlan interface can have its own IP subnet and routing.
85+
Macvlan interfaces are ideal for situations where containers or virtual machines require the same network access as the host system.
86+
87+
3. `ipvlan` is similar to `macvlan`, with the key difference being that ipvlan shares the parent's MAC address, which requires less configuration from the networking equipment.
88+
This makes deployments simpler in certain situations where MAC address control or limits are in place.
89+
It offers two operational modes: L2 mode (the default) where it behaves similarly to a MACVLAN, and L3 mode for routing based traffic isolation (rather than bridged).
90+
91+
When using the `bridge` interface you must also configure a bridge on your Talos nodes.
92+
That can be done by updating Talos Linux machine configuration:
93+
94+
```yaml
95+
machine:
96+
interfaces:
97+
- interface: br0
98+
addresses:
99+
- 172.16.1.60/24
100+
bridge:
101+
stp:
102+
enabled: true
103+
interfaces:
104+
- eno1 # This must be changed to your matching interface name
105+
routes:
106+
- network: 0.0.0.0/0 # The route's network (destination).
107+
gateway: 172.16.1.254 # The route's gateway (if empty, creates link scope route).
108+
metric: 1024 # The optional metric for the route.
109+
```
110+
111+
More information about the configuration of bridges can be found [here](https://github.com/k8snetworkplumbingwg/multus-cni/tree/master/docs)
112+
113+
## Attaching the `NetworkAttachmentDefinition` to your `Pod` or `Deployment`
114+
115+
After the `NetworkAttachmentDefinition` is configured, you can attach that interface to your your `Deployment` or `Pod`.
116+
In this example we use a pod:
117+
118+
```yaml
119+
apiVersion: v1
120+
kind: Pod
121+
metadata:
122+
name: samplepod
123+
annotations:
124+
k8s.v1.cni.cncf.io/networks: macvlan-conf
125+
spec:
126+
containers:
127+
- name: samplepod
128+
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
129+
image: alpine
130+
```
131+
132+
## Notes on using KubeVirt in combination with Multus
133+
134+
If you would like to use KubeVirt and expose your virtual machine to the outside world with Multus, make sure to configure a `bridge` instead of `macvlan` or `ipvlan`, because that doesn't work, according to the KubeVirt [Documentation](https://kubevirt.io/user-guide/virtual_machines/interfaces_and_networks/#invalid-cnis-for-secondary-networks).
135+
136+
> Invalid CNIs for secondary networks
137+
> The following list of CNIs is known not to work for bridge interfaces - which are most common for secondary interfaces.
138+
>
139+
> * macvlan
140+
> * ipvlan
141+
>
142+
> The reason is similar: the bridge interface type moves the pod interface MAC address to the VM, leaving the pod interface with a different address.
143+
> The aforementioned CNIs require the pod interface to have the original MAC address.

‎website/content/v1.8/kubernetes-guides/network/multus.md

+143
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
title: "Multus CNI"
3+
description: "A brief instruction on howto use Multus on Talos Linux"
4+
---
5+
6+
[Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods.
7+
Typically, in Kubernetes each pod only has one network interface (apart from a loopback) -- with Multus you can create a multi-homed pod that has multiple interfaces.
8+
This is accomplished by Multus acting as a "meta-plugin", a CNI plugin that can call multiple other CNI plugins.
9+
10+
## Installation
11+
12+
Multus can be deployed by simply applying the `thick` `DaemonSet` with `kubectl`.
13+
14+
```bash
15+
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml
16+
```
17+
18+
This will create a `DaemonSet` and a CRD: `NetworkAttachmentDefinition`.
19+
This can be used to specify your network configuration.
20+
21+
## Configuration
22+
23+
### Patching the `DaemonSet`
24+
25+
For Multus to properly work with Talos a change need to be made to the `DaemonSet`.
26+
Instead of of mounting the volume called `host-run-netns` on `/run/netns` it has to be mounted on `/var/run/netns`.
27+
28+
Edit the `DaemonSet` and change the volume `host-run-netns` from `/run/netns` to `/var/run/netns`.
29+
30+
```yaml
31+
...
32+
- name: host-run-netns
33+
hostPath:
34+
path: /var/run/netns/
35+
```
36+
37+
Failing to do so will leave your cluster crippled.
38+
Running pods will remain running but new pods and deployments will give you the following error in the events:
39+
40+
```text
41+
Normal Scheduled 3s default-scheduler Successfully assigned virtualmachines/samplepod to virt2
42+
Warning FailedCreatePodSandBox 3s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b" Netns:"/var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=virtualmachines;K8S_POD_NAME=samplepod;K8S_POD_INFRA_CONTAINER_ID=3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b;K8S_POD_UID=8304765e-fd7e-4968-9144-c42c53be04f4" Path:"" ERRORED: error configuring pod [virtualmachines/samplepod] networking: [virtualmachines/samplepod/8304765e-fd7e-4968-9144-c42c53be04f4:cbr0]: error adding container to network "cbr0": DelegateAdd: cannot set "" interface name to "eth0": validateIfName: no net namespace /var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293 found: failed to Statfs "/var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293": no such file or directory
43+
': StdinData: {"capabilities":{"portMappings":true},"clusterNetwork":"/host/etc/cni/net.d/10-flannel.conflist","cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","type":"multus-shim"}
44+
```
45+
46+
### Creating your `NetworkAttachmentDefinition`
47+
48+
The `NetworkAttachmentDefinition` configuration is used to define your bridge where your second pod interface needs to be attached to.
49+
50+
```yaml
51+
apiVersion: "k8s.cni.cncf.io/v1"
52+
kind: NetworkAttachmentDefinition
53+
metadata:
54+
name: macvlan-conf
55+
spec:
56+
config: '{
57+
"cniVersion": "0.3.0",
58+
"type": "macvlan",
59+
"master": "eth0",
60+
"mode": "bridge",
61+
"ipam": {
62+
"type": "host-local",
63+
"subnet": "192.168.1.0/24",
64+
"rangeStart": "192.168.1.200",
65+
"rangeEnd": "192.168.1.216",
66+
"routes": [
67+
{ "dst": "0.0.0.0/0" }
68+
],
69+
"gateway": "192.168.1.1"
70+
}
71+
}'
72+
```
73+
74+
In this example `macvlan` is used as a bridge type.
75+
There are 3 types of bridges: `bridge`, `macvlan` and `ipvlan`:
76+
77+
1. `bridge` is a way to connect two Ethernet segments together in a protocol-independent way.
78+
Packets are forwarded based on Ethernet address, rather than IP address (like a router).
79+
Since forwarding is done at Layer 2, all protocols can go transparently through a bridge.
80+
In terms of containers or virtual machines, a bridge can also be used to connect the virtual interfaces of each container/VM to the host network, allowing them to communicate.
81+
82+
2. `macvlan` is a driver that makes it possible to create virtual network interfaces that appear as distinct physical devices each with unique MAC addresses.
83+
The underlying interface can route traffic to each of these virtual interfaces separately, as if they were separate physical devices.
84+
This means that each macvlan interface can have its own IP subnet and routing.
85+
Macvlan interfaces are ideal for situations where containers or virtual machines require the same network access as the host system.
86+
87+
3. `ipvlan` is similar to `macvlan`, with the key difference being that ipvlan shares the parent's MAC address, which requires less configuration from the networking equipment.
88+
This makes deployments simpler in certain situations where MAC address control or limits are in place.
89+
It offers two operational modes: L2 mode (the default) where it behaves similarly to a MACVLAN, and L3 mode for routing based traffic isolation (rather than bridged).
90+
91+
When using the `bridge` interface you must also configure a bridge on your Talos nodes.
92+
That can be done by updating Talos Linux machine configuration:
93+
94+
```yaml
95+
machine:
96+
interfaces:
97+
- interface: br0
98+
addresses:
99+
- 172.16.1.60/24
100+
bridge:
101+
stp:
102+
enabled: true
103+
interfaces:
104+
- eno1 # This must be changed to your matching interface name
105+
routes:
106+
- network: 0.0.0.0/0 # The route's network (destination).
107+
gateway: 172.16.1.254 # The route's gateway (if empty, creates link scope route).
108+
metric: 1024 # The optional metric for the route.
109+
```
110+
111+
More information about the configuration of bridges can be found [here](https://github.com/k8snetworkplumbingwg/multus-cni/tree/master/docs)
112+
113+
## Attaching the `NetworkAttachmentDefinition` to your `Pod` or `Deployment`
114+
115+
After the `NetworkAttachmentDefinition` is configured, you can attach that interface to your your `Deployment` or `Pod`.
116+
In this example we use a pod:
117+
118+
```yaml
119+
apiVersion: v1
120+
kind: Pod
121+
metadata:
122+
name: samplepod
123+
annotations:
124+
k8s.v1.cni.cncf.io/networks: macvlan-conf
125+
spec:
126+
containers:
127+
- name: samplepod
128+
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
129+
image: alpine
130+
```
131+
132+
## Notes on using KubeVirt in combination with Multus
133+
134+
If you would like to use KubeVirt and expose your virtual machine to the outside world with Multus, make sure to configure a `bridge` instead of `macvlan` or `ipvlan`, because that doesn't work, according to the KubeVirt [Documentation](https://kubevirt.io/user-guide/virtual_machines/interfaces_and_networks/#invalid-cnis-for-secondary-networks).
135+
136+
> Invalid CNIs for secondary networks
137+
> The following list of CNIs is known not to work for bridge interfaces - which are most common for secondary interfaces.
138+
>
139+
> * macvlan
140+
> * ipvlan
141+
>
142+
> The reason is similar: the bridge interface type moves the pod interface MAC address to the VM, leaving the pod interface with a different address.
143+
> The aforementioned CNIs require the pod interface to have the original MAC address.

0 commit comments

Comments
 (0)