Description
Is there an existing issue for this?
- There is no existing issue for this feature
What are you currently unable to do
When creating a physical Network into the Incus cluster, it is mandatory for every Incus node to have a set up of the corresponding bridge.
incus cluster list
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS | MESSAGE |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-gwnode1 | https://172.31.0.161:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node1 | https://172.31.0.151:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node2 | https://172.31.0.152:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node3 | https://172.31.0.153:8443 | database-standby | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
incus network create provider_302 --type=physical parent=br302 --target inc1-gwnode1
incus network create provider_302 --type=physical parent=br302 --target inc1-node1
incus network create provider_302 --type=physical parent=br302 --target inc1-node2
incus network create provider_302 --type=physical parent=br302 --target inc1-node3
incus network create provider_302 --type=physical \
ovn.ingress_mode=routed \
ipv4.ovn.ranges=100.64.1.2-100.64.1.254 \
ipv6.ovn.ranges=2001:db8::2-2001:0db8::ffff:ffff:ffff:ffff \
ipv4.routes=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 \
ipv6.routes=2001:db8:1::/48 \
ipv4.gateway=100.64.1.1/24 \
ipv6.gateway=2001:db8::1/64 \
dns.nameservers=8.8.8.8
In the above scenario, when using OVN the user would like to have a centralized network node for all the North/South traffic and not distributed as Incus by design.
The reason for this requirement could be scaling where there are thousound of chassis and stretching an L2 domain over an IP Fabric becames a huge challenge.
Another reason would be number of OVS's Openflows, for every Incus network created, the OVS's Openflow flows for OVN Logical Router's NAT would be installed on all Incus nodes, causing all the nodes to deal with million of Openflows of NAT unnecessarly when dealing with high scaling clusters.
For such scaling scenario be viable for Incus, first we can use Gateway Chassis to deal with North/South traffic and centralizing this layer on a few network chassis.
Into Incus's cluster we can set a parameter for the cluster node:
incus cluster set inc1-gwnode1 scheduler.instance=manual
incus cluster role add inc1-gwnode1 ovn-chassis
incus cluster list
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS | MESSAGE |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-gwnode1 | https://172.31.0.161:8443 | ovn-chassis | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node1 | https://172.31.0.151:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node2 | https://172.31.0.152:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node3 | https://172.31.0.153:8443 | database-standby | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
For a regular OVN Logical Router, the ha chassis group will have only the Gateway chassis as members:
ovn-nbctl list logical_router_port incus-net92-lr-lrp-ext
_uuid : 5c86ed08-de63-4903-b602-527204705b46
enabled : []
external_ids : {}
gateway_chassis : []
ha_chassis_group : ed63891a-326a-4dfe-8d6f-9c4ce6880421
ipv6_prefix : []
ipv6_ra_configs : {}
mac : "00:16:3e:c4:6d:13"
name : incus-net92-lr-lrp-ext
networks : ["100.64.1.13/24", "2001:db8::5/64"]
options : {gateway_mtu="1500"}
peer : []
status : {hosting-chassis="c202f93e-a669-4f14-8db1-208901c990bf"}
ovn-nbctl ha-chassis-group-list ed63891a-326a-4dfe-8d6f-9c4ce6880421
ed63891a-326a-4dfe-8d6f-9c4ce6880421 (incus-net92)
408b3b47-5316-4333-b3be-32501ad4a9ea (c202f93e-a669-4f14-8db1-208901c990bf)
priority 2125
ovn-sbctl find chassis name=c202f93e-a669-4f14-8db1-208901c990bf
_uuid : 75ea11d5-251a-4df1-8a39-ab6d890436cf
encaps : [5e73e729-4736-488b-8951-7d8fdb0be945]
external_ids : {}
hostname : inc1-gwnode1
name : "c202f93e-a669-4f14-8db1-208901c990bf"
What do you think would need to be added
After make clear the reasons, next step would be to make optional for the user to create the physical network on all Incus nodes, when the user setup would require only for Gateway chassis.