[Feature Request] Provide option to disable default gateway creation or set handling precedence

[tz1112]

Description

In the declaration of a subnet, a gateway can be specified manually. If no value is set for the gateway parameter, the first valid IP address of the specified subnet range seems to be allocated to establish cluster-internal connectivity by default (e.g.: given CIDR 192.168.100.0/24, 192.168.100.1 is allocated unless specified otherwise). In some use cases, manual handling of the gateway is required (for example for integration of custom routing procedures or custom ovn-handling). In these cases it would be helpful to disable the allocation of the default gateway IP, for example by providing an option to set field subnet.spec.gateway to None. Alternatively, it would be helpful to set precedence to custom resources with the same IP as the gateway. Consider the following scenario: If the gateway is set to a specific IP (e.g. 192.168.100.254) and a custom resource in the network is using the same address, network traffic does never reach the custom resource (as traffic is routed to the gateway). For some custom routing procedures, traffic being forwarded to the custom resource is required.
The current workaround in our project is to handle custom routing by propagating an arbitrary gateway to all hosts in the subnet manually, nevertheless the default gateway is still created (either at subnet.spec.gateway or the default .1 address) and responds to requests. This is automatically detected by some applications, which leads to interference with our production workflow. It would help us a lot if there was an option to disable the gateway creation entirely or forward all traffic to a custom resource with the same IP address as the gateway. See the minimal example below with further explanations for details.

Who will benefit from this feature?

People with custom cluster-internal routing requirements

Anything else?

Minimum example:

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: example-subnet
spec:
  protocol: IPv4
  provider: example.default.ovn
  cidrBlock: 192.168.100.0/24
  gateway: 192.168.100.254/24   // if no gateway is set here, 192.168.100.1 is allocated as default gateway. If the gateway is set to 192.168.100.254 and we place our custom router at that IP, the ovn-controller responds instead of our custom router, breaking the workflow
  excludeIps:
  - 192.168.100.0..192.168.100.10
  namespaces:
  - example-namespace

---

apiVersion: v1
kind: Pod
metadata:
  name: host
  namespace: example-namespace
  annotations:
    ovn.kubernetes.io/ip_address: 192.168.100.222
    ovn.kubernetes.io/routes: |
          [{
            "gw": "192.168.100.254"   // curently, a custom router is placed at this address. Nevertheless, the default gateway at 192.168.100.1 is created and responds to requests
          }]
spec:
  containers:
  - name: host
    image: docker.io/library/nginx:alpine

[zhangzujian]

We can add routes in the subnet CRD like this:

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: example-subnet
spec:
  protocol: IPv4
  provider: example.default.ovn
  cidrBlock: 192.168.100.0/24
  gateway: 192.168.100.1
  excludeIps:
  - 192.168.100.0..192.168.100.10
  namespaces:
  - example-namespace
  routes:
  - dst: 0.0.0.0/0
    gw: 192.168.100.254
  - dst: 1.1.1.0/24
    gw: 192.168.100.1

Routes specified in the .spec.routes field will be automatically configured in the pods.

What do you think about this implementation?

[tz1112]

Thank you very much for your quick response.

Your suggestion is very well-suited to our specific requirements, thank you! It would make a lot of manual route tweaking obsolete in our current environment.

If I may add an additional request: we have not found a reliable way to modify a pods DNS server using manifests yet. In the pod annotations, label k8s.v1.cni.cncf.io/network-status specifies a DNS entry, but this value never gets populated. By default, the ovn.kubernetes.io/* labels do not contain a DNS entry either; adding this value manually does not seem to have any effect (which was expected, just an educated guess on our side). From my perspective, adding a DNS entry in the subnet CDR would be a very helpful addition, to support use cases where a custom DNS resolver in a subnet should be queried. I think simple substitution or addition to preexisting values in /etc/resolv.conf (or comparable config files of similar services) should be sufficient.
If there are existing approaches to this problem already which we have missed so far or if you want me to create a dedicated issue for this feature request, please let me know.

Suggested CRD addition (.spec.dns)

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: example-subnet
spec:
  protocol: IPv4
  provider: example.default.ovn
  cidrBlock: 192.168.100.0/24
  gateway: 192.168.100.1
  dns: 192.168.100.1
  excludeIps:
  - 192.168.100.0..192.168.100.10
  namespaces:
  - example-namespace
  routes:
  - dst: 0.0.0.0/0
    gw: 192.168.100.254
  - dst: 1.1.1.0/24
    gw: 192.168.100.1

Thanks again for your great support so far!

[zhangzujian]

Pod's /etc/resolv.conf is generated/controlled by kubelet. The only way I know to change a Pod's DNS config is to set its DNS policy/config.

[oilbeater]

How about use the PolicyRoutes field in the VPC spec to redirect traffic to the custom router?

[pasztorl]

hi! my setup is a bit different, but maybe connected to this topic.
the node have a public interface with networkattachmenet macvlan added which created net1 connected to the node's public interface. i can add ip with the macvlan.default.kubernetes.io/ip_address= annotation and i can add the subnet with k8s.v1.cni.cncf.io/networks=default/macvlan. my problem is that if i define the subnet i must add a spec.gateway, but this interface belongs a public physical network and that network have a gateway.
also if i add default route to the subnet spec.routes field, i can't see in the pod, the default route still remains on the default pod cidr gateway on eth0
what i want to achiveve that the pod have a public ip and can reachable from remote nodes. currently it is only reacheable from the same public network cidr.
i also tried eip and worked, but the software i need to run needs to bind to the public ip which is not in the pod with the eip case, so this is why i trying to use macvlan.

if i allow NET_ADMIN to the pod, i can drop the default route by hand and add the new default route on net1, but it would be great if i can do this without net admin using the annotations. is it possible?

[tz1112]

@oilbeater: Thank you for the reply, I am actually using this as of right now and it works great using a custom router. However, from my point of view, the creation of a subnet still induces the creation of a logical router on the subnet gateway IP (or the first valid IP of the subnet range if no dedicated gateway is specified) in OVN. My suggestion is to offer a mechanism to prevent the redundant creation of logical routers, as it may cause interference with custom routing handling.

Additionally, in most routing scenarios, pods require multiple network interfaces. Therefore, I would really like an option to include the interface in custom route definitions. See the example below (taken from documentation):

Thank you again!

apiVersion: v1
kind: Pod
metadata:
  name: custom-routes
  annotations:
    ovn.kubernetes.io/routes: |
      [{
        "dst": "192.168.0.101/24",
        "gw": "10.16.0.254",
        "interface": "net2"       // Proposal: allow explicit interface specification for routes
      }, {
        "gw": "10.16.0.254"
      }]
spec:
  containers:
  - name: nginx
    image: docker.io/library/nginx:alpine 

[oilbeater]

@tz1112 Thank you for sharing your input. I believe the proposal to add an interface for custom route definition is a valuable enhancement to the current protocol.

Regarding your need to create a logical switch without a port link to the router, I fully understand your requirement. However, implementing this would require significant changes to the current system. Since the custom route already offers a way to bypass the gateway, I recommend maintaining the current approach. If we introduce a method to remove the gateway at the subnet level, users might also expect the same capability at the namespace or pod level. Therefore, I suggest implementing this functionality through user-defined webhooks that rely on custom route annotations to meet specific needs.