[BUG] unable to route LoadBalancer traffic to underlay pods

[patriziobassi]

Kube-OVN Version

v1.12.6

Kubernetes Version

v1.30

Operation-system/Kernel Version

"Ubuntu 22.04.4 LTS"
5.15.0-126-generic

Description

we have a setup where k8s workers have 3 different nics:

1. management nic (ens3), with the default gw, used for ssh for instance
2. podnet, used by kube-ovn for underlay traffic for pods
3. vipnet, used by metallb to announce LoadBalancer IPs and receive traffic.

when an external client connects to the LB ip, it get correctly routed to the K8s worker announcing the traffic (externalTrafficPolicy Local/Cluster).
Metallb configure ipvs to route to the underlay pod address correctly, but then the traffic gets lost.

with tcpdump we see the traffic like this:

17:26:58.994530 vipnet In IP .42144 > .6379: Flags [S], seq 1139930890, win 35840, options [mss 8960,sackOK,TS val 435725208 ecr 0,nop,wscale 9], length 0
17:26:58.994637 ens3 Out IP .11999 > .6379: Flags [S], seq 1139930890, win 35840, options [mss 8860,sackOK,TS val 435725208 ecr 0,nop,wscale 9], length 0

So the worker doesn't have a rule to route the traffic to the "podnet" interface which is the interface kube-ovn knows and uses to allocate IPs to pods and thus forwards to the ens3 interface where the worker has the default gateway: basically the traffic is leaving the worker instead of being processed internally. so it's lost.

When using the LoadBalancer with POD in the overlay network we do not get this problem because the kernel routing table already has ovn0 interface:
via 100.64.0.1 dev ovn0 src
100.64.0.0/16 dev ovn0 proto kernel scope link src 100.64.0.5

If we manually try to add a routing rule with "ip r add ....." kube-ovn-cni will immediately delete it.

How to add a "sink" in kube-ovn in order to catch with traffic with underlay networks too?

thank you

Steps To Reproduce

.

Current Behavior

.

Expected Behavior

.

[dosubot]

Kube-OVN handles routing for underlay networks by allowing the configuration of custom routing rules via pod annotations. You can configure custom routes using the util.RoutesAnnotation on a pod, which lets you specify a list of routes with a gateway and an optional destination ^[1].

In your case, it seems that the traffic is not being routed correctly to the "podnet" interface. To address this, you can try configuring custom routing rules to ensure that traffic is directed to the correct interface. Here's a general approach you can take:

1. Custom Routes via Annotations: Use pod annotations to specify custom routes that direct traffic to the "podnet" interface. This can be done by creating a JSON object with the desired routes and applying it to the pod using the util.RoutesAnnotation.

2. Logical Router Static Routes: Kube-OVN provides an API for managing logical router static routes. You can use the OVNNbClient interface to create, update, or delete static routes at the logical router level. This might help in configuring the necessary routes for your setup.

3. Avoid Manual Route Deletion: If Kube-OVN is deleting manually added routes, ensure that the routes are configured through the supported mechanisms (annotations or logical router static routes) to prevent automatic deletion.

By configuring the routes through these supported methods, you should be able to direct the LoadBalancer traffic to the correct interface and avoid the issue of traffic being lost.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[zbb88888]

did you use the "U2O" function?

[changluyi]

#4894

This pr may help you solve the underlay + metallb scenario

[patriziobassi]

did you use the "U2O" function?

Yes we do

[patriziobassi]

#4894

This pr may help you solve the underlay + metallb scenario

Thank you!
Will It be backported to 1.12 branch?

[oilbeater]

Will It be backported to 1.12 branch?

@patriziobassi This is a significant change affecting both the data plan and the control plan, so we will not backport it to the stable release.

[patriziobassi]

Will It be backported to 1.12 branch?

@patriziobassi This is a significant change affecting both the data plan and the control plan, so we will not backport it to the stable release.

@oilbeater i see. i understood the 1.13 branch was the stable one too. So can we expect this fix to land in the 1.13.x branch or is it targetting 1.14x?

thank you a lot