BGP Route Leaking Between SONiC switches

Unable to perform BGP route leaking between VRFs on SoNIC switches SW7 and SW8 as expected. The route leaking functionality works on other platforms (e.g., Ubuntu), but it fails on the SoNIC environment.

---------------
SONiC A (SW8):
- VRF1111:
    - Bound to VLAN1111 (IP Address: 10.10.11.1).
    - VM (VM-B) is directly connected to VLAN1111:
        - IP Address: 10.10.11.10
        - Default GW: 10.10.11.1 (GW of VLAN1111)
- VLAN1300:
    - IP Address: 10.10.2.3
    - Not bound to any VRF so it is Default VRF
- Default VRF:
    - Requires route leaking with VRF1111.
    - Requires route leaking with the default VRF of SW7.


SONiC B (SW7):
- VRF2000:
    - Bound to VLAN2000.
    - VM (VM-B) is directly connected to VLAN2000:
        - IP Address: 10.10.4.10
        - Default GW: 10.10.4.1 (GW of VLAN2000)
- VLAN1300:
    - IP Address: 10.10.2.2
    - Not bound to any VRF so it is Default VRF
- Default VRF:
     - Requires route leaking with VRF2000.
     - Requires route leaking with the default VRF of SW8.


FRR config SW8:  
```
sonic8# show running-config

router bgp 65001
 bgp router-id 1.1.1.0
 neighbor 10.10.2.2 remote-as 65002
 !
 address-family ipv4 unicast
  redistribute connected route-map map-VrfDefault
  neighbor 10.10.2.2 next-hop-self
  neighbor 10.10.2.2 soft-reconfiguration inbound
  neighbor 10.10.2.2 route-map REDISTRIBUTE_ANY_10.10.2.2 in
  neighbor 10.10.2.2 route-map REDISTRIBUTE_ANY_10.10.2.2 out
  rd vpn export 65001:10000
  rt vpn import 65001:10000 65001:10001
  rt vpn export 65001:10000
  export vpn
  import vpn
 exit-address-family
exit
!
router bgp 65001 vrf Vrf1111
 bgp router-id 1.1.1.2
 !
 address-family ipv4 unicast
  redistribute connected route-map map-Vrf1111
  rd vpn export 65001:10001
  rt vpn import 65001:10001 65001:10000
  rt vpn export 65001:10001
  export vpn
  import vpn
 exit-address-family
exit
!

route-map map-VrfDefault permit 1000
 set extcommunity rt 65001:10000
exit
!
route-map map-Vrf1111 permit 1000
 set extcommunity rt 65001:10001
exit
!
route-map REDISTRIBUTE_ANY_10.10.2.2 permit 10
 match ip address prefix-list ALLOW_ROUTES_10.10.2.2
exit

ip prefix-list ALLOW_ROUTES_10.10.2.2 seq 10 permit any
```

Route SW8:
```
sonic8# show ip route vrf all

VRF Vrf1111:
B>* 10.1.0.1/32 [20/0] is directly connected, Loopback0 (vrf default), weight 1, 01:24:27
B>* 10.10.0.0/24 [20/0] is directly connected, eth0 (vrf default), weight 1, 01:24:27
B>* 10.10.2.0/24 [20/0] is directly connected, Vlan1300 (vrf default), weight 1, 01:18:54
B>* 10.10.4.0/24 [20/0] via 10.10.2.2, Vlan1300 (vrf default), weight 1, 00:23:28
C>* 10.10.11.0/24 is directly connected, Vlan1111, 01:27:09
B>* 172.0.1.0/24 [20/0] is directly connected, Vlan2100 (vrf default), weight 1, 01:24:27

VRF default:
C>* 10.1.0.1/32 is directly connected, Loopback0, 01:27:27
C>* 10.10.0.0/24 is directly connected, eth0, 01:27:31
C>* 10.10.2.0/24 is directly connected, Vlan1300, 01:18:54
B>* 10.10.4.0/24 [20/0] via 10.10.2.2, Vlan1300, weight 1, 00:23:28
B>* 10.10.11.0/24 [20/0] is directly connected, Vrf1111 (vrf Vrf1111), weight 1,
```

BGP summary SW8:
```
sonic8# show ip bgp summary

IPv4 Unicast Summary:
BGP router identifier 1.1.1.0, local AS number 65001 VRF default vrf-id 0
BGP table version 12
RIB entries 11, using 1408 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.10.2.2       4      65002        10        10       12    0    0 00:01:25            4        6 N/A

Total number of neighbors 1
```

BGP advertise route SW8:
```
sonic8# show ip bgp neighbors 10.10.2.2 advertised-routes
BGP table version is 12, local router ID is 1.1.1.0, vrf id 0
Default local pref 100, local AS 65001
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 10.1.0.1/32      0.0.0.0                  0         32768 ?
 *> 10.10.0.0/24     0.0.0.0                  0         32768 ?
 *> 10.10.2.0/24     0.0.0.0                  0         32768 ?
 *> 10.10.4.0/24     0.0.0.0                                0 65002 ?
 *> 10.10.11.0/24    0.0.0.0                  0         32768 ?
 *> 172.0.1.0/24     0.0.0.0                  0         32768 ?

Total number of prefixes 6
```

BGP receive route SW8:
```
sonic8# show ip bgp neighbors 10.10.2.2 received-routes
BGP table version is 12, local router ID is 1.1.1.0, vrf id 0
Default local pref 100, local AS 65001
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 10.1.0.1/32      10.10.2.2                0             0 65002 ?
 *> 10.10.0.0/24     10.10.2.2                0             0 65002 ?
 *> 10.10.2.0/24     10.10.2.2                0             0 65002 ?
 *> 10.10.4.0/24     10.10.2.2                0             0 65002 ?
 *> 10.10.11.0/24    10.10.2.2                              0 65002 65001 ?
 *> 172.0.1.0/24     10.10.2.2                              0 65002 65001 ?

Total number of prefixes 6
```

FRR config SW7:
```
sonic7# show running-config

router bgp 65002
 bgp router-id 2.2.2.0
 neighbor 10.10.2.3 remote-as 65001
 !
 address-family ipv4 unicast
  network 10.10.4.0/24
  redistribute connected route-map map-VrfDefault
  neighbor 10.10.2.3 next-hop-self
  neighbor 10.10.2.3 soft-reconfiguration inbound
  neighbor 10.10.2.3 route-map REDISTRIBUTE_ANY_10.10.2.3 in
  neighbor 10.10.2.3 route-map REDISTRIBUTE_ANY_10.10.2.3 out
  rd vpn export 65002:10100
  rt vpn import 65002:10100 65002:11100
  rt vpn export 65002:10100
  export vpn
  import vpn
 exit-address-family
exit
!
router bgp 65002 vrf Vrf2000
 bgp router-id 2.2.2.3
 !
 address-family ipv4 unicast
  redistribute connected route-map map-Vrf2000
  rd vpn export 65002:11100
  rt vpn import 65002:11100 65002:10100
  rt vpn export 65002:11100
  export vpn
  import vpn
 exit-address-family
exit
 !
 address-family ipv4 unicast
  redistribute static route-map STATIC_ROUTE_FILTER
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute static route-map STATIC_ROUTE_FILTER
 exit-address-family
exit
!
route-map map-VrfDefault permit 1000
 set extcommunity rt 65002:10100
exit
!
route-map map-Vrf2000 permit 1000
 set extcommunity rt 65002:11100
exit
!
route-map REDISTRIBUTE_ANY_10.10.2.3 permit 10
 match ip address prefix-list ALLOW_ROUTES_10.10.2.3
exit
!
ip prefix-list ALLOW_ROUTES_10.10.2.3 seq 10 permit any

```

Route SW7:
```
sonic7# show ip route vrf all

VRF Vrf2000:
B>* 10.1.0.1/32 [20/0] is directly connected, Loopback0 (vrf default), weight 1, 00:26:20
B>* 10.10.0.0/24 [20/0] is directly connected, eth0 (vrf default), weight 1, 00:26:20
B>* 10.10.2.0/24 [20/0] is directly connected, Vlan1300 (vrf default), weight 1, 00:26:10
C>* 10.10.4.0/24 is directly connected, Vlan2000, 00:26:10
B>* 10.10.11.0/24 [20/0] via 10.10.2.3, Vlan1300 (vrf default), weight 1, 00:26:00
B>* 172.0.1.0/24 [20/0] via 10.10.2.3, Vlan1300 (vrf default), weight 1, 00:26:00

VRF default:
C>* 10.1.0.1/32 is directly connected, Loopback0, 00:26:25
C>* 10.10.0.0/24 is directly connected, eth0, 00:26:29
C>* 10.10.2.0/24 is directly connected, Vlan1300, 00:26:10
B>* 10.10.4.0/24 [20/0] is directly connected, Vrf2000 (vrf Vrf2000), weight 1, 00:26:10
B>* 10.10.11.0/24 [20/0] via 10.10.2.3, Vlan1300, weight 1, 00:26:00
B>* 172.0.1.0/24 [20/0] via 10.10.2.3, Vlan1300, weight 1, 00:26:00
```

BGP summary SW7:
```
sonic7# show ip bgp summary

IPv4 Unicast Summary:
BGP router identifier 2.2.2.0, local AS number 65002 VRF default vrf-id 0
BGP table version 10
RIB entries 11, using 1408 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.10.2.3       4      65001        12        12       10    0    0 00:03:06            5        6 N/A

Total number of neighbors 1
```

BGP advertise route SW7:
```
sonic7# show ip bgp neighbors 10.10.2.3 advertised-routes
BGP table version is 10, local router ID is 2.2.2.0, vrf id 0
Default local pref 100, local AS 65002
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 10.1.0.1/32      0.0.0.0                  0         32768 ?
 *> 10.10.0.0/24     0.0.0.0                  0         32768 ?
 *> 10.10.2.0/24     0.0.0.0                  0         32768 ?
 *> 10.10.4.0/24     0.0.0.0                  0         32768 ?
 *> 10.10.11.0/24    0.0.0.0                                0 65001 ?
 *> 172.0.1.0/24     0.0.0.0                                0 65001 ?

Total number of prefixes 6
```

BGP receive route SW7:
```
sonic7# show ip bgp neighbors 10.10.2.3 received-routes
BGP table version is 10, local router ID is 2.2.2.0, vrf id 0
Default local pref 100, local AS 65002
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 10.1.0.1/32      10.10.2.3                0             0 65001 ?
 *> 10.10.0.0/24     10.10.2.3                0             0 65001 ?
 *> 10.10.2.0/24     10.10.2.3                0             0 65001 ?
 *> 10.10.4.0/24     10.10.2.3                              0 65001 65002 ?
 *> 10.10.11.0/24    10.10.2.3                0             0 65001 ?
 *> 172.0.1.0/24     10.10.2.3                0             0 65001 ?

Total number of prefixes 6
```

Expected Behaviour:
VM-A, connected to VRF1111 on SW8, can successfully ping VM-B, which is connected to VRF2000 on SW7.

Actual Behaviour:
VM-A cannot ping to VM-B

```
admin@VM_A:~$ ping 10.10.4.10
PING 10.10.4.10 (10.10.4.10) 56(84) bytes of data.
```

Versions
- SONiC version
- FRR version
```
admin@sonic8:~$ show version

SONiC Software Version: SONiC.202411.0-08f108333
SONiC OS Version: 12
Distribution: Debian 12.8
Kernel: 6.1.0-22-2-amd64
Build commit: 08f108333
Build date: Mon Dec 16 10:16:54 UTC 2024
Built by: skylab@skylab

Platform: x86_64-ufispace_s8901_54xc-r0
HwSKU: UFISPACE-S8901-54XC
ASIC: broadcom
ASIC Count: 1
Serial Number: WWJ1E9VH00002P1
Model Number: S8901-54XC-2B1B
Hardware Revision: N/A
Uptime: 00:14:13 up  2:23,  2 users,  load average: 1.18, 1.28, 1.18
Date: Thu 28 Nov 2024 00:14:13

Docker images:
REPOSITORY                    TAG                  IMAGE ID       SIZE
docker-macsec                 latest               5acca967d693   344MB
docker-dhcp-relay             latest               f1b90cc98b0f   321MB
docker-syncd-brcm             202411.0-08f108333   781c74667028   753MB
docker-syncd-brcm             latest               781c74667028   753MB
docker-sonic-telemetry        202411.0-08f108333   80b3ceb16c4a   402MB
docker-sonic-telemetry        latest               80b3ceb16c4a   402MB
docker-teamd                  202411.0-08f108333   802cfac9cbd0   341MB
docker-teamd                  latest               802cfac9cbd0   341MB
docker-sonic-mgmt-framework   202411.0-08f108333   3e71df11e89b   402MB
docker-sonic-mgmt-framework   latest               3e71df11e89b   402MB
docker-snmp                   202411.0-08f108333   ec9c24f24573   356MB
docker-snmp                   latest               ec9c24f24573   356MB
docker-sflow                  202411.0-08f108333   b0b67b300b0b   342MB
docker-sflow                  latest               b0b67b300b0b   342MB
docker-router-advertiser      202411.0-08f108333   38457b809e0a   312MB
docker-router-advertiser      latest               38457b809e0a   312MB
docker-sonic-restapi          202411.0-08f108333   23cf4169fed4   330MB
docker-sonic-restapi          latest               23cf4169fed4   330MB
docker-platform-monitor       202411.0-08f108333   ee4116df3be8   431MB
docker-platform-monitor       latest               ee4116df3be8   431MB
docker-orchagent              202411.0-08f108333   09dd8f5415ba   354MB
docker-orchagent              latest               09dd8f5415ba   354MB
docker-nat                    202411.0-08f108333   548454bff351   344MB
docker-nat                    latest               548454bff351   344MB
docker-mux                    202411.0-08f108333   3f7720fa9023   363MB
docker-mux                    latest               3f7720fa9023   363MB
docker-lldp                   202411.0-08f108333   111a40a538c4   357MB
docker-lldp                   latest               111a40a538c4   357MB
docker-sonic-gnmi             202411.0-08f108333   38443c99015a   402MB
docker-sonic-gnmi             latest               38443c99015a   402MB
docker-fpm-frr                202411.0-08f108333   ddde5deb1174   375MB
docker-fpm-frr                latest               ddde5deb1174   375MB
docker-eventd                 202411.0-08f108333   7c93c33ea791   312MB
docker-eventd                 latest               7c93c33ea791   312MB
docker-database               202411.0-08f108333   232aab64e2e9   320MB
docker-database               latest               232aab64e2e9   320MB
docker-sonic-bmp              202411.0-08f108333   e263e66221c7   313MB
docker-sonic-bmp              latest               e263e66221c7   313MB
docker-gbsyncd-credo          202411.0-08f108333   9d946e23ee6a   325MB
docker-gbsyncd-credo          latest               9d946e23ee6a   325MB
docker-gbsyncd-broncos        202411.0-08f108333   6126f0496c19   352MB
docker-gbsyncd-broncos        latest               6126f0496c19   352MB
```

```
admin@sonic8:~$ vtysh

Hello, this is FRRouting (version 10.0.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
```





Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

BGP Route Leaking Between SONiC switches #40

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BGP Route Leaking Between SONiC switches #40

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions