Skip to content

Commit b988188

Browse files
Merge pull request sonic-net#260 from BRCM-SONIC/amulya_sviscale
Amulya sviscale
2 parents a8fc0c2 + 502a2d5 commit b988188

File tree

1 file changed

+83
-37
lines changed

1 file changed

+83
-37
lines changed

L3/L3_performance_and_scaling_enhancements/L3_Vlan_interface_scaling.md

+83-37
Original file line numberDiff line numberDiff line change
@@ -43,15 +43,17 @@ SONIC currently supports a maximum of 1k L3 Vlan interfaces on Broadcom hardware
4343

4444
## 1.1 Functional Requirements
4545
Scaling improvements
46-
|--------------------------------------|------|-----|-----|-----|
47-
| MAC | TD3 | TD4 | TH | TH2 |
48-
|--------------------------------------|------|-----|-----|-----|
49-
|System MAC configured SVI | 4k | 4k | 4k | 4k |
50-
|MCLAG Gateway MAC with same ip SVI | 4k | 4k | 4k | 4k |
51-
|MCLAG with unique ip SVI | 4k | 4k | 1k | 1k |
52-
|SAG MAC configured SVI | 4k | 4k | 1k | 1k |
53-
|VRRP MAC SVI | 4k | 4k | 1k | 1k |
54-
|--------------------------------------|------|-----|-----|-----|
46+
|--------------------------------------|------|-----|-----|-----|-----|
47+
| MAC | TD3 | TD4 | TH | TH2 | TH3 |
48+
|--------------------------------------|------|-----|-----|-----|-----|
49+
|System MAC configured SVI | 4k | 4k | 4k | 4k | 4k |
50+
|MCLAG Gateway MAC with same ip SVI | 4k | 4k | 4k | 4k | 4k |
51+
|MCLAG with unique ip SVI | 4k | 4k | 1k | 1k | 1k |
52+
|MCLAG with peer gateway enabled SVI | 4k | 4k | 1k | 1k | 1k |
53+
|SAG MAC configured SVI | 4k | 4k | 1k | 1k | 1k |
54+
|VRRP MAC SVI | 4k | 4k | 1k | 1k | 1k |
55+
|--------------------------------------|------|-----|-----|-----|-----|
56+
5557

5658
## 1.2 Configuration and Management Requirements
5759
No new configuration or show commands introduced.
@@ -60,17 +62,21 @@ No new configuration or show commands introduced.
6062
Covered in Functional requirements
6163

6264
## 1.4 Warm Boot Requirements
63-
Upgrade to higher version should take care of installing one entry per MAC and delete the individual station tcam entries per SVI on TH/TH2 platforms
65+
66+
Upgrade to higher version should take care of installing one entry per MAC and delete the individual station tcam entries per SVI on TH/TH2/TH3 platforms
6467
Upgrade to higher version should take care of moving to l2 table from myStationTcam for all SVI interfaces on TD3 platform.
6568

6669
# 2 Design
6770

6871
## 2.1 Use L2 table for inner mac termination for TD3 & TD4
6972
TD4 already uses L2 table for inner mac termination. No change for TD4 platform.
7073
In TD3, for all SVI interfaces, L2 table will be programmed instead of MyStationTcam.
71-
Entry will be added with respective mac,vfi and flags as BCM_L2_STATIC. One entry per each SVI will be programmed.If 4k VRRP SVI interfaces are configured ,8k L2 table entries will be consumed.Rest of the cases,4k L2 tables entries will be consumed.
74+
75+
Entry will be added with respective mac,vfi and flags as BCM_L2_STATIC. One entry per each SVI will be programmed.
76+
Note :- If 4k VRRP SVI interfaces are configured ,8k L2 table entries will be consumed.Rest of the cases 4k L2 table entries are consumed.
7277

7378
## 2.2 MyStationTcam lookup key modification
79+
This is applicable for only TH/TH2/TH3 platforms.
7480
Use only MAC as the key for MyStationTcam entries, this will give way to use only one MyStationTcam entry for all SVI interfaces, i.e, we could be able to scale to 4K L3 Interfaces. This change is for the following 2 MACs only.
7581
1. System MAC
7682
2. McLAG Gateway MAC
@@ -79,11 +85,12 @@ For SVI interfaces the lookup keys is only (DMAC)
7985
For Phy interfaces / RouterPorts the lookup key is retained the same [Port Id + DMAC]
8086
For LAG router interfaces the lookup key is retained the same [Trunk Id + DMAC]
8187

82-
## 2.2.1 MyStationTcam programming
88+
89+
### 2.2.1 MyStationTcam programming
8390
A new data structure is added and is managed by the sai data manager.This data structure handles the list of Vlan interfaces associated with the given mac and the corresponding MyStationTcam hardware id.
8491
When the first SVI interface configured with any of the above macs the corresponding tcam entry gets added. When the last SVI interface associated with the mac is deleted , the corresponding tcam entry will be deleted.
8592

86-
## 2.2.2 Routing behaviour differences
93+
### 2.2.2 Routing behaviour differences
8794
With the key change to only (MAC) from (MAC,Vlan), if the SVI interfaces are configured each with one type of MAC. Station tcam will look like below now.
8895
For example
8996
1. System MAC - 3c2c992ed875
@@ -99,7 +106,9 @@ If a packet comes with (DA=MCLAG Gateway MAC, Vlan = !MCLAG Vlan). It will be su
99106
If a packet comes with (DA=Any of the above two macs, Vlan=L2). It will be subjected to routing.
100107
Inorder to address these cases, added below IFP drop rules during system bringup. Also, counters attached to each of the drop rules.
101108

102-
### 2.3 VSI Profile
109+
110+
111+
## 2.3 VSI Profile
103112
For each SVI interface there is a VSI profile maintained to classify the SVI interfaces.
104113
2 VSI profile bits are consumed. One to identify the SAG MAC vlan and another to identify the MCLAG Gateway MAC vlan.
105114

@@ -109,36 +118,73 @@ For each SVI interface there is a VSI profile maintained to classify the SVI int
109118
#define _BRCM_SAI_VSI_ROUTER_MAC_L3_INTF (1<<3)------->MCLAG Gateway MAC VSI
110119
#define _BRCM_SAI_MAX_VSI_PROFILE_BITS_LENGTH 4
111120

112-
### 2.4 IFP Drop rules
113-
Below drop rules will be programmed only in TH/TH2 devices.
114-
1. DA=System MAC Vlan = SAG VSI profile ---> Drop
115-
2. DA=System MAC Vlan = MCLAG VSI profile ---> Drop
116-
3. DA=MCLAG GWMAC Vlan = !MCLAG VSI profile ---> Drop
121+
122+
## 2.4 IFP Drop rules
123+
Below drop rules will be programmed only in TH/TH2/TH3 devices.
124+
1. DA=System MAC Vlan = SAG VSI profile ---> Drop
125+
2. DA=System MAC Vlan = MCLAG VSI profile ---> Drop
126+
3. DA=MCLAG GWMAC Vlan = !MCLAG VSI profile ---> Drop
117127
4. DA=System MAC Vlan = L2 vlan ---> Drop
118128

129+
### 2.4.1 Debug commands to check the drop rule counters are below
130+
debugsh>show system internal sai vlan ifp drop counters
131+
debugsh>clear system internal vlan ifp drop counters
132+
root@sonic:/home/admin# debugsh
133+
===========================================================================================
134+
Welcome to SONiC Debug shell.
135+
136+
WARNING: The commands available in this shell are for internal use only. Use with caution.
137+
===========================================================================================
138+
debugsh> show system internal sai vlan ifp drop counters
139+
140+
| Drop entry | Counters |
141+
------------------------------------------------------------
142+
| DA = SystemMAC, Vlan = SAG-VSI | 0 |
143+
| DA = SystemMAC, Vlan = MclagGW-VSI | 0 |
144+
| MyStationHit, Vlan = L2-VSI | 20 |
145+
| DA = MclagGW-MAC, Vlan = Non-MclagGw-VSI | 0 |
146+
debugsh>
147+
debugsh>
148+
debugsh> clear system internal vlan ifp drop counters
149+
debugsh>
150+
debugsh>
151+
debugsh> show system internal sai vlan ifp drop counters
152+
153+
| Drop entry | Counters |
154+
------------------------------------------------------------
155+
| DA = SystemMAC, Vlan = SAG-VSI | 0 |
156+
| DA = SystemMAC, Vlan = MclagGW-VSI | 0 |
157+
| MyStationHit, Vlan = L2-VSI | 0 |
158+
| DA = MclagGW-MAC, Vlan = Non-MclagGw-VSI | 0 |
159+
debugsh>
160+
119161
# 4 Warm Boot Support
120-
Upgrade scenario will be supported. It will be make before break. On upgrade to higher version, in TH/TH2 devices one entry per each of the MACs will be programmed first, followed by deletion of all the individual entries programmed per each SVI interface. Incase of TD3 device, l2 table entry will be added first and then corresponding myStationTcam entry will be deleted.
162+
Upgrade scenario will be supported. It will be make before break. On upgrade to higher version, in TH/TH2/TH3 devices one entry per each of the MACs will be programmed first, followed by deletion of all the individual entries programmed per each SVI interface. Incase of TD3 device, l2 table entry will be added first and then corresponding myStationTcam entry will be deleted.
121163

122-
# 5 Unit Test
164+
# 5 Unit Tests
123165
Configure setup with base MCLAG config.
124166
Configure SAG MAC,MCLAG Gateway MAC.
125167
Configure SAG SVI interface, MCLAG Gateway MAC SVI interface and plain L3 SVI interface.
126-
127-
Testcase | Validate | Remarks |
128-
---------|----------------------------------------------------|--------------------------|
129-
1. | Send l3 pkt with DA= System MAC vlan=SAG vlan | Packet should be dropped |
130-
2. | Send l3 pkt with DA= System MAC vlan=MCLAG vlan | Packet should be dropped |
131-
3. | Send l3 pkt with DA= System MAC vlan=L2 vlan | Packet should be dropped |
132-
4. | Send l3 pkt with DA= System MAC vlan=L3 vlan | Packet should be routed |
133-
5. | Send l3 pkt with DA= SAG MAC vlan=MCLAG vlan | Packet should be dropped |
134-
6. | Send l3 pkt with DA= SAG MAC vlan=SAG vlan | Packet should be routed |
135-
7. | Send l3 pkt with DA= SAG MAC vlan=L3 vlan | Packet should be dropped |
136-
8. | Send l3 pkt with DA= SAG MAC vlan=L2 vlan | Packet should be dropped |
137-
9. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L2 vlan | Packet should be dropped |
138-
10. | Send l3 pkt with DA= MCLAG Gw MAC vlan=SAG vlan | Packet should be dropped |
139-
11. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L3 vlan | Packet should be dropped |
140-
12. | Send l3 pkt with DA= MCLAG Gw MAC vlan=MCLAG vlan | Packet should be routed |
141-
168+
Testcase | Validate | Remarks |
169+
---------|----------------------------------------------------|--------------------------------|
170+
1. | Send l3 pkt with DA= System MAC vlan=SAG vlan | Packet should be dropped |
171+
2. | Send l3 pkt with DA= System MAC vlan=MCLAG vlan | Packet should be dropped |
172+
3. | Send l3 pkt with DA= System MAC vlan=L2 vlan | Packet should be dropped |
173+
4. | Send l3 pkt with DA= System MAC vlan=L3 vlan | Packet should be routed |
174+
5. | Send l3 pkt with DA= SAG MAC vlan=MCLAG vlan | Packet should be dropped |
175+
6. | Send l3 pkt with DA= SAG MAC vlan=SAG vlan | Packet should be routed |
176+
7. | Send l3 pkt with DA= SAG MAC vlan=L3 vlan | Packet should be dropped |
177+
8. | Send l3 pkt with DA= SAG MAC vlan=L2 vlan | Packet should be dropped |
178+
9. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L2 vlan | Packet should be dropped |
179+
10. | Send l3 pkt with DA= MCLAG Gw MAC vlan=SAG vlan | Packet should be dropped |
180+
11. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L3 vlan | Packet should be dropped |
181+
12. | Send l3 pkt with DA= MCLAG Gw MAC vlan=MCLAG vlan | Packet should be routed |
182+
13. | Configure 4k Mclag Svi.Perform reboot. | |
183+
| Apply 4k static arp/nd.Start traffic | Validate no traffic disruption |
184+
14. | Configure 4k Mclag Svi.Perform fast reboot. | |
185+
| Apply 4k static arp/nd.Start traffic | Validate no traffic disruption |
186+
15. | Configure 4k Mclag Svi.Perform warm reboot. | |
187+
| Apply 4k static arp/nd.Start traffic | Validate no traffic disruption |
142188

143189
# 6 Internal Design Information
144190
NA

0 commit comments

Comments
 (0)