Skip to content

Commit bd900bf

Browse files
michaelli10michaelli10
authored
Merge pull request sonic-net#304 from michaelli10/cpu_packet_debug_counters
SONiC CPU Packet Debug Counters updates
2 parents 8fa89f4 + da5898c commit bd900bf

File tree

1 file changed

+143
-123
lines changed

1 file changed

+143
-123
lines changed

‎system/SONiC_CPU_Packet_Debug_Counters.md

+143-123
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@ SONiC CPU Packet Debug Counters
1818
|:---:|:-----------:|:------------------:|-----------------------------------|
1919
| 0.1 | 04/13/2021 | Michael Li | Initial version |
2020
| 0.2 | 08/16/2021 | Michael Li | Design detail updates |
21+
| 0.3 | 08/30/2021 | Michael Li | Added guidance for testing pkt drops
22+
| 0.4 | 10/20/2021 | Michael Li | Pkt classification corrections and CLI output updates
23+
2124

2225
# About this Manual
2326
This document provides general information about the SONiC CPU Packet Debug Counters feature implementation in SONiC.
@@ -94,7 +97,7 @@ References:
9497
| PTP | 19 | (EType==0x88F7)
9598
| LLDP | 18 | (EType==0x88CC) && (MAC DA == {01:80:C2:00:00:00} or {01:80:C2:00:00:03} or {01:80:C2:00:00:0e})
9699
| VRRP | 17 | (EType==0x800) && (IpProtocol == 112)
97-
| ICCP | 16 | (IpProtocol == 6) && (IpProtocol == 8888)
100+
| ICCP | 16 | (IpProtocol == 6) && (L4DstPort == 8888)
98101
| OSPFv4 | 15 | (EType==0x800) && (IpProtocol == 89)
99102
| BGPv4 | 14 | (EType==0x800) && (IpProtocol == 6) && (L4Port == 179)
100103
| BGPv6 | 14 | (EType==0x86DD) && (IpProtocol == 6) && (L4Port == 179)
@@ -111,15 +114,15 @@ References:
111114
| ICMP | 8 | (IpProtocol == 1)
112115
| SSH | 7 | (IpProtocol == 6) && ((L4Port == 22))|
113116
| INET | 7 | (Etype==0x800)
114-
| INETv6 | 7 | (Etype==0x88DD)
117+
| INETv6 | 7 | (Etype==0x86DD)
115118
| Subnet | 6 | Not classified (Rx queue counters only)
116119
| NAT | 5 | Not classified (Rx queue counters only)
117120
| MTU | 4 | Not classified (Rx queue counters only)
118121
| Sflow | 3 | Rx: Pkt sample metadata, Tx: (L4Port == 6343 for inband sflow datagram to collector)
119122
| TTL | 0 | TTL == 0 or TTL == 1
120123

121124
### 3.1.4 Packet Drop Error types
122-
Errors in KNET (and in Rx kernel path) which result in packet drops are counted per protocol/queue on each interface. Some common errors are listed below.
125+
Errors in KNET (and in Rx kernel path) which result in packet drops are counted per pkt_type/queue on each interface. Some common errors are listed below.
123126
### **Table 2: Packet Drop Errors**
124127
| Error | Description
125128
|----------------------|-----------------------------------
@@ -144,161 +147,166 @@ There are no new changes to SyncD.
144147
There are no new changes to manageability infrastructure
145148

146149
## 3.6 CLI
147-
KNET Packet stats at switch and interface levels can be shown through click commands or by dumping KNET pkt_stats procfs file. Rx and Tx error counter details are available for protocol and rx queue counters at switch and interface levels.
150+
KNET Packet stats at switch and interface levels can be shown through click commands or by dumping KNET pkt_stats procfs file. Rx and Tx error counter details are available for pkt_type and rx queue counters at switch and interface levels.
151+
152+
### 3.6.1 Packet Type Stats
153+
* Show pkt_type stats (real output will not show entries with zero counts)
148154

149-
### 3.6.1 Protocol Stats
150-
* Show protocol stats (real output will not show entries with zero counts)
151155
```
152-
DUT# show knet stats protocol
156+
DUT# show knet stats pkt-type // show totals from all interfaces
157+
DUT# show knet stats pkt-type -a // show all individual interfaces
153158
or
154-
DUT# cat /proc/bcm/knet/stats/protocol
155-
156-
Total rx = 101127
157-
Total tx = 235644
158-
protocol rx rx_err tx tx_err
159-
------------------------------------------------------------
160-
LACP 0 0 0 0
161-
UDLD 0 0 0 0
162-
STP 0 0 0 0
163-
PVRSTP 0 0 0 0
164-
BFDv4 0 0 0 0
165-
BFDv6 0 0 0 0
166-
PTP 0 0 0 0
167-
LLDP 24291 0 54 0
168-
VRRP 0 0 0 0
169-
ICCP 0 0 0 0
170-
OSPF 0 0 0 0
171-
BGPv4 0 0 0 0
172-
BGPv6 0 0 0 0
173-
PIM 0 0 0 0
174-
IGMP 0 0 0 0
175-
ARP Req 4428 0 4428 0
176-
ARP Rply 4428 0 4428 0
177-
ARP 8856 0 8856 0
178-
NDP 0 0 0 0
179-
DHCPv4 0 0 0 0
180-
DHCPv6 0 0 0 0
181-
ICMP EchoReq 0 0 0 0
182-
ICMP EchoRply 0 0 0 0
183-
ICMP 0 0 0 0
184-
SSH 0 0 0 0
185-
SFLOW 0 0 0 0
186-
TTL 0 0 0 0
187-
INETv4 0 0 0 0
188-
INETv6 0 0 0 0
189-
UNKNOWN 0 0 0 0
159+
DUT# cat /proc/bcm/knet-cb/stats/pkt_type
160+
161+
admin@sonic:~$ show knet stats pkt-type
162+
KNET Pkt Type Stats
163+
Pkt Type Rx Pkts Rx errors Tx Pkts Tx errors
164+
---------------------- ---------- ---------- ---------- ----------
165+
PKT_TYPE_UNKNOWN 0 0 0 0
166+
PKT_TYPE_LACP 466 4310 468 0
167+
PKT_TYPE_UDLD 0 0 0 0
168+
PKT_TYPE_STP 0 0 0 0
169+
PKT_TYPE_PVRSTP 0 0 0 0
170+
PKT_TYPE_BFDV4 0 0 0 0
171+
PKT_TYPE_BFDV6 0 0 0 0
172+
PKT_TYPE_PTP 0 0 0 0
173+
PKT_TYPE_LLDP 720 0 1297 0
174+
PKT_TYPE_VRRP 0 0 0 0
175+
PKT_TYPE_ICCP 0 0 4314 0
176+
PKT_TYPE_OSPF 0 0 0 0
177+
PKT_TYPE_BGPV4 0 0 0 0
178+
PKT_TYPE_BGPV6 0 0 0 0
179+
PKT_TYPE_PIM 0 0 0 0
180+
PKT_TYPE_IGMP 0 0 0 0
181+
PKT_TYPE_ARPREQ 13 0 14 0
182+
PKT_TYPE_ARPRPLY 14 0 13 0
183+
PKT_TYPE_ARP 0 0 0 0
184+
PKT_TYPE_NDP 0 0 0 0
185+
PKT_TYPE_DHCPV4 0 0 0 0
186+
PKT_TYPE_DHCPV6 0 0 0 0
187+
PKT_TYPE_ICMP_ECHOREQ 0 0 0 0
188+
PKT_TYPE_ICMP_ECHORPLY 0 0 0 0
189+
PKT_TYPE_ICMP 0 0 0 0
190+
PKT_TYPE_SSH 0 0 0 0
191+
PKT_TYPE_INETV4 4313 0 0 0
192+
PKT_TYPE_INETV6 0 0 0 0
193+
PKT_TYPE_SFLOW 0 0 0 0
194+
PKT_TYPE_TTL 0 0 0 0
195+
Total 5526 4310 6106 0
190196
```
191-
* Show interface level protocol stats
197+
* Show interface level pkt type stats
198+
192199
```
193-
DUT# show knet Ethernet0 stats protocol
200+
DUT# show knet stats pkt-type Ethernet0
194201
or
195-
DUT# cat /proc/bcm/knet/Ethernet0/stats/protocol
196-
197-
Total Ethernet0 rx = 101127
198-
Total Ethernet0 tx = 235644
199-
protocol rx rx_err tx tx_err
200-
------------------------------------------------------------
201-
LACP 0 0 0 0
202-
UDLD 0 0 0 0
203-
STP 0 0 0 0
204-
PVRSTP 0 0 0 0
205-
BFDv4 0 0 0 0
202+
DUT# cat /proc/bcm/knet-cb/stats/Ethernet0/pkt_type
203+
204+
Ethernet0 Pkt Type Stats
205+
Pkt Type Rx Pkts Rx errors Tx Pkts Tx errors
206+
---------------------- ---------- ---------- ---------- ----------
207+
PKT_TYPE_LLDP 30 0 30 0
208+
PKT_TYPE_ICCP 0 0 875 0
209+
PKT_TYPE_ARPREQ 3 0 3 0
210+
PKT_TYPE_ARPRPLY 3 0 3 0
211+
PKT_TYPE_INETV4 871 0 0 0
212+
Total 907 0 911 0
206213
...
207214
```
208-
* Show interface level protocol stats with error details
215+
* Show interface level pkt type stats with error details
209216
```
210-
DUT# show knet Ethernet0 stats protocol details
217+
DUT# show knet stats pkt-type Ethernet0 -v
211218
or
212-
DUT# echo "detail=1" > /proc/bcm/knet/Ethernet0/stats/protocol
213-
DUT# cat /proc/bcm/knet/Ethernet0/stats/protocol
214-
DUT# echo "detail=0" > /proc/bcm/knet/Ethernet0/stats/protocol
215-
216-
Total Ethernet0 rx = 101127
217-
Total Ethernet0 tx = 235644
218-
protocol rx rx_err tx tx_err
219-
------------------------------------------------------------
220-
LACP 0 0 0 0
221-
UDLD 0 0 0 0
222-
STP 0 0 0 0
223-
PVRSTP 685 222 0 193
224-
LINK_DOWN 208 168
225-
NO_SKB 2 5
226-
NO_BUFFER 0 20
227-
KERNEL_DROP 12 0
228-
BFDv4 0 0 0 0
219+
DUT# echo "detail=1" > /proc/bcm/knet-cb/stats/debug
220+
DUT# cat /proc/bcm/knet-cb/Ethernet0/stats/pkt_type
221+
DUT# echo "detail=0" > /proc/bcm/knet-cb/stats/debug
222+
223+
Ethernet0 Pkt Type Stats
224+
Pkt Type Rx Pkts Rx errors Tx Pkts Tx errors
225+
---------------------- ---------- ---------- ---------- ----------
226+
PKT_TYPE_LACP 127 934 126 0
227+
Kernel netstack drop 934 0
228+
PKT_TYPE_LLDP 157 1 283 0
229+
Link down 1 0
230+
PKT_TYPE_ICCP 0 0 933 0
231+
PKT_TYPE_ARPREQ 3 0 3 0
232+
PKT_TYPE_ARPRPLY 3 0 3 0
233+
PKT_TYPE_INETV4 929 0 0 0
234+
Total 1219 935 1348 0
229235
...
230236
```
231237

238+
* Clear all KNET pkt statistics
239+
```
240+
DUT# sonic-clear knet stats
241+
```
232242

233-
* Clear protocol pkt statistics
243+
* Clear specific pkt type statistics
234244
```
235-
DUT# sonic-clear knet stats protocol
236-
DUT# sonic-clear knet Ethernet0 stats protocol
245+
DUT# show knet stats pkt-type -c
246+
DUT# show knet stats pkt-type -c Ethernet0
237247
or
238-
DUT# echo "clear" > /proc/bcm/knet/stats/protocol
239-
DUT# echo "clear" > /proc/bcm/knet/Ethernet0/stats/protocol
248+
DUT# echo "clear" > /proc/bcm/knet-cb/stats/pkt_type
249+
DUT# echo "clear" > /proc/bcm/knet-cb/stats/Ethernet0/pkt_type
240250
```
241251
### 3.6.2 Rx Queue Stats
242252
```
243-
DUT# show knet stats rx_queue
253+
DUT# show knet stats rx-queue // show totals from all interfaces
254+
DUR# show knet stats rx-queue -a // show all individual interfaces
244255
or
245-
DUT# cat /proc/bcm/knet/stats/rx_queue
246-
247-
Total rx = 24918
248-
name queue rx rx_err
249-
----------------------------------------------
250-
SFLOW 3 0 0
251-
ARP 10 238 0
252-
LLDP 18 24291 0
253-
LACP 23 389 0
256+
DUT# cat /proc/bcm/knet-cb/stats/rx_queue
257+
258+
KNET Rx Queue Stats
259+
Queue Rx Pkts Rx errors Description
260+
---------- ---------- ---------- --------------------------------
261+
10 10 0 arp_req,arp_resp,neigh_discovery
262+
16 1465 0 iccp
263+
18 243 1 lldp
264+
23 179 1471 lacp
265+
Total 1897 1472
254266
```
255267

256268
* Show interface level rx queue stats
257269
```
258-
DUT# show knet Ethernet0 stats rx_queue
270+
DUT# show knet stats rx-queue Ethernet0
259271
or
260-
DUT# cat /proc/bcm/knet/Ethernet0/stats/rx_queue
261-
262-
Total Ethernet0 rx = 24918
263-
name queue rx rx_err
264-
----------------------------------------------
265-
SFLOW 3 0 0
266-
ARP 10 238 0
267-
LLDP 18 24291 0
268-
LACP 23 389 0
272+
DUT# cat /proc/bcm/knet-cb/Ethernet0/stats/rx_queue
273+
274+
Ethernet0 Rx Queue Stats
275+
Queue Rx Pkts Rx errors Description
276+
---------- ---------- ---------- --------------------------------
277+
10 10 0 arp_req,arp_resp,neigh_discovery
278+
16 1465 0 iccp
279+
18 243 1 lldp
280+
23 179 1471 lacp
281+
Total 1897 1472
269282
```
270283

271284
* Show interface level rx queue stats with error details
272285
```
273-
DUT# show knet Ethernet0 stats rx_queue details
286+
DUT# show knet stats rx_queue Ethernet0 -v
274287
or
275-
DUT# echo "detail=1" > /proc/bcm/knet/Ethernet0/stats/rx_queue
276-
DUT# cat /proc/bcm/knet/Ethernet0/stats/rx_queue
277-
DUT# echo "detail=0" > /proc/bcm/knet/Ethernet0/stats/rx_queue
278-
279-
Total Ethernet0 rx = 24918
280-
name queue rx rx_err
281-
----------------------------------------------
282-
SFLOW 3 0 0
283-
ARP 10 238 220
284-
LINK_DOWN 208
285-
KERNEL_DROP 12
286-
LLDP 18 24291 15
287-
KERNEL_DROP 15
288-
LACP 23 389 0
289-
DEFAULT 0 0 7
290-
NO_FILTER_MATCH 2
291-
UNKN_NETIF 2
292-
UNKN_DEST_TYPE 3
288+
DUT# echo "detail=1" > /proc/bcm/knet-cb/stats/debug
289+
DUT# cat /proc/bcm/knet-cb/stats/Ethernet0/rx_queue
290+
DUT# echo "detail=1" > /proc/bcm/knet-cb/stats/debug
291+
292+
Queue Rx Pkts Rx errors Description
293+
---------- ---------- ---------- --------------------------------
294+
10 11 0 arp_req,arp_resp,neigh_discovery
295+
16 1591 0 iccp
296+
18 267 1 lldp
297+
1 Link down
298+
23 193 1597 lacp
299+
1597 Kernel netstack drop
300+
Total 2062 1598
293301
```
294302

295303
* Clear Rx Queue stats
296304
```
297-
DUT# sonic-clear knet stats rx_queue
298-
DUT# sonic-clear knet Ethernet0 stats rx_queue
305+
DUT# show knet stats rx-queue -c
306+
DUT# show knet stats rx-queue Ethernet0 -c
299307
or
300-
DUT# echo "clear" > /proc/bcm/knet/stats/rx_queue
301-
DUT# echo "clear" > /proc/bcm/knet/Ethernet0/stats/rx_queue
308+
DUT# echo "clear" > /proc/bcm/knet-cb/stats/rx_queue
309+
DUT# echo "clear" > /proc/bcm/knet-cb/stats/Ethernet0/rx_queue
302310
```
303311

304312
# 4 Flow Diagrams
@@ -360,4 +368,16 @@ echo "max_tx_dcbs=4" > /proc/bcm/knet/dma
360368
```
361369
DUT# cat /proc/bcm/knet/dstats | grep "Tx used DCBs hi wm"
362370
Tx used DCBs hi wm 4
371+
```
372+
## 10.2 Disabling CPU Pkt KNET debug stats
373+
Add "enable_stats=0" kernel parameter when inserting the knetcb kernel module to completely disable KNET debug stats logic. Modify the following file on the target and reboot.
374+
375+
```
376+
vi /etc/init.d/opennsl-modules
377+
378+
// on TD4 (ngknet modules)
379+
insmod /lib/modules/4.19.0-9-2-amd64/extra/linux_ngknetcb.ko enable_stats=0
380+
381+
// on legacy devices
382+
insmod /lib/modules/4.19.0-9-2-amd64/extra/linux-knet-cb.ko enable_stats=0
363383
```

0 commit comments

Comments
 (0)