Skip to content

Commit fbad274

Browse files
ayurkiv-nvdaayurkiv
and
ayurkiv
authored
Add 'show' and 'clear' command for PG drop (sonic-net#1461)
* What I did Added new CLI commands to view and clear PG dropped packet statistics. Added the new CLI commands to the command reference guide. * How I did it I have added script which can generate table that shows current PG dropped packet statistic and also can clean it Clean and Show is implemented with using serialized stats from previous execution. To calculate current stats we do next step: - take old serialized data (if no 'clear' executed all serialized data is 0) - take current stat - show diff between current and old * How to verify it run show priority-group drop counters - show current stat sonic-clear priority-group drop counters - clear current stat * New command output (if the output of a command-line utility has changed) show priority-group drop counters Ingress PG dropped packets: Port PG0 PG1 PG2 PG3 PG4 PG5 PG6 PG7 --------- ----- ----- ----- ----- ----- ----- ----- ----- Ethernet0 800 801 802 803 804 805 806 807 Ethernet4 400 401 402 403 404 405 406 407 Ethernet8 100 101 102 103 104 105 106 107 ... sonic-clear priority-group drop counters Cleared PG drop counters Co-authored-by: ayurkiv <[email protected]>
1 parent 0de99c3 commit fbad274

File tree

6 files changed

+440
-2
lines changed

6 files changed

+440
-2
lines changed

clear/main.py

+14
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,20 @@ def clear_wm_pg_shared():
216216
command = 'watermarkstat -c -t pg_shared'
217217
run_command(command)
218218

219+
@priority_group.group()
220+
def drop():
221+
"""Clear priority-group dropped packets stats"""
222+
pass
223+
224+
@drop.command('counters')
225+
def clear_pg_counters():
226+
"""Clear priority-group dropped packets counter """
227+
228+
if os.geteuid() != 0 and os.environ.get("UTILITIES_UNIT_TESTING", "0") != "2":
229+
exit("Root privileges are required for this operation")
230+
command = 'pg-drop -c clear'
231+
run_command(command)
232+
219233
@priority_group.group(name='persistent-watermark')
220234
def persistent_watermark():
221235
"""Clear queue persistent WM. One does not simply clear WM, root is required"""

doc/Command-Reference.md

+19-2
Original file line numberDiff line numberDiff line change
@@ -6153,11 +6153,14 @@ This command displays the user watermark for the queues (Egress shared pool occu
61536153
61546154
**show priority-group**
61556155
6156-
This command displays the user watermark or persistent-watermark for the Ingress "headroom" or "shared pool occupancy" per priority-group for all ports
6156+
This command displays:
6157+
1) The user watermark or persistent-watermark for the Ingress "headroom" or "shared pool occupancy" per priority-group for all ports.
6158+
2) Dropped packets per priority-group for all ports
61576159
61586160
- Usage:
61596161
```
61606162
show priority-group (watermark | persistent-watermark) (headroom | shared)
6163+
show priority-group drop counters
61616164
```
61626165
61636166
- Example:
@@ -6187,6 +6190,18 @@ This command displays the user watermark or persistent-watermark for the Ingress
61876190
admin@sonic:~$ show priority-group persistent-watermark headroom
61886191
```
61896192
6193+
- Example (Ingress dropped packets per PG):
6194+
```
6195+
admin@sonic:~$ show priority-group drop counters
6196+
Ingress PG dropped packets:
6197+
Port PG0 PG1 PG2 PG3 PG4 PG5 PG6 PG7
6198+
----------- ----- ----- ----- ----- ----- ----- ----- -----
6199+
Ethernet0 0 0 0 0 0 0 0 0
6200+
Ethernet4 0 0 0 0 0 0 0 0
6201+
Ethernet8 0 0 0 0 0 0 0 0
6202+
Ethernet12 0 0 0 0 0 0 0 0
6203+
```
6204+
61906205
In addition to user watermark("show queue|priority-group watermark ..."), a persistent watermark is available.
61916206
It hold values independently of user watermark. This way user can use "user watermark" for debugging, clear it, etc, but the "persistent watermark" will not be affected.
61926207
@@ -6216,7 +6231,7 @@ This command displays the user persistet-watermark for the queues (Egress shared
62166231
admin@sonic:~$ show queue persistent-watermark multicast
62176232
```
62186233
6219-
- NOTE: Both "user watermark" and "persistent watermark" can be cleared by user:
6234+
- NOTE: "user watermark", "persistent watermark" and "ingress dropped packets" can be cleared by user:
62206235
62216236
```
62226237
admin@sonic:~$ sonic-clear queue persistent-watermark unicast
@@ -6226,6 +6241,8 @@ This command displays the user persistet-watermark for the queues (Egress shared
62266241
admin@sonic:~$ sonic-clear priority-group persistent-watermark shared
62276242
62286243
admin@sonic:~$ sonic-clear priority-group persistent-watermark headroom
6244+
6245+
admin@sonic:~$ sonic-clear priority-group drop counters
62296246
```
62306247
62316248
#### Buffer Pool

scripts/pg-drop

+250
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
#!/usr/bin/env python3
2+
3+
#####################################################################
4+
#
5+
# pg-drop is a tool for show/clear ingress pg dropped packet stats.
6+
#
7+
#####################################################################
8+
import _pickle as pickle
9+
import argparse
10+
import os
11+
import sys
12+
from collections import OrderedDict
13+
14+
from natsort import natsorted
15+
from tabulate import tabulate
16+
17+
# mock the redis for unit test purposes #
18+
try:
19+
if os.environ["UTILITIES_UNIT_TESTING"] == "2":
20+
modules_path = os.path.join(os.path.dirname(__file__), "..")
21+
tests_path = os.path.join(modules_path, "tests")
22+
sys.path.insert(0, modules_path)
23+
sys.path.insert(0, tests_path)
24+
import mock_tables.dbconnector
25+
26+
except KeyError:
27+
pass
28+
29+
from swsscommon.swsscommon import SonicV2Connector
30+
31+
STATUS_NA = 'N/A'
32+
33+
COUNTER_TABLE_PREFIX = "COUNTERS:"
34+
35+
COUNTERS_PORT_NAME_MAP = "COUNTERS_PORT_NAME_MAP"
36+
COUNTERS_PG_NAME_MAP = "COUNTERS_PG_NAME_MAP"
37+
COUNTERS_PG_PORT_MAP = "COUNTERS_PG_PORT_MAP"
38+
COUNTERS_PG_INDEX_MAP = "COUNTERS_PG_INDEX_MAP"
39+
40+
def get_dropstat_dir():
41+
dropstat_dir_prefix = '/tmp/dropstat'
42+
return "{}-{}/".format(dropstat_dir_prefix, os.getuid())
43+
44+
class PgDropStat(object):
45+
46+
def __init__(self):
47+
self.counters_db = SonicV2Connector(host='127.0.0.1')
48+
self.counters_db.connect(self.counters_db.COUNTERS_DB)
49+
50+
dropstat_dir = get_dropstat_dir()
51+
self.port_drop_stats_file = os.path.join(dropstat_dir, 'pg_drop_stats')
52+
53+
def get_port_id(oid):
54+
"""
55+
Get port ID using object ID
56+
"""
57+
port_id = self.counters_db.get(self.counters_db.COUNTERS_DB, COUNTERS_PG_PORT_MAP, oid)
58+
if port_id is None:
59+
print("Port is not available for oid '{}'".format(oid), file=sys.stderr)
60+
sys.exit(1)
61+
return port_id
62+
63+
# Get all ports
64+
self.counter_port_name_map = self.counters_db.get_all(self.counters_db.COUNTERS_DB, COUNTERS_PORT_NAME_MAP)
65+
if self.counter_port_name_map is None:
66+
print("COUNTERS_PORT_NAME_MAP is empty!", file=sys.stderr)
67+
sys.exit(1)
68+
69+
self.port_pg_map = {}
70+
self.port_name_map = {}
71+
72+
for port in self.counter_port_name_map:
73+
self.port_pg_map[port] = {}
74+
self.port_name_map[self.counter_port_name_map[port]] = port
75+
76+
# Get PGs for each port
77+
counter_pg_name_map = self.counters_db.get_all(self.counters_db.COUNTERS_DB, COUNTERS_PG_NAME_MAP)
78+
if counter_pg_name_map is None:
79+
print("COUNTERS_PG_NAME_MAP is empty!", file=sys.stderr)
80+
sys.exit(1)
81+
82+
for pg in counter_pg_name_map:
83+
port = self.port_name_map[get_port_id(counter_pg_name_map[pg])]
84+
self.port_pg_map[port][pg] = counter_pg_name_map[pg]
85+
86+
self.pg_drop_types = {
87+
"pg_drop" : {"message" : "Ingress PG dropped packets:",
88+
"obj_map" : self.port_pg_map,
89+
"idx_func": self.get_pg_index,
90+
"counter_name" : "SAI_INGRESS_PRIORITY_GROUP_STAT_DROPPED_PACKETS",
91+
"header_prefix": "PG"},
92+
}
93+
94+
def get_pg_index(self, oid):
95+
"""
96+
return PG index (0-7)
97+
98+
oid - object ID for entry in redis
99+
"""
100+
pg_index = self.counters_db.get(self.counters_db.COUNTERS_DB, COUNTERS_PG_INDEX_MAP, oid)
101+
if pg_index is None:
102+
print("Priority group index is not available for oid '{}'".format(table_id), file=sys.stderr)
103+
sys.exit(1)
104+
return pg_index
105+
106+
def build_header(self, pg_drop_type):
107+
"""
108+
Construct header for table with PG counters
109+
"""
110+
if pg_drop_type is None:
111+
print("Header info is not available!", file=sys.stderr)
112+
sys.exit(1)
113+
114+
self.header_list = ['Port']
115+
header_map = pg_drop_type["obj_map"]
116+
single_key = list(header_map.keys())[0]
117+
header_len = len(header_map[single_key])
118+
min_idx = sys.maxsize
119+
120+
for name, counter_oid in header_map[single_key].items():
121+
curr_idx = int(pg_drop_type["idx_func"](counter_oid))
122+
min_idx = min(min_idx, curr_idx)
123+
124+
self.min_idx = min_idx
125+
self.header_list += ["{}{}".format(pg_drop_type["header_prefix"], idx) for idx in range(self.min_idx, self.min_idx + header_len)]
126+
127+
def get_counters(self, table_prefix, port_obj, idx_func, counter_name):
128+
"""
129+
Get the counters of a specific table.
130+
"""
131+
port_drop_ckpt = {}
132+
# Grab the latest clear checkpoint, if it exists
133+
if os.path.isfile(self.port_drop_stats_file):
134+
port_drop_ckpt = pickle.load(open(self.port_drop_stats_file, 'rb'))
135+
136+
# Header list contains the port name followed by the PGs. Fields is used to populate the pg values
137+
fields = ["0"]* (len(self.header_list) - 1)
138+
139+
for name, obj_id in port_obj.items():
140+
full_table_id = table_prefix + obj_id
141+
old_collected_data = port_drop_ckpt.get(name,{})[full_table_id] if len(port_drop_ckpt) > 0 else 0
142+
idx = int(idx_func(obj_id))
143+
pos = idx - self.min_idx
144+
counter_data = self.counters_db.get(self.counters_db.COUNTERS_DB, full_table_id, counter_name)
145+
if counter_data is None:
146+
fields[pos] = STATUS_NA
147+
elif fields[pos] != STATUS_NA:
148+
fields[pos] = str(int(counter_data) - old_collected_data)
149+
return fields
150+
151+
def print_all_stat(self, table_prefix, key):
152+
"""
153+
Print table that show stats per PG
154+
"""
155+
table = []
156+
type = self.pg_drop_types[key]
157+
self.build_header(type)
158+
# Get stat for each port
159+
for port in natsorted(self.counter_port_name_map):
160+
row_data = list()
161+
data = self.get_counters(table_prefix, type["obj_map"][port], type["idx_func"], type["counter_name"])
162+
row_data.append(port)
163+
row_data.extend(data)
164+
table.append(tuple(row_data))
165+
166+
print(type["message"])
167+
print(tabulate(table, self.header_list, tablefmt='simple', stralign='right'))
168+
169+
def get_counts(self, counters, oid):
170+
"""
171+
Get the PG drop counts for an individual counter.
172+
"""
173+
counts = {}
174+
table_id = COUNTER_TABLE_PREFIX + oid
175+
for counter in counters:
176+
counter_data = self.counters_db.get(self.counters_db.COUNTERS_DB, table_id, counter)
177+
if counter_data is None:
178+
counts[table_id] = 0
179+
else:
180+
counts[table_id] = int(counter_data)
181+
return counts
182+
183+
def get_counts_table(self, counters, object_table):
184+
"""
185+
Returns a dictionary containing a mapping from an object (like a port)
186+
to its PG drop counts. Counts are contained in a dictionary that maps
187+
counter oid to its counts.
188+
"""
189+
counter_object_name_map = self.counters_db.get_all(self.counters_db.COUNTERS_DB, object_table)
190+
current_stat_dict = OrderedDict()
191+
192+
if counter_object_name_map is None:
193+
return current_stat_dict
194+
195+
for obj in natsorted(counter_object_name_map):
196+
current_stat_dict[obj] = self.get_counts(counters, counter_object_name_map[obj])
197+
return current_stat_dict
198+
199+
def clear_drop_counts(self):
200+
"""
201+
Clears the current PG drop counter.
202+
"""
203+
204+
counter_pg_drop_array = [ "SAI_INGRESS_PRIORITY_GROUP_STAT_DROPPED_PACKETS"]
205+
try:
206+
pickle.dump(self.get_counts_table(
207+
counter_pg_drop_array,
208+
COUNTERS_PG_NAME_MAP),
209+
open(self.port_drop_stats_file, 'wb+'))
210+
except IOError as e:
211+
print(e)
212+
sys.exit(e.errno)
213+
print("Cleared PG drop counter")
214+
215+
def main():
216+
parser = argparse.ArgumentParser(description='Display PG drop counter',
217+
formatter_class=argparse.RawTextHelpFormatter,
218+
epilog="""
219+
Examples:
220+
pg-drop -c show
221+
pg-drop -c clear
222+
""")
223+
224+
parser.add_argument('-c', '--command', type=str, help='Desired action to perform')
225+
226+
args = parser.parse_args()
227+
command = args.command
228+
229+
dropstat_dir = get_dropstat_dir()
230+
# Create the directory to hold clear results
231+
if not os.path.exists(dropstat_dir):
232+
try:
233+
os.makedirs(dropstat_dir)
234+
except IOError as e:
235+
print(e)
236+
sys.exit(e.errno)
237+
238+
pgdropstat = PgDropStat()
239+
240+
if command == 'clear':
241+
pgdropstat.clear_drop_counts()
242+
elif command == 'show':
243+
pgdropstat.print_all_stat(COUNTER_TABLE_PREFIX, "pg_drop" )
244+
else:
245+
print("Command not recognized")
246+
sys.exit(0)
247+
248+
249+
if __name__ == "__main__":
250+
main()

show/main.py

+11
Original file line numberDiff line numberDiff line change
@@ -605,6 +605,17 @@ def wm_pg_shared():
605605
command = 'watermarkstat -t pg_shared'
606606
run_command(command)
607607

608+
@priority_group.group()
609+
def drop():
610+
"""Show priority-group"""
611+
pass
612+
613+
@drop.command('counters')
614+
def pg_drop_counters():
615+
"""Show dropped packets for priority-group"""
616+
command = 'pg-drop -c show'
617+
run_command(command)
618+
608619
@priority_group.group(name='persistent-watermark')
609620
def persistent_watermark():
610621
"""Show priority-group persistent WM"""

0 commit comments

Comments
 (0)