Debug dump utility dash objects update (sonic-net#3387)

#### What I did
Added support for the following DASH objects for the dump utility:
`dash_acl_group`
`dash_acl_out`
`dash_acl_rule`
`dash_appliance`
`dash_prefix_tag`
`dash_eni`
`dash_qos`
`dash_route`
`dash_route_rule`
`dash_vnet_mapping`
`dash_vnet`

This PR also adds Match infra update in order to consider Dash objects, so new match requests can be created with the fields present in the dash objects to obtain the fields/keys.

#### How I did it
Added `protobuf` library and `libdashapi` to `sonic-utilities`, this is required as the dash objects are stored in the APPL_DB in protobuf format, and we need the .proto files which are available in the `libdashapi` package. We also use `redis` package instead of the `SonicV2Connector` since `SonicV2Connector` `get_all` function from the connector considers null terminated strings so the complete protobuf data is not obtained using `get_all` function
#### How to verify it
`dump state all <dash_object>`
Examples:
```
admin@sonic:~$ dump state dash_vnet Vnet1 -t
+-------------------+-----------+--------------------------------------------------------------------------------------------+
| dash_vnet_table   | DB_NAME   | DUMP                                                                                       |
+===================+===========+============================================================================================+
| Vnet1             | APPL_DB   | +-----------------------+----------------------------------------------------+             |
|                   |           | | Keys                  | field-value pairs                                  |             |
|                   |           | +=======================+====================================================+             |
|                   |           | | DASH_VNET_TABLE:Vnet1 | +---------+--------------------------------------+ |             |
|                   |           | |                       | | field   | value                                | |             |
|                   |           | |                       | |---------+--------------------------------------| |             |
|                   |           | |                       | | vni     | 50                                   | |             |
|                   |           | |                       | | guid    | 5526cce8-26ab-4193-b946-ccc0e8f930b0 | |             |
|                   |           | |                       | +---------+--------------------------------------+ |             |
|                   |           | +-----------------------+----------------------------------------------------+             |
+-------------------+-----------+--------------------------------------------------------------------------------------------+
| Vnet1             | ASIC_DB   | +------------------------------------------------------+---------------------------------+ |
|                   |           | | Keys                                                 | field-value pairs               | |
|                   |           | +======================================================+=================================+ |
|                   |           | | ASIC_STATE:SAI_OBJECT_TYPE_VNET:oid:0x7a000000000021 | +-------------------+---------+ | |
|                   |           | |                                                      | | field             |   value | | |
|                   |           | |                                                      | |-------------------+---------| | |
|                   |           | |                                                      | | SAI_VNET_ATTR_VNI |    100  | | |
|                   |           | |                                                      | +-------------------+---------+ | |
|                   |           | +------------------------------------------------------+---------------------------------+ |
|                   |           | +----------------------+--------------------+                                              |
|                   |           | | vid                  | rid                |                                              |
|                   |           | +======================+====================+                                              |
|                   |           | | oid:0x7a000000000021 | oid:0xffff70009130 |                                              |
|                   |           | +----------------------+--------------------+                                              |
+-------------------+-----------+--------------------------------------------------------------------------------------------+
```
```
admin@sonic:~$ dump state dash_acl_rule all
{
    "group1:rule1": {
        "APPL_DB": {
            "keys": [
                {
                    "DASH_ACL_RULE_TABLE:group1:rule1": {
                        "action": "ACTION_PERMIT",
                        "terminating": true,
                        "src_addr": [
                            "0.0.0.0/0"
                        ],
                        "dst_addr": [
                            "0.0.0.0/0"
                        ],
                        "src_port": [
                            {
                                "value": 80
                            }
                        ],
                        "dst_port": [
                            {
                                "value": 5005
                            }
                        ]
                    }
                }
            ],
            "tables_not_found": []
        }
    }
}
```

Author: gpunathilell

azure-pipelines.yml

@@ -84,6 +84,7 @@ stages:
         sudo dpkg -i libyang_1.0.73_amd64.deb
         sudo dpkg -i libyang-cpp_1.0.73_amd64.deb
         sudo dpkg -i python3-yang_1.0.73_amd64.deb
+        sudo dpkg -i libprotobuf*.deb
       workingDirectory: $(Pipeline.Workspace)/target/debs/bullseye/
       displayName: 'Install Debian dependencies'
 
@@ -104,6 +105,27 @@ stages:
       workingDirectory: $(Pipeline.Workspace)/
       displayName: 'Install swss-common dependencies'
 
+
+    - task: DownloadPipelineArtifact@2
+      inputs:
+        source: specific
+        project: build
+        pipeline: sonic-net.sonic-dash-api
+        artifact: sonic-dash-api
+        runVersion: 'latestFromBranch'
+        runBranch: 'refs/heads/$(BUILD_BRANCH)'
+        path: $(Build.ArtifactStagingDirectory)/download
+        patterns: |
+          libdashapi*.deb
+      displayName: "Download dash api"
+
+    - script: |
+        set -xe
+        sudo apt-get update
+        sudo dpkg -i $(Build.ArtifactStagingDirectory)/download/libdashapi_*.deb
+      workingDirectory: $(Pipeline.Workspace)/
+      displayName: 'Install libdashapi libraries'
+
     - script: |
         set -xe
         sudo pip3 install swsssdk-2.0.1-py3-none-any.whl

dump/dash_util.py

@@ -0,0 +1,90 @@
+import base64
+import uuid
+import socket
+import ipaddress
+from google.protobuf.message import Message
+from dash_api.types_pb2 import Guid, IpAddress, IpPrefix
+from google.protobuf.json_format import MessageToDict
+
+
+def format_ip(node):
+    return str(ipaddress.IPv4Address(socket.ntohl(node)))
+
+
+def format_mac(node):
+    b64 = base64.b64decode(node)
+    return ':'.join(b64.hex()[i:i + 2] for i in range(0, 12, 2))
+
+
+def format_guid_dict(node):
+    b64 = base64.b64decode(node['value'])
+    return str(uuid.UUID(bytes=b64))
+
+
+def format_ip_address_dict(node):
+    if 'ipv4' in node:
+        return format_ip(node['ipv4'])
+
+
+def format_ip_prefix(node):
+    ip = format_ip_address_dict(node['ip'])
+    mask = format_ip_address_dict(node['mask'])
+    network = ipaddress.IPv4Network(f'{ip}/{mask}', strict=False)
+    return str(network)
+
+
+def get_decoded_value(pb, pb_data):
+    pb.ParseFromString(pb_data[b'pb'])
+    json_string = MessageToDict(pb, preserving_proto_field_name=True)
+    json_string = find_known_types_sec(pb, json_string)
+    return json_string
+
+
+decode_types = [IpAddress, Guid, IpPrefix]
+decode_types = [cls.__module__ + '.' + cls.__name__ for cls in decode_types]
+decode_fn = {'IpAddress': format_ip_address_dict,
+             'Guid': format_guid_dict,
+             'mac_address': format_mac,
+             'IpPrefix': format_ip_prefix}
+
+
+def find_known_types_sec(pb2_obj, pb2_dict):
+
+    def process_msg_field(obj, proto_dict, field_name):
+        class_name = type(obj).__name__
+        obj_type = f"{type(obj).__module__}.{type(obj).__name__}"
+        if obj_type in decode_types:
+            proto_dict[field_name] = decode_fn[class_name](proto_dict[field_name])
+        else:
+            find_index(obj, proto_dict[field_name])
+
+    def process_rep_field(obj, proto_dict, field_name):
+        final_list = []
+        requires_change = False
+        for ind, value in enumerate(obj):
+            if isinstance(value, Message):
+                obj_type = f"{type(value).__module__}.{type(value).__name__}"
+                if obj_type in decode_types:
+                    requires_change = True
+                    class_name = type(value).__name__
+                    final_list.append(decode_fn[class_name](proto_dict[field_name][ind]))
+                else:
+                    find_index(value, pb2_dict[field_name][ind])
+        if requires_change:
+            proto_dict[field_name] = final_list
+
+    def find_index(proto_obj, proto_dict=pb2_dict):
+        for field_descriptor, value in proto_obj.ListFields():
+            field_name = field_descriptor.name
+            field_type = field_descriptor.type
+            if field_type == field_descriptor.TYPE_MESSAGE:
+                obj = getattr(proto_obj, field_name)
+                if field_descriptor.label == field_descriptor.LABEL_REPEATED:
+                    process_rep_field(obj, proto_dict, field_name)
+                else:
+                    process_msg_field(obj, proto_dict, field_name)
+            elif field_name in decode_fn:
+                proto_dict[field_name] = decode_fn[field_name](proto_dict[field_name])
+
+    find_index(pb2_obj)
+    return pb2_dict

dump/main.py

@@ -93,7 +93,7 @@ def state(ctx, module, identifier, db, table, key_map, verbose, namespace):
     vidtorid = extract_rid(collected_info, namespace, ctx.obj.conn_pool)
 
     if not key_map:
-        collected_info = populate_fv(collected_info, module, namespace, ctx.obj.conn_pool)
+        collected_info = populate_fv(collected_info, module, namespace, ctx.obj.conn_pool, obj.return_pb2_obj())
 
     for id in vidtorid.keys():
         collected_info[id]["ASIC_DB"]["vidtorid"] = vidtorid[id]
@@ -145,7 +145,7 @@ def filter_out_dbs(db_list, collected_info):
     return collected_info
 
 
-def populate_fv(info, module, namespace, conn_pool):
+def populate_fv(info, module, namespace, conn_pool, dash_object):
     all_dbs = set()
     for id in info.keys():
         for db_name in info[id].keys():
@@ -157,7 +157,9 @@ def populate_fv(info, module, namespace, conn_pool):
             db_cfg_file.connect(plugins.dump_modules[module].CONFIG_FILE, namespace)
         else:
             conn_pool.get(db_name, namespace)
-    
+    if dash_object:
+        conn_pool.get_dash_conn(namespace)
+        redis_conn = conn_pool.cache.get(namespace, {}).get("DASH_"+CONN, None)
     db_conn = conn_pool.cache.get(namespace, {}).get(CONN, None)
 
     final_info = {}
@@ -170,10 +172,17 @@ def populate_fv(info, module, namespace, conn_pool):
             for key in info[id][db_name]["keys"]:
                 if db_name == "CONFIG_FILE":
                     fv = db_cfg_file.get(db_name, key)
+                elif dash_object and db_name == "APPL_DB":
+                    try:
+                        from dump.dash_util import get_decoded_value
+                        pb_data = redis_conn.hgetall(key)
+                        fv = get_decoded_value(dash_object, pb_data)
+                    except ModuleNotFoundError:
+                        print("Issue in importing dash module!")
+                        return final_info
                 else:
                     fv = db_conn.get_all(db_name, key)
                 final_info[id][db_name]["keys"].append({key: fv})
-
     return final_info
 
 

dump/match_infra.py

@@ -6,6 +6,8 @@
 from swsscommon.swsscommon import SonicV2Connector, SonicDBConfig
 from sonic_py_common import multi_asic
 from utilities_common.constants import DEFAULT_NAMESPACE
+import redis
+
 
 # Constants
 CONN = "conn"
@@ -60,6 +62,7 @@ def __init__(self, **kwargs):
         self.just_keys = kwargs["just_keys"] if "just_keys" in kwargs else True
         self.ns = kwargs["ns"] if "ns" in kwargs else ""
         self.match_entire_list = kwargs["match_entire_list"] if "match_entire_list" in kwargs else False
+        self.PbObj = kwargs["pb"] if "pb" in kwargs else None
         err = self.__static_checks()
         verbose_print(str(err))
         if err:
@@ -194,6 +197,51 @@ def hgetall(self, db, key):
         return self.conn.get_all(db, key)
 
 
+class RedisPySource(SourceAdapter):
+    """ Concrete Adaptor Class for connecting to APPL_DB using Redis library"""
+
+    def __init__(self, conn_pool, pb_obj):
+        self.conn = None
+        self.pool = conn_pool
+        self.pb_obj = pb_obj
+
+    def get_decoded_value(self, pb_obj, key_val):
+        try:
+            from dump.dash_util import get_decoded_value
+        except ModuleNotFoundError as e:
+            verbose_print("RedisPySource: decoded value cannot be obtained \
+                          since dash related library import issues\n" + str(e))
+            return
+        return get_decoded_value(pb_obj, key_val)
+
+    def connect(self, db, ns):
+        try:
+            self.conn = self.pool.get_dash_conn(ns)
+        except Exception as e:
+            verbose_print("RedisPySource: Connection Failed\n" + str(e))
+            return False
+        return True
+
+    def get_separator(self):
+        return ":"
+
+    def getKeys(self, db, table, key_pattern):
+        bin_keys = self.conn.keys(table + self.get_separator() + key_pattern)
+        return [key1.decode() for key1 in bin_keys]
+
+    def get(self, db, key):
+        key_val = self.conn.hgetall(key)
+        return self.get_decoded_value(self.pb_obj, key_val)
+
+    def hget(self, db, key, field):
+        key_val = self.conn.hgetall(key)
+        decoded_dict = self.get_decoded_value(self.pb_obj, key_val)
+        return decoded_dict.get(field)
+
+    def hgetall(self, db, key):
+        key_val = self.conn.hgetall(key)
+        return self.get_decoded_value(self.pb_obj, key_val)
+
 class JsonSource(SourceAdapter):
     """ Concrete Adaptor Class for connecting to JSON Data Sources """
 
@@ -249,26 +297,53 @@ def initialize_connector(self, ns):
                 SonicDBConfig.load_sonic_db_config()
         return SonicV2Connector(namespace=ns, use_unix_socket_path=True)
 
+    def initialize_redis_conn(self, ns):
+        """Return redis connection for APPL_DB,
+        as APPL_DB is the only one which stores data in protobuf
+        format which is not obtained fully by the SonicV2Connector
+        get_all API
+        """
+        # The get_all function for a SonicV2Connector does not support binary data due to which we
+        # have to use the redis Library. Relevant Issue: https://github.com/sonic-net/sonic-swss-common/issues/886
+        return redis.Redis(unix_socket_path=SonicDBConfig.getDbSock("APPL_DB", ns),
+                           db=SonicDBConfig.getDbId("APPL_DB", ns))
+
     def get(self, db_name, ns, update=False):
         """ Returns a SonicV2Connector Object and caches it for further requests """
         if ns not in self.cache:
             self.cache[ns] = {}
+        if CONN not in self.cache[ns]:
             self.cache[ns][CONN] = self.initialize_connector(ns)
+        if CONN_TO not in self.cache[ns]:
             self.cache[ns][CONN_TO] = set()
         if update or db_name not in self.cache[ns][CONN_TO]:
             self.cache[ns][CONN].connect(db_name)
             self.cache[ns][CONN_TO].add(db_name)
         return self.cache[ns][CONN]
 
+    def get_dash_conn(self, ns):
+        """ Returns a Redis Connection Object and caches it for further requests """
+        if ns not in self.cache:
+            self.cache[ns] = {}
+        if "DASH_"+CONN not in self.cache[ns]:
+            self.cache[ns]["DASH_"+CONN] = self.initialize_redis_conn(ns)
+        return self.cache[ns]["DASH_"+CONN]
+
     def clear(self, namespace=None):
         if not namespace:
             self.cache.clear()
         elif namespace in self.cache:
             del self.cache[namespace]
 
-    def fill(self, ns, conn, connected_to):
+    def fill(self, ns, conn, connected_to, dash_object=False):
         """ Update internal cache """
-        self.cache[ns] = {CONN: conn, CONN_TO: set(connected_to)}
+        if ns not in self.cache:
+            self.cache[ns] = {}
+        if dash_object:
+            self.cache[ns]["DASH_"+CONN] = conn
+            return
+        self.cache[ns][CONN] = conn
+        self.cache[ns][CONN_TO] = set(connected_to)
 
 
 class MatchEngine:
@@ -293,10 +368,16 @@ def get_redis_source_adapter(self):
     def get_json_source_adapter(self):
         return JsonSource()
 
+    def get_redis_py_adapter(self, pb_obj):
+        return RedisPySource(self.conn_pool, pb_obj)
+
     def __get_source_adapter(self, req):
         src = None
         d_src = ""
-        if req.db:
+        if req.PbObj:
+            d_src = req.db
+            src = self.get_redis_py_adapter(req.PbObj)
+        elif req.db:
             d_src = req.db
             src = self.get_redis_source_adapter()
         else:

dump/plugins/__init__.py

@@ -1,15 +1,22 @@
 import os
-import sys
 import pkgutil
 import importlib
 from .executor import Executor
+from sonic_py_common.syslogger import SysLogger
 
 dump_modules = {}
 pkg_dir = os.path.dirname(__file__)
 
+log = SysLogger()
 # import child classes automatically
 for (module_loader, name, ispkg) in pkgutil.iter_modules([pkg_dir]):
-    importlib.import_module('.' + name, __package__)
+    try:
+        importlib.import_module('.' + name, __package__)
+    except ModuleNotFoundError as e:
+        if e.name != "dash_api":
+            # dash_api is only used in a specific platform
+            log.log_debug("dump utility - dash_api package not found for platform")
+            raise
 
 # Classes inheriting Executor
 dump_modules = {cls.__name__.lower(): cls for cls in Executor.__subclasses__()}

dump/plugins/dash_acl_group.py

@@ -0,0 +1,41 @@
+from dump.helper import create_template_dict
+from dump.match_infra import MatchRequest
+from swsscommon.swsscommon import SonicDBConfig
+from dash_api.acl_group_pb2 import AclGroup
+from .executor import Executor
+
+
+APPL_DB_SEPARATOR = SonicDBConfig.getSeparator("APPL_DB")
+
+
+class Dash_Acl_Group(Executor):
+    """
+    Debug Dump Plugin for DASH ACL Group
+    """
+    ARG_NAME = "dash_acl_group_name"
+
+    def __init__(self, match_engine=None):
+        super().__init__(match_engine)
+        self.is_dash_object = True
+
+    def get_all_args(self, ns=""):
+        req = MatchRequest(db="APPL_DB", table="DASH_ACL_GROUP_TABLE", key_pattern="*", ns=ns)
+        ret = self.match_engine.fetch(req)
+        appliance_tables = ret["keys"]
+        return [key.split(APPL_DB_SEPARATOR)[-1] for key in appliance_tables]
+
+    def execute(self, params):
+        self.ret_temp = create_template_dict(dbs=["APPL_DB"])
+        dash_acl_group_table_name = params[self.ARG_NAME]
+        self.ns = params["namespace"]
+        self.init_dash_acl_group_table_appl_info(dash_acl_group_table_name)
+        return self.ret_temp
+
+    def init_dash_acl_group_table_appl_info(self, dash_acl_group_table_name):
+        req = MatchRequest(db="APPL_DB", table="DASH_ACL_GROUP_TABLE",
+                           key_pattern=dash_acl_group_table_name, ns=self.ns)
+        ret = self.match_engine.fetch(req)
+        self.add_to_ret_template(req.table, req.db, ret["keys"], ret["error"])
+
+    def return_pb2_obj(self):
+        return AclGroup()