Skip to content

Commit 4d07bbb

Browse files
arlakshmarlakshm
authored
[Yang][cfggen] update sonic-cfggen to generate config_db from Yang data (#7712)
Why I did it This PR adds changes in sonic-config-engine to consume configuration data in SONiC Yang schema and generate config_db entries How I did it Add a new file sonic_yang_cfg_generator . This file has the functions to parse yang data json and convert them in config_db json format. Validate the converted config_db entries to make sure all the dependencies and constraints are met. Add a new option -Y to the sonic-cfggen command for this purpose Add unit tests This capability is support only in sonic-config-engine Python3 package only
1 parent 714894c commit 4d07bbb

File tree

11 files changed

+800
-38
lines changed

11 files changed

+800
-38
lines changed

‎files/build_templates/sonic_debian_extension.j2

+16-15
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,22 @@ if [[ $CONFIGURED_ARCH == armhf || $CONFIGURED_ARCH == arm64 ]]; then
164164
sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install libxslt-dev libz-dev
165165
fi
166166

167+
# Install sonic-yang-models Python 3 package, install dependencies
168+
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/libyang_*.deb
169+
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/libyang-cpp_*.deb
170+
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/python2-yang_*.deb
171+
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/python3-yang_*.deb
172+
SONIC_YANG_MODEL_PY3_WHEEL_NAME=$(basename {{sonic_yang_models_py3_wheel_path}})
173+
sudo cp {{sonic_yang_models_py3_wheel_path}} $FILESYSTEM_ROOT/$SONIC_YANG_MODEL_PY3_WHEEL_NAME
174+
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install $SONIC_YANG_MODEL_PY3_WHEEL_NAME
175+
sudo rm -rf $FILESYSTEM_ROOT/$SONIC_YANG_MODEL_PY3_WHEEL_NAME
176+
177+
# Install sonic-yang-mgmt Python3 package
178+
SONIC_YANG_MGMT_PY3_WHEEL_NAME=$(basename {{sonic_yang_mgmt_py3_wheel_path}})
179+
sudo cp {{sonic_yang_mgmt_py3_wheel_path}} $FILESYSTEM_ROOT/$SONIC_YANG_MGMT_PY3_WHEEL_NAME
180+
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install $SONIC_YANG_MGMT_PY3_WHEEL_NAME
181+
sudo rm -rf $FILESYSTEM_ROOT/$SONIC_YANG_MGMT_PY3_WHEEL_NAME
182+
167183
# Install SONiC config engine Python 2 package
168184
CONFIG_ENGINE_PY2_WHEEL_NAME=$(basename {{config_engine_py2_wheel_path}})
169185
sudo cp {{config_engine_py2_wheel_path}} $FILESYSTEM_ROOT/$CONFIG_ENGINE_PY2_WHEEL_NAME
@@ -184,21 +200,6 @@ sudo cp {{config_engine_py3_wheel_path}} $FILESYSTEM_ROOT/$CONFIG_ENGINE_PY3_WHE
184200
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install $CONFIG_ENGINE_PY3_WHEEL_NAME
185201
sudo rm -rf $FILESYSTEM_ROOT/$CONFIG_ENGINE_PY3_WHEEL_NAME
186202

187-
# Install sonic-yang-models py3 package, install dependencies
188-
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/libyang_*.deb
189-
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/libyang-cpp_*.deb
190-
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/python2-yang_*.deb
191-
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/python3-yang_*.deb
192-
SONIC_YANG_MODEL_PY3_WHEEL_NAME=$(basename {{sonic_yang_models_py3_wheel_path}})
193-
sudo cp {{sonic_yang_models_py3_wheel_path}} $FILESYSTEM_ROOT/$SONIC_YANG_MODEL_PY3_WHEEL_NAME
194-
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install $SONIC_YANG_MODEL_PY3_WHEEL_NAME
195-
sudo rm -rf $FILESYSTEM_ROOT/$SONIC_YANG_MODEL_PY3_WHEEL_NAME
196-
197-
# Install sonic-yang-mgmt Python3 package
198-
SONIC_YANG_MGMT_PY3_WHEEL_NAME=$(basename {{sonic_yang_mgmt_py3_wheel_path}})
199-
sudo cp {{sonic_yang_mgmt_py3_wheel_path}} $FILESYSTEM_ROOT/$SONIC_YANG_MGMT_PY3_WHEEL_NAME
200-
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install $SONIC_YANG_MGMT_PY3_WHEEL_NAME
201-
sudo rm -rf $FILESYSTEM_ROOT/$SONIC_YANG_MGMT_PY3_WHEEL_NAME
202203

203204
# Install sonic-platform-common Python 2 package
204205
PLATFORM_COMMON_PY2_WHEEL_NAME=$(basename {{platform_common_py2_wheel_path}})

‎rules/docker-config-engine-buster.mk

+8-2
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,15 @@
33
DOCKER_CONFIG_ENGINE_BUSTER = docker-config-engine-buster.gz
44
$(DOCKER_CONFIG_ENGINE_BUSTER)_PATH = $(DOCKERS_PATH)/docker-config-engine-buster
55

6-
$(DOCKER_CONFIG_ENGINE_BUSTER)_DEPENDS += $(LIBSWSSCOMMON) $(PYTHON3_SWSSCOMMON)
6+
$(DOCKER_CONFIG_ENGINE_BUSTER)_DEPENDS += $(LIBSWSSCOMMON) \
7+
$(LIBYANG) \
8+
$(LIBYANG_CPP) \
9+
$(LIBYANG_PY3) \
10+
$(PYTHON3_SWSSCOMMON)
711
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SWSSSDK_PY3)
8-
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY3)
12+
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY3) \
13+
$(SONIC_YANG_MGMT_PY3) \
14+
$(SONIC_YANG_MODELS_PY3)
915
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_CONFIG_ENGINE_PY3)
1016
$(DOCKER_CONFIG_ENGINE_BUSTER)_LOAD_DOCKERS += $(DOCKER_BASE_BUSTER)
1117
$(DOCKER_CONFIG_ENGINE_BUSTER)_FILES += $(SWSS_VARS_TEMPLATE)

‎rules/sonic-config.mk

+7-2
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,13 @@ SONIC_PYTHON_WHEELS += $(SONIC_CONFIG_ENGINE_PY2)
1111

1212
SONIC_CONFIG_ENGINE_PY3 = sonic_config_engine-1.0-py3-none-any.whl
1313
$(SONIC_CONFIG_ENGINE_PY3)_SRC_PATH = $(SRC_PATH)/sonic-config-engine
14-
$(SONIC_CONFIG_ENGINE_PY3)_DEPENDS += $(SONIC_PY_COMMON_PY3)
15-
$(SONIC_CONFIG_ENGINE_PY3)_DEBS_DEPENDS += $(PYTHON3_SWSSCOMMON)
14+
$(SONIC_CONFIG_ENGINE_PY3)_DEPENDS += $(SONIC_PY_COMMON_PY3) \
15+
$(SONIC_YANG_MGMT_PY3) \
16+
$(SONIC_YANG_MODELS_PY3)
17+
$(SONIC_CONFIG_ENGINE_PY3)_DEBS_DEPENDS += $(LIBYANG) \
18+
$(LIBYANG_CPP) \
19+
$(LIBYANG_PY3) \
20+
$(PYTHON3_SWSSCOMMON)
1621
# Synthetic dependency to avoid building the Python 2 and 3 packages
1722
# simultaneously and any potential conflicts which may arise
1823
$(SONIC_CONFIG_ENGINE_PY3)_DEPENDS += $(SONIC_CONFIG_ENGINE_PY2)

‎rules/sonic_bgpcfgd.mk

+7-2
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,12 @@ $(SONIC_BGPCFGD)_SRC_PATH = $(SRC_PATH)/sonic-bgpcfgd
77
# as part of its unit tests.
88
# TODO: Refactor unit tests so that these dependencies are not needed
99

10-
$(SONIC_BGPCFGD)_DEPENDS += $(SONIC_CONFIG_ENGINE_PY3)
11-
$(SONIC_BGPCFGD)_DEBS_DEPENDS += $(PYTHON3_SWSSCOMMON)
10+
$(SONIC_BGPCFGD)_DEPENDS += $(SONIC_CONFIG_ENGINE_PY3) \
11+
$(SONIC_YANG_MGMT_PY3) \
12+
$(SONIC_YANG_MODELS_PY3)
13+
$(SONIC_BGPCFGD)_DEBS_DEPENDS += $(LIBYANG) \
14+
$(LIBYANG_CPP) \
15+
$(LIBYANG_PY3) \
16+
$(PYTHON3_SWSSCOMMON)
1217
$(SONIC_BGPCFGD)_PYTHON_VERSION = 3
1318
SONIC_PYTHON_WHEELS += $(SONIC_BGPCFGD)

‎src/sonic-config-engine/setup.py

+18-9
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,9 @@
2020
# Python3 has enum module and so pyangbind should be installed outside
2121
# dependencies section of setuptools followed by uninstall of enum43
2222
# 'pyangbind==0.8.1',
23-
'Jinja2>=2.10'
23+
'Jinja2>=2.10',
24+
'sonic-yang-mgmt>=1.0',
25+
'sonic-yang-models>=1.0'
2426
]
2527
else:
2628
# Python 2-only dependencies
@@ -34,6 +36,20 @@
3436
'importlib-resources==3.3.1' # importlib-resources v4.0.0 was released 2020-12-23 and drops support for Python 2
3537
]
3638

39+
# Common modules for python2 and python3
40+
py_modules = [
41+
'config_samples',
42+
'lazy_re',
43+
'minigraph',
44+
'openconfig_acl',
45+
'portconfig',
46+
'redis_bcc',
47+
]
48+
if sys.version_info.major == 3:
49+
# Python 3-only modules
50+
py_modules += [
51+
'sonic_yang_cfg_generator'
52+
]
3753

3854
setup(
3955
name = 'sonic-config-engine',
@@ -42,14 +58,7 @@
4258
author = 'Taoyu Li',
4359
author_email = '[email protected]',
4460
url = 'https://github.com/Azure/sonic-buildimage',
45-
py_modules = [
46-
'config_samples',
47-
'lazy_re',
48-
'minigraph',
49-
'openconfig_acl',
50-
'portconfig',
51-
'redis_bcc',
52-
],
61+
py_modules = py_modules,
5362
scripts = [
5463
'sonic-cfggen',
5564
],

‎src/sonic-config-engine/sonic-cfggen

+15-1
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,11 @@ from swsscommon.swsscommon import SonicV2Connector, ConfigDBConnector, SonicDBCo
4848

4949
PY3x = sys.version_info >= (3, 0)
5050

51-
#TODO: Remove STR_TYPE, FILE_TYPE once SONiC moves to Python 3.x
51+
# TODO: Remove STR_TYPE, FILE_TYPE once SONiC moves to Python 3.x
52+
# TODO: Remove the import SonicYangCfgDbGenerator once SONiC moves to python3.x
5253
if PY3x:
5354
from io import IOBase
55+
from sonic_yang_cfg_generator import SonicYangCfgDbGenerator
5456
STR_TYPE = str
5557
FILE_TYPE = IOBase
5658
else:
@@ -258,6 +260,7 @@ def main():
258260
parser=argparse.ArgumentParser(description="Render configuration file from minigraph data and jinja2 template.")
259261
group = parser.add_mutually_exclusive_group()
260262
group.add_argument("-m", "--minigraph", help="minigraph xml file", nargs='?', const='/etc/sonic/minigraph.xml')
263+
group.add_argument("-Y", "--yang", help="yang data json file", nargs='?', const='/etc/sonic/config_yang.json')
261264
group.add_argument("-M", "--device-description", help="device description xml file")
262265
group.add_argument("-k", "--hwsku", help="HwSKU")
263266
parser.add_argument("-n", "--namespace", help="namespace name", nargs='?', const=None, default=None)
@@ -325,6 +328,17 @@ def main():
325328

326329
_process_json(args, data)
327330

331+
if args.yang is not None:
332+
#TODO: Remove this check onces SONiC moves to python3.x
333+
if PY3x:
334+
yang_file = args.yang
335+
config_db_json = SonicYangCfgDbGenerator().generate_config(
336+
yang_data_file=yang_file)
337+
deep_update(data, config_db_json)
338+
else:
339+
print('-Y/--yang option is not available in Python2', file=sys.stderr)
340+
sys.exit(1)
341+
328342
if args.minigraph is not None:
329343
minigraph = args.minigraph
330344
if platform:

‎src/sonic-config-engine/sonic_yang_cfg_generator.py

+60
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
"""sonic_yang_cfg_generator
2+
version_added: "1.0"
3+
author: Arvindsrinivasan Lakshmi naraismhan ([email protected])
4+
short_description: Parse sonic_yang data and generate the config_db entries
5+
"""
6+
7+
from __future__ import print_function
8+
9+
import json
10+
import sys
11+
12+
13+
import sonic_yang
14+
# TODO: Remove this once we no longer support Python 2
15+
if sys.version_info.major == 3:
16+
UNICODE_TYPE = str
17+
else:
18+
UNICODE_TYPE = unicode
19+
20+
YANG_MODELS_DIR = "/usr/local/yang-models"
21+
DEFAULT_YANG_DATA_FILE = "/etc/sonic/config_yang.json"
22+
23+
24+
class SonicYangCfgDbGenerator:
25+
26+
def __init__(self, yang_models_dir=YANG_MODELS_DIR):
27+
self.yang_models_dir = yang_models_dir
28+
self.yang_parser = sonic_yang.SonicYang(self.yang_models_dir)
29+
self.yang_parser.loadYangModel()
30+
31+
def get_config_db_from_yang_data(self,
32+
yang_data_file=DEFAULT_YANG_DATA_FILE):
33+
self.yang_data_file = yang_data_file
34+
config_db_json = dict()
35+
with open(self.yang_data_file, "r") as yang_file:
36+
try:
37+
self.yang_data = json.load(yang_file)
38+
config_db_json = self.yang_parser.XlateYangToConfigDB(
39+
yang_data=self.yang_data)
40+
except json.JSONDecodeError as e:
41+
print("Unable to parse Yang data file {} Error: {}".format(
42+
yang_data_file, e))
43+
return config_db_json
44+
45+
def validate_config_db_json(self, config_db_json):
46+
self.yang_parser.loadData(configdbJson=config_db_json)
47+
try:
48+
self.yang_parser.validate_data_tree()
49+
return True
50+
except sonic_yang.SonicYangException as e:
51+
print("yang data in {} is not valid".format(self.yang_data_file))
52+
return False
53+
54+
def generate_config(self, yang_data_file=DEFAULT_YANG_DATA_FILE):
55+
config_db_json = self.get_config_db_from_yang_data(
56+
yang_data_file=yang_data_file)
57+
if self.validate_config_db_json(config_db_json):
58+
return config_db_json
59+
else:
60+
return {}

0 commit comments

Comments
 (0)