Skip to content

Commit cc30771

Browse files
ZhaohuiSZhaohuiS
authored
Add python3 virtual environment for docker-ptf (#10599)
Why I did it Migrate ptftests script to python3, in order to do an incremental migration, add python virtual environment firstly, install all required python packages in virtual env as well. Then migrate ptftests scripts from python2 to python3 one by one avoid impacting non-changed scripts. Signed-off-by: Zhaohui Sun [email protected] How I did it Add python3 virtual environment for docker-ptf. Add submodule ptf-py3 and install patched ptf 0.9.3 into virtual environment as well, two ptf issues were reported here: p4lang/ptf#173 p4lang/ptf#174 Signed-off-by: Zhaohui Sun <[email protected]>
1 parent aa62e33 commit cc30771

9 files changed

+153
-1
lines changed

‎.gitmodules

+3
Original file line numberDiff line numberDiff line change
@@ -103,3 +103,6 @@
103103
[submodule "src/sonic-p4rt/sonic-pins"]
104104
path = src/sonic-p4rt/sonic-pins
105105
url = https://github.com/Azure/sonic-pins.git
106+
[submodule "src/ptf-py3"]
107+
path = src/ptf-py3
108+
url = https://github.com/p4lang/ptf.git

‎dockers/docker-ptf/Dockerfile.j2

+66-1
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,11 @@ FROM {{ prefix }}multiarch/debian-debootstrap:arm64-stretch
77
FROM {{ prefix }}debian:buster
88
{% endif %}
99

10+
{% from "dockers/dockerfile-macros.j2" import install_python_wheels, copy_files %}
11+
12+
USER root
13+
WORKDIR /root
14+
1015
MAINTAINER Pavel Shirshov
1116

1217
RUN echo "deb [arch=amd64] http://debian-archive.trafficmanager.net/debian buster-backports main" >> /etc/apt/sources.list
@@ -51,6 +56,13 @@ RUN sed --in-place 's/httpredir.debian.org/debian-archive.trafficmanager.net/' /
5156
python-libpcap \
5257
python-scapy \
5358
python-six \
59+
python3 \
60+
python3-venv \
61+
python3-pip \
62+
python3-dev \
63+
python3-scapy \
64+
python3-six \
65+
libpcap-dev \
5466
tacacs+ \
5567
rsyslog \
5668
ntp \
@@ -59,7 +71,9 @@ RUN sed --in-place 's/httpredir.debian.org/debian-archive.trafficmanager.net/' /
5971
arping \
6072
bridge-utils \
6173
libteam-utils \
62-
gdb
74+
gdb \
75+
automake \
76+
iproute2
6377

6478
# Install all python modules from pypi. python-scapy is exception, ptf debian package requires python-scapy
6579
# TODO: Clean up this step
@@ -117,10 +131,61 @@ RUN rm -rf /debs \
117131
&& pip install pybrctl pyro4 rpyc yabgp \
118132
&& pip install unittest-xml-reporting \
119133
&& pip install pyrasite \
134+
&& pip install retrying \
120135
&& mkdir -p /opt \
121136
&& cd /opt \
122137
&& wget https://raw.githubusercontent.com/p4lang/ptf/master/ptf_nn/ptf_nn_agent.py
123138

139+
RUN python3 -m venv env-python3
140+
141+
# Activating a virtualenv. The virtualenv automatically works for RUN, ENV and CMD.
142+
ENV VIRTUAL_ENV=/root/env-python3
143+
ARG BACKUP_OF_PATH="$PATH"
144+
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
145+
146+
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 PYTHONIOENCODING=UTF-8
147+
148+
RUN python3 -m pip install --upgrade --ignore-installed pip
149+
150+
# Install all python modules from pypi. python3-scapy is exception, ptf debian package requires python3-scapy
151+
RUN python3 -m pip install setuptools \
152+
&& pip3 install supervisor \
153+
&& pip3 install ipython==5.4.1 \
154+
&& pip3 install Cython \
155+
&& pip3 install cffi \
156+
&& pip3 install nnpy \
157+
&& pip3 install dpkt \
158+
&& pip3 install ipaddress \
159+
&& pip3 install pysubnettree \
160+
&& pip3 install paramiko \
161+
&& pip3 install Flask \
162+
&& pip3 install exabgp \
163+
&& pip3 install pyaml \
164+
&& pip3 install pybrctl pyro4 rpyc yabgp \
165+
&& pip3 install unittest-xml-reporting \
166+
&& pip3 install pyrasite \
167+
&& pip3 install python-libpcap \
168+
&& pip3 install enum34 \
169+
&& pip3 install grpcio \
170+
&& pip3 install grpcio-tools \
171+
&& pip3 install protobuf \
172+
&& pip3 install six==1.16.0 \
173+
&& pip3 install itsdangerous \
174+
&& pip3 install retrying \
175+
&& pip3 install jinja2 \
176+
&& pip3 install scapy==2.4.5
177+
178+
{% if docker_ptf_whls.strip() -%}
179+
# Copy locally-built Python wheel dependencies
180+
{{ copy_files("python-wheels/", docker_ptf_whls.split(' '), "/python-wheels/") }}
181+
182+
# Install locally-built Python wheel dependencies
183+
{{ install_python_wheels(docker_ptf_whls.split(' ')) }}
184+
{% endif %}
185+
186+
# Deactivating a virtualenv.
187+
ENV PATH="$BACKUP_OF_PATH"
188+
124189
## Adjust sshd settings
125190
RUN mkdir /var/run/sshd \
126191
&& echo 'root:root' | chpasswd \

‎platform/vs/docker-ptf.mk

+1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# docker image for docker-ptf
22

33
DOCKER_PTF = docker-ptf.gz
4+
$(DOCKER_PTF)_PYTHON_WHEELS += $(PTF_PY3)
45
$(DOCKER_PTF)_PATH = $(DOCKERS_PATH)/docker-ptf
56
$(DOCKER_PTF)_DEPENDS += $(LIBTHRIFT) $(PYTHON_THRIFT) $(PTF) $(PYTHON_SAITHRIFT)
67
SONIC_DOCKER_IMAGES += $(DOCKER_PTF)

‎rules/ptf-py3.dep

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
2+
SPATH := $($(PTF_PY3)_SRC_PATH)
3+
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/ptf-py3.mk rules/ptf-py3.dep
4+
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
5+
SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))
6+
7+
$(PTF_PY3)_CACHE_MODE := GIT_CONTENT_SHA
8+
$(PTF_PY3)_DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST)
9+
$(PTF_PY3)_DEP_FILES := $(DEP_FILES)
10+
$(PTF_PY3)_SMDEP_FILES := $(SMDEP_FILES)
11+
$(PTF_PY3)_SMDEP_PATHS := $(SPATH)

‎rules/ptf-py3.mk

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# ptf package
2+
3+
PTF_PY3 = ptf-0.9.3-py3-none-any.whl
4+
$(PTF_PY3)_SRC_PATH = $(SRC_PATH)/ptf-py3
5+
$(PTF_PY3)_PYTHON_VERSION = 3
6+
$(PTF_PY3)_TEST = n
7+
SONIC_PYTHON_WHEELS += $(PTF_PY3)

‎src/ptf-py3

Submodule ptf-py3 added at 405513b

‎src/ptf-py3.patch/0001-Remove-ord-in-get_mac-to-avoid-TypeError.patch

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
From a8b13b9fbaa16ddd305ba2df2238ef606ef222a7 Mon Sep 17 00:00:00 2001
2+
From: Zhaohui Sun <[email protected]>
3+
Date: Wed, 13 Apr 2022 09:24:46 +0000
4+
Subject: [PATCH 1/2] Remove ord in get_mac() to avoid TypeError
5+
6+
Signed-off-by: Zhaohui Sun <[email protected]>
7+
---
8+
src/ptf/netutils.py | 2 +-
9+
1 file changed, 1 insertion(+), 1 deletion(-)
10+
11+
diff --git a/src/ptf/netutils.py b/src/ptf/netutils.py
12+
index 752e76c..6aabe79 100644
13+
--- a/src/ptf/netutils.py
14+
+++ b/src/ptf/netutils.py
15+
@@ -54,7 +54,7 @@ def get_if_index(iff):
16+
17+
18+
def get_mac(iff):
19+
- return ":".join(["%02x" % ord(char) for char in get_if(iff, SIOCGIFHWADDR)[18:24]])
20+
+ return ":".join(["%02x" % char for char in get_if(iff, SIOCGIFHWADDR)[18:24]])
21+
22+
23+
def set_promisc(s, iff, val=1):
24+
--
25+
2.25.1
26+

‎src/ptf-py3.patch/0002-Fill-byte-formatted-client-mac-address-in-DHCP-Disco.patch

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
From 6e570e00ea05882d2db1e480ed041ea631bf37da Mon Sep 17 00:00:00 2001
2+
From: Zhaohui Sun <[email protected]>
3+
Date: Wed, 13 Apr 2022 09:25:28 +0000
4+
Subject: [PATCH 2/2] Fill byte formatted client mac address in DHCP Discover
5+
packet
6+
7+
Signed-off-by: Zhaohui Sun <[email protected]>
8+
---
9+
src/ptf/testutils.py | 9 +++++----
10+
1 file changed, 5 insertions(+), 4 deletions(-)
11+
12+
diff --git a/src/ptf/testutils.py b/src/ptf/testutils.py
13+
index ab67cea..83a9075 100755
14+
--- a/src/ptf/testutils.py
15+
+++ b/src/ptf/testutils.py
16+
@@ -2712,12 +2712,13 @@ def __dhcp_mac_to_chaddr(mac_addr="00:01:02:03:04:05"):
17+
"""
18+
Private helper function to convert a 6-byte MAC address of form:
19+
'00:01:02:03:04:05'
20+
- into a 16-byte chaddr byte string of form:
21+
- '\x00\x01\x02\x03\x04\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
22+
+ into a 16-byte chaddr byte of form:
23+
+ b'\x00\x01\x02\x03\x04\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
24+
25+
"""
26+
- chaddr = "".join([chr(int(octet, 16)) for octet in mac_addr.split(":")])
27+
- chaddr += "\x00" * 10
28+
+ import binascii
29+
+ chaddr = binascii.unhexlify(mac_addr.replace(':', ''))
30+
+ chaddr += b'\x00\x00\x00\x00\x00\x00'
31+
return chaddr
32+
33+
34+
--
35+
2.25.1
36+

‎src/ptf-py3.patch/series

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
0001-Remove-ord-in-get_mac-to-avoid-TypeError.patch
2+
0002-Fill-byte-formatted-client-mac-address-in-DHCP-Disco.patch

0 commit comments

Comments
 (0)