Greatly simplify qemu setup

Author: leezu

ci/README.md

@@ -111,90 +111,37 @@ significantly. You can set this directory explicitly by setting CCACHE_DIR envir
 variable. All ccache instances are currently set to be 10 Gigabytes max in size.
 
 
-## Testing with QEMU
-To run the unit tests under qemu:
-```
-./build.py -p armv7 && ./build.py -p test.arm_qemu ./runtime_functions.py run_ut_py3_qemu
-```
-
-To get a shell on the container and debug issues with the emulator itself, we build the container
-and then execute it interactively. We can afterwards use port 2222 on the host to connect with SSH.
-
-
-```
-ci/build.py -p test.arm_qemu -b && docker run -p2222:2222 -ti mxnetci/build.test.arm_qemu
-```
+## Testing with ARM / Edge devices with QEMU
 
-Then from another terminal:
+We build on [QEMU](https://www.qemu.org/) and Linux [Kernel Support for
+miscellaneous Binary
+Formats](https://www.kernel.org/doc/html/v5.6/admin-guide/binfmt-misc.html) for
+testing MXNet on edge devices. Test can be invoked with the same syntax as for
+non-virtualized platforms:
 
 ```
-ssh -o StrictHostKeyChecking=no -p 2222 qemu@localhost
+./build.py -p armv7
+./build.py -p test.armv7 /work/runtime_functions.sh unittest_ubuntu_python3_armv7
 ```
 
-There are two pre-configured users: `root` and `qemu` both without passwords.
-
-
-### Example of reproducing a test result with QEMU on ARM
-
-
-You might want to enable a debug build first:
-
-```
-$ git diff
-diff --git a/ci/docker/runtime_functions.sh b/ci/docker/runtime_functions.sh
-index 39631f9..666ceea 100755
---- a/ci/docker/runtime_functions.sh
-+++ b/ci/docker/runtime_functions.sh
-@@ -172,6 +172,7 @@ build_armv7() {
-         -DUSE_LAPACK=OFF \
-         -DBUILD_CPP_EXAMPLES=OFF \
-         -Dmxnet_LINKER_LIBS=-lgfortran \
-+        -DCMAKE_BUILD_TYPE=Debug \
-         -G Ninja /work/mxnet
-
-     ninja -v
+For the test step to succeed, you must run Linux kernel 4.8 or later and have qemu installed.
 
+On Debian and Ubuntu systems, run the following command to install the dependencies:
 ```
+sudo apt install binfmt-support qemu-user-static
 
-Then we build the project for armv7, the test container and start QEMU inside docker:
-
-```
-ci/build.py -p armv7
-ci/build.py -p test.arm_qemu -b && docker run -p2222:2222 -ti mxnetci/build.test.arm_qemu
+# Use qemu-binfmt-conf.sh to register all binary types with the kernel
+wget https://raw.githubusercontent.com/qemu/qemu/stable-4.1/scripts/qemu-binfmt-conf.sh
+chmod +x qemu-binfmt-conf.sh
+sudo ./qemu-binfmt-conf.sh --persistent yes --qemu-suffix "-static" --qemu-path "/usr/bin" --systemd ALL
 ```
 
-
-
-At this point we copy artifacts and sources to the VM, in another terminal (host) do the following:
+If you run into segmentation faults at the beginning of the emulated tests, you
+probably have a ancient version of Qemu on your system (or found a bug in
+upstream Qemu). In that situation, you can rely on the
+`multiarch/qemu-user-static` Docker project to register a set of up-to-date Qemu
+binaries from their Docker image with your kernel:
 
 ```
-# Copy mxnet sources to the VM
-rsync --delete -e 'ssh -p2222' --exclude='.git/' -zvaP ./ qemu@localhost:mxnet
-
-
-# Ssh into the vm
-ssh -p2222 qemu@localhost
-
-cd mxnet
-
-# Execute a single failing C++ test
-build/tests/mxnet_unit_tests --gtest_filter="ACTIVATION_PERF.ExecuteBidirectional"
-
-# To install MXNet:
-sudo pip3 install --upgrade --force-reinstall build/mxnet-1.3.1-py2.py3-none-any.whl
-
-# Execute a single python test:
-
-nosetests-3.4 -v -s tests/python/unittest/test_ndarray.py
-
-
-# Debug with cgdb
-sudo apt install -y libstdc++6-6-dbg
-cgdb build/tests/mxnet_unit_tests
-
-(gdb) !pwd
-/home/qemu/mxnet
-(gdb) set substitute-path /work /home/qemu
-(gdb) set substitute-path /build/gcc-6-6mK9AW/gcc-6-6.3.0/build/arm-linux-gnueabihf/libstdc++-v3/include/ /usr/include/c++/6/
-(gdb) r --gtest_filter="ACTIVATION_PERF.ExecuteBidirectional"
+docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
 ```

ci/dev_menu.py

@@ -167,7 +167,7 @@ def provision_virtualenv(venv_path=DEFAULT_PYENV):
     ('[Docker] Python3 ARMv7 unittests (QEMU)',
     [
         "ci/build.py -p armv7",
-        "ci/build.py -p test.arm_qemu ./runtime_functions.py run_ut_py3_qemu"
+        "ci/build.py -p test.armv7 /work/runtime_functions.sh unittest_ubuntu_python3_armv7"
     ]),
     ('Clean (RESET HARD) repository (Warning! erases local changes / DATA LOSS)',
        Confirm("ci/docker/runtime_functions.sh clean_repo"))

ci/docker/Dockerfile.build.android_armv7

@@ -46,10 +46,10 @@ RUN git clone --recursive -b v0.3.9 https://github.com/xianyi/OpenBLAS.git && \
     mkdir /usr/local/openblas-android && \
     cd /usr/local/OpenBLAS && \
     export TOOLCHAIN=/usr/local/android-ndk-r19/toolchains/llvm/prebuilt/linux-x86_64 && \
-    make NOFORTRAN=1 ARM_SOFTFP_ABI=1 \
+    make NOFORTRAN=1 ARM_SOFTFP_ABI=1 NO_SHARED=1 \
         LDFLAGS="-L/usr/local/android-ndk-r19/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/lib/gcc/arm-linux-androideabi/4.9.x -lm" \
         CC=$TOOLCHAIN/bin/armv7a-linux-androideabi16-clang AR=$TOOLCHAIN/bin/arm-linux-androideabi-ar && \
-    make PREFIX=/usr/local/openblas-android install && \
+    make PREFIX=/usr/local/openblas-android NO_SHARED=1 install && \
     cd /usr/local && \
     rm -rf OpenBLAS
 ENV OpenBLAS_HOME=/usr/local/openblas-android

ci/docker/Dockerfile.build.android_armv8

@@ -46,10 +46,10 @@ RUN git clone --recursive -b v0.3.9 https://github.com/xianyi/OpenBLAS.git && \
     mkdir /usr/local/openblas-android && \
     cd /usr/local/OpenBLAS && \
     export TOOLCHAIN=/usr/local/android-ndk-r19/toolchains/llvm/prebuilt/linux-x86_64 && \
-    make NOFORTRAN=1 \
+    make NOFORTRAN=1 NO_SHARED=1 \
         LDFLAGS="-L/usr/local/android-ndk-r21/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/lib/gcc/aarch64-linux-android/4.9.x -lm" \
         CC=$TOOLCHAIN/bin/aarch64-linux-android21-clang AR=$TOOLCHAIN/bin/aarch64-linux-android-ar && \
-    make PREFIX=/usr/local/openblas-android install && \
+    make PREFIX=/usr/local/openblas-android NO_SHARED=1 install && \
     cd /usr/local && \
     rm -rf OpenBLAS
 ENV OpenBLAS_HOME=/usr/local/openblas-android

ci/docker/Dockerfile.build.armv6

@@ -50,8 +50,8 @@ ENV CMAKE_TOOLCHAIN_FILE=/usr/local/armv6-eabihf--glibc--stable-2020.02-2/share/
 
 RUN git clone --recursive -b v0.3.9 https://github.com/xianyi/OpenBLAS.git && \
     cd /usr/local/OpenBLAS && \
-    make NOFORTRAN=1 CC=/usr/local/armv6-eabihf--glibc--stable-2020.02-2/bin/arm-linux-gcc && \
-    make PREFIX=/usr/local/armv6-eabihf--glibc--stable-2020.02-2/arm-buildroot-linux-gnueabihf/sysroot install && \
+    make NOFORTRAN=1 NO_SHARED=1 CC=/usr/local/armv6-eabihf--glibc--stable-2020.02-2/bin/arm-linux-gcc && \
+    make PREFIX=/usr/local/armv6-eabihf--glibc--stable-2020.02-2/arm-buildroot-linux-gnueabihf/sysroot NO_SHARED=1 install && \
     cd /usr/local && \
     rm -rf OpenBLAS
 

ci/docker/Dockerfile.build.armv7

@@ -45,8 +45,8 @@ ENV CMAKE_TOOLCHAIN_FILE=/usr/local/arm-linux-gnueabihf-toolchain.cmake
 
 RUN git clone --recursive -b v0.3.9 https://github.com/xianyi/OpenBLAS.git && \
     cd /usr/local/OpenBLAS && \
-    make NOFORTRAN=1 CC=arm-linux-gnueabihf-gcc && \
-    make PREFIX=/usr/local/arm-linux-gnueabihf install && \
+    make NOFORTRAN=1 NO_SHARED=1 CC=arm-linux-gnueabihf-gcc && \
+    make PREFIX=/usr/local/arm-linux-gnueabihf NO_SHARED=1 install && \
     cd /usr/local && \
     rm -rf OpenBLAS
 

ci/docker/Dockerfile.build.armv8

@@ -45,8 +45,8 @@ ENV CMAKE_TOOLCHAIN_FILE=/usr/aarch64-linux-gnu-toolchain.cmake
 
 RUN git clone --recursive -b v0.3.9 https://github.com/xianyi/OpenBLAS.git && \
     cd /usr/local/OpenBLAS && \
-    make NOFORTRAN=1 CC=aarch64-linux-gnu-gcc && \
-    make PREFIX=/usr/aarch64-linux-gnu install && \
+    make NOFORTRAN=1 NO_SHARED=1 CC=aarch64-linux-gnu-gcc && \
+    make PREFIX=/usr/aarch64-linux-gnu NO_SHARED=1 install && \
     cd /usr/local && \
     rm -rf OpenBLAS
 

ci/docker/Dockerfile.build.test.arm_qemu renamed to ci/docker/Dockerfile.build.test.armv7

@@ -16,32 +16,26 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-# Dockerfile to build and run MXNet on Ubuntu 16.04 for CPU
+# Dockerfile to test MXNet on Ubuntu 20.04 ARMv7 CPU
 
-FROM ubuntu:16.04
+FROM arm32v7/ubuntu:20.04
 
-WORKDIR /work
+WORKDIR /usr/local
 
-RUN apt-get update
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
-COPY install/ubuntu_arm_qemu.sh /work
-RUN /work/ubuntu_arm_qemu.sh
-
-COPY install/ubuntu_arm_qemu_bin.sh /work
-RUN /work/ubuntu_arm_qemu_bin.sh
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
+    python3 \
+    python3-pip \
+    python3-numpy \
+    python3-scipy \
+    python3-nose \
+    python3-nose-timer \
+    python3-requests \
+ && rm -rf /var/lib/apt/lists/*
 
 ARG USER_ID=0
 ARG GROUP_ID=0
 COPY install/ubuntu_adduser.sh /work/
 RUN /work/ubuntu_adduser.sh
 
 COPY runtime_functions.sh /work/
-COPY qemu/* /work/
-
-# SSH to the Qemu VM
-EXPOSE 2222/tcp
-
-CMD ["./runtime_functions.py","run_qemu_interactive"]
+WORKDIR /work/mxnet

ci/docker/install/ubuntu_arm_qemu.sh renamed to ci/docker/Dockerfile.build.test.armv8

@@ -1,5 +1,4 @@
-#!/usr/bin/env bash
-
+# -*- mode: dockerfile -*-
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -16,22 +15,27 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+#
+# Dockerfile to test MXNet on Ubuntu 20.04 ARMv8 CPU
+
+FROM arm64v8/ubuntu:20.04
 
-# build and install are separated so changes to build don't invalidate
-# the whole docker cache for the image
+WORKDIR /usr/local
 
-set -exuo pipefail
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
+    python3 \
+    python3-pip \
+    python3-numpy \
+    python3-scipy \
+    python3-nose \
+    python3-nose-timer \
+    python3-requests \
+ && rm -rf /var/lib/apt/lists/*
 
-apt-get install -y \
-    cmake \
-    curl \
-    wget \
-    git \
-    qemu \
-    qemu-system-arm \
-    unzip \
-    bzip2 \
-    vim-nox \
-    toilet
+ARG USER_ID=0
+ARG GROUP_ID=0
+COPY install/ubuntu_adduser.sh /work/
+RUN /work/ubuntu_adduser.sh
 
-pip3 install ipython
+COPY runtime_functions.sh /work/
+WORKDIR /work/mxnet