docker builds broken on armv7

[richvdh]

Our Docker build is currently broken on armv7, since python-cryptography now requires a rust compiler to build:

From https://app.circleci.com/pipelines/github/matrix-org/synapse/19720/workflows/47040f65-8516-4995-b250-9cae722e4efc/jobs/25662:

#25 5590.   Building wheel for cryptography (PEP 517): started
#25 5605.   Building wheel for cryptography (PEP 517): finished with status 'error'
#25 5605.   ERROR: Command errored out with exit status 1:
#25 5605.    command: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpyl06y7df
#25 5605.        cwd: /tmp/pip-install-mmoy6v11/cryptography_468fe58643ff4d02b4439954d410409c
...
#25 5605.   generating cffi module 'build/temp.linux-armv7l-3.8/_openssl.c'
#25 5605.   running build_rust
#25 5605.   
#25 5605.       =============================DEBUG ASSISTANCE=============================
#25 5605.       If you are seeing a compilation error please try the following steps to
#25 5605.       successfully install cryptography:
#25 5605.       1) Upgrade to the latest pip and try again. This will fix errors for most
#25 5605.          users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
#25 5605.       2) Read https://cryptography.io/en/latest/installation.html for specific
#25 5605.          instructions for your platform.
#25 5605.       3) Check our frequently asked questions for more information:
#25 5605.          https://cryptography.io/en/latest/faq.html
#25 5605.       4) Ensure you have a recent Rust toolchain installed:
#25 5605.          https://cryptography.io/en/latest/installation.html#rust
#25 5605.       5) If you are experiencing issues with Rust for *this release only* you may
#25 5605.          set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
#25 5605.       =============================DEBUG ASSISTANCE=============================
#25 5605.   
#25 5605.   error: Can not find Rust compiler
#25 5605.   ----------------------------------------
#25 5605.   ERROR: Failed building wheel for cryptography

amd64 and arm64 are fine, because we use precompiled wheels from PyPI.

(see also pyca/cryptography#5771)

[richvdh]

The immediate fix is presumably either to pin cryptography or set CRYPTOGRAPHY_DONT_BUILD_RUST=1, but longer term I guess we need to install a (modern) rust compiler in the dockerfile.

[callahad]

...or drop ARMv7 Docker images? Do we have any usage stats?

[callahad]

Hm. Apparently Raspberry Pis still ship with a 32-bit (ARMv7) kernel, even if the host processor is 64-bit, so we do need to get armv7 builds working again.

[richvdh]

if our goal is simply targetting raspberry pis, we could plausibly pull wheels from https://www.piwheels.org/.

[callahad]

Okay, let's:

• Pin cryptography to 3.3.2 to solve our immediate problems for 1.27 and 1.28 and close this bug.
• File a separate bug to unpin cryptography and add rustc to our build Dockerfiles for 1.29+, labeled as T-Task and P1.

Of note, our builds use the python:3.8-slim image, which is based on Debian 10 (buster). This ships with Rust 1.41, which has issues building cryptography. It sounds like those are being worked out upstream. I have an ARM box, so I'm happy to have the second issue assigned to me as I can probably iterate on it more quickly.

[callahad]

Cryptography 3.4.5 has reduced its minimum supported Rust version to 1.41, so we should be fine on that front.

[callahad]

Actually, apt install build-essential libssl-dev libffi-dev rustc is sufficient to get pip install --no-binary :all: cryptography==3.4.5 to work inside a python:3.8-slim image; rustc is the only item missing from our current Dockerfile.

...so I suspect we can just jam rustc in there and we're good to go.

[clokep]

I think this is fixed?

[callahad]

Oh how I wish it were.

Building Docker images on native ARMv7 hosts should work just fine.

Cross-building with qemu fails, which means normal docker buildx as used in our CircleCI configuration fails.

Erik traced it to an upstream bug in qemu which has been known for a couple of years.

Erik also proposed a patch to our Dockerfile with the caveat "but that breaks standard docker builds :/" However, it does seem to work fine for me locally for both buildx and normal builds.

...but do we want to carry this debt, versus just dropping ARMv7l?

diff --git a/docker/Dockerfile b/docker/Dockerfile
index d619ee08e..289bc3c74 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -14,7 +14,35 @@
 ARG PYTHON_VERSION=3.8
 
 ###
-### Stage 0: builder
+### Stage: prebuilder
+###
+### New versions of cryptography require rust to be installed, but due to
+### https://bugs.launchpad.net/qemu/+bug/1805913 downloading cargo registries
+### and dependencies doesn't work on multiarch builds. We fix this by
+### prebuilding `cryptography` first in the native arch, and then copying over
+### the registry and running cargo in offline mode.
+###
+FROM --platform=$BUILDPLATFORM docker.io/python:${PYTHON_VERSION}-slim as prebuilder
+
+RUN apt-get update && apt-get install -y \
+        build-essential \
+        libffi-dev \
+        libssl-dev \
+        rustc \
+        zlib1g-dev \
+        && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir /root/.cargo
+ENV CARGO_HOME=/root/.cargo
+
+RUN pip download --no-binary :all: --no-deps cryptography
+
+RUN tar -xf cryptography*.tar.gz --wildcards cryptography*/src/rust/
+
+RUN cd cryptography*/src/rust && cargo fetch
+
+###
+### Stage: builder
 ###
 FROM docker.io/python:${PYTHON_VERSION}-slim as builder
 
@@ -32,6 +60,14 @@ RUN apt-get update && apt-get install -y \
     zlib1g-dev \
  && rm -rf /var/lib/apt/lists/*
 
+COPY --from=prebuilder /root/.cargo /root/.cargo
+
+ENV CARGO_NET_OFFLINE=true
+ENV CARGO_HOME=/root/.cargo
+
+RUN pip install --prefix="/install" --no-warn-script-location \
+        cryptography
+
 # Build dependencies that are not available as wheels, to speed up rebuilds
 RUN pip install --prefix="/install" --no-warn-script-location \
         cryptography \
@@ -57,7 +93,7 @@ RUN pip install --prefix="/install" --no-warn-script-location \
         /synapse[all]
 
 ###
-### Stage 1: runtime
+### Stage: runtime
 ###
 
 FROM docker.io/python:${PYTHON_VERSION}-slim