Unable to upgrade grpcio to newer versions due to armv7 link error

[allenporter]

The problem

grpcio is currently pinned to an older version because of a link error that appears on armv7. As a result, many other libraries that depend on grpcio are pinned to older versions, and we are starting to see that maintaining all the necessary pins is getting very difficult.

This was originally pinned in Sept 2020 in #40678 to help with #40148 and #40266

The pin was last reverted in #50605 but had to be added back in #51289 as reports like #51167 appeared again. Related #51198 and #51286

As the conflicting pins in downstream libraries come up, it shows up like #51290 with fixes like #55115 and #54239. Now also seeing #55687 and #55883 which probably can be fixed with more pins, but it seems better just to focus on the root issue and get to a newer grpcio.

This issue tracks getting things into a state where grpcio can be upgraded to a newer version since I don't think we're tracking it anywhere else.

What is version of Home Assistant Core has the issue?

core-2021.9.7

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

No response

Link to integration documentation on our website

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

This can be reproduced in a development environment like this:

example.py

#!/usr/bin/python

from google_nest_sdm.google_nest import main

def main():
  print("Hello world")

if __name__ == "__main__":
  main()

requirements.txt

grpcio==1.38.0  # No links for 1.40.0
google-nest-sdm

Dockerfile

FROM ghcr.io/home-assistant/armv7-base-python:3.8-alpine3.12

ENV WHEELS_LINKS=https://wheels.home-assistant.io/alpine-3.12/armv7/

WORKDIR /code

COPY requirements.txt .
COPY example.py .

RUN python3 --version

RUN \
    pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links ${WHEELS_LINKS} \
        -r requirements.txt

CMD python3 ./example.py

$ uname -a
Linux raspberrypi 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux

$ cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 108.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3
...
Hardware	: BCM2711
Revision	: b03111
Serial		: 10000000c466b355
Model		: Raspberry Pi 4 Model B Rev 1.1

$ docker build --tag example-wheel-image .
...

$ docker run example-wheel-image:latest
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
Traceback (most recent call last):
  File "./example.py", line 3, in <module>
    from google_nest_sdm.google_nest import main
  File "/usr/local/lib/python3.8/site-packages/google_nest_sdm/google_nest.py", line 36, in <module>
    from .google_nest_subscriber import GoogleNestSubscriber
  File "/usr/local/lib/python3.8/site-packages/google_nest_sdm/google_nest_subscriber.py", line 14, in <module>
    from google.cloud import pubsub_v1
  File "/usr/local/lib/python3.8/site-packages/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/usr/local/lib/python3.8/site-packages/google/cloud/pubsub_v1/types.py", line 36, in <module>
    from google.pubsub_v1.types import pubsub as pubsub_gapic_types
  File "/usr/local/lib/python3.8/site-packages/google/pubsub_v1/__init__.py", line 18, in <module>
    from .services.publisher import PublisherClient
  File "/usr/local/lib/python3.8/site-packages/google/pubsub_v1/services/publisher/__init__.py", line 18, in <module>
    from .client import PublisherClient
  File "/usr/local/lib/python3.8/site-packages/google/pubsub_v1/services/publisher/client.py", line 27, in <module>
    from google.api_core import gapic_v1  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/google/api_core/gapic_v1/__init__.py", line 18, in <module>
    from google.api_core.gapic_v1 import config
  File "/usr/local/lib/python3.8/site-packages/google/api_core/gapic_v1/config.py", line 23, in <module>
    import grpc
  File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 23, in <module>
    from grpc._cython import cygrpc as _cygrpc
ImportError: Error loading shared library ld-linux-armhf.so.3: No such file or directory (needed by /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-arm-linux-gnueabihf.so)
[cmd] /bin/ash exited 1
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

Here is an example image that works:

Dockerfile:

FROM arm32v7/python:3.8-buster

WORKDIR /code

WORKDIR /code

COPY requirements.txt .
COPY example.py .

RUN python3 --version

RUN \
    pip3 install --no-cache-dir -r requirements.txt

CMD python3 ./example.py

$ docker build --tag example-image .
$ docker run example-image:latest
Hello world

[allenporter]

By installing libc6-compat we are able to satisfy the missing dependency, however that then reproduces the issue in #51286

$ docker run -it --rm --name ex --entrypoint sh example-wheel-image 
/code # ls
example.py        requirements.txt
# apk add libc6-compat
# python /code/example.py 
Traceback (most recent call last):
  File "/code/example.py", line 3, in <module>
    from google_nest_sdm.google_nest import main
  File "/usr/local/lib/python3.8/site-packages/google_nest_sdm/google_nest.py", line 36, in <module>
    from .google_nest_subscriber import GoogleNestSubscriber
  File "/usr/local/lib/python3.8/site-packages/google_nest_sdm/google_nest_subscriber.py", line 14, in <module>
    from google.cloud import pubsub_v1
  File "/usr/local/lib/python3.8/site-packages/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/usr/local/lib/python3.8/site-packages/google/cloud/pubsub_v1/types.py", line 36, in <module>
    from google.pubsub_v1.types import pubsub as pubsub_gapic_types
  File "/usr/local/lib/python3.8/site-packages/google/pubsub_v1/__init__.py", line 18, in <module>
    from .services.publisher import PublisherClient
  File "/usr/local/lib/python3.8/site-packages/google/pubsub_v1/services/publisher/__init__.py", line 18, in <module>
    from .client import PublisherClient
  File "/usr/local/lib/python3.8/site-packages/google/pubsub_v1/services/publisher/client.py", line 27, in <module>
    from google.api_core import gapic_v1  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/google/api_core/gapic_v1/__init__.py", line 18, in <module>
    from google.api_core.gapic_v1 import config
  File "/usr/local/lib/python3.8/site-packages/google/api_core/gapic_v1/config.py", line 23, in <module>
    import grpc
  File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 23, in <module>
    from grpc._cython import cygrpc as _cygrpc
ImportError: Error relocating /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-arm-linux-gnueabihf.so: __strftime_l: symbol not found

[allenporter]

# pip install pyelftools
#  /usr/local/bin/readelf.py -d /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-arm-linu
x-gnueabihf.so

Dynamic section at offset 0x708f00 contains 28 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [librt.so.1]
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [ld-linux-armhf.so.3]
 0x0000000c (INIT)                       0x64080
 0x0000000d (FINI)                       0x63e938
 0x00000019 (INIT_ARRAY)                 0x7037e4
 0x0000001b (INIT_ARRAYSZ)               1132 (bytes)
 0x0000001a (FINI_ARRAY)                 0x703c50
 0x0000001c (FINI_ARRAYSZ)               4 (bytes)
 0x00000004 (HASH)                       0x114
 0x00000005 (STRTAB)                     0x13b88
 0x00000006 (SYMTAB)                     0x59a8
 0x0000000a (STRSZ)                      154513 (bytes)
 0x0000000b (SYMENT)                     16 (bytes)
 0x00000003 (PLTGOT)                     0x719000
 0x00000002 (PLTRELSZ)                   11112 (bytes)
 0x00000014 (PLTREL)                     REL
 0x00000017 (JMPREL)                     0x61518
 0x00000011 (REL)                        0x3b458
 0x00000012 (RELSZ)                      155840 (bytes)
 0x00000013 (RELENT)                     8 (bytes)
 0x6ffffffe (VERNEED)                    0x3b358
 0x6fffffff (VERNEEDNUM)                 5
 0x6ffffff0 (VERSYM)                     0x3971a
 0x6ffffffa (RELCOUNT)                   16732
 0x00000000 (NULL)                       0x0
# apk add binutils
# nm /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-arm-linux-gnueabihf.so  | grep str
ftime_l
         U __strftime_l@@GLIBC_2.4
$ /usr/local/bin/readelf.py -s /lib/libc.musl-armv7.so.1 | grep strftime_l
   634: 00030cfd   412 FUNC    WEAK   DEFAULT    8 strftime_l

[allenporter]

# apk add patchelf
# patchelf --remove-needed ld-linux-armhf.so.3 /usr/local/lib/python3.8/si
te-packages/grpc/_cython/cygrpc.cpython-38-arm-linux-gnueabihf.so
# python ./example.py
...
ImportError: Error relocating /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-arm-linux-gnueabihf.so: __strftime_l: symbol not found
# readelf -d /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cp
ython-38-arm-linux-gnueabihf.so  | grep Shared
 0x00000001 (NEEDED)                     Shared library: [librt.so.1]
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
# readelf -s --dyn-syms /usr/local/lib/python3.8/site-packages/grpc/_cytho
n/cygrpc.cpython-38-arm-linux-gnueabihf.so | grep strftime_l
  1697: 00000000     0 FUNC    GLOBAL DEFAULT  UND __strftime_l@GLIBC_2.4 (3)
 53672: 00000000     0 FUNC    GLOBAL DEFAULT  UND __strftime_l@@GLIBC_2.4
 # readelf -s --dyn-syms /lib/libc.musl-armv7.so.1 | grep strftime_l
   634: 00030cfd   412 FUNC    WEAK   DEFAULT    8 strftime_l

@pvizeli thanks for the suggestions. I've tried a few methods to add extra symbols, including various elf tools and python libraries, but don't yet see an easy way to rewrite the symbols, either adding an alias in the libc dynamic symbol table or rewriting the cython library side symbols to look for a new name. auditwheel just seems to call out to patchelf which doesn't have this capability. It seems like doing ELF level patching is an option, just don't see a ready to go solution yet.

@pvizeli does it make sense for me to keep exploring elf patching? or exploring writing a custom wheel build process for grpcio? Any suggestions are welcome.

[allenporter]

This is a hack that gets it working:

# cat hack.c 
#include <time.h>
#include <string.h>

char * __strdup(const char * s)
{
	return strdup(s);
}

size_t __strftime_l (char *  __restrict a, size_t b, const char *  __restrict c, const struct tm *  __restrict d, locale_t e)
{
	strftime_l(a, b, c, d, e);
}

int backtrace(void **buffer, int size)
{
	return 0;
}

int __register_atfork(void (*prepare) (void), void (*parent) (void), void (*child) (void), void *__dso_handle)
{
	return 0;
}

# gcc -o /lib/libhack.so.1 hack.c -shared
# patchelf --add-needed libhack.so.1 /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-arm-linux-gnueabihf.so
# python ./example.py
Hello world

Given the number of missing dependencies here, i'm assuming this means we need to change flags when building the grpcio wheel. (e.g. to explicitly use musl?)

[pvizeli]

A good question would be, why is that issue just on armv7. I think the answer is that the build files threatening something different. I'm very sure that the evil google use here own libc and just doesn't link it statically on armv7 while he doing it on the other platforms.

Maybe we can include your "hack" (which is basically just what you normally do for supporting multiple platforms) in the grpc. I think that is something that Google needs to fix or we should kick all Google stuff out from the HA core.

[allenporter]

Well, that escalated quickly! :) Before we talk about kicking Google out of HA core, i think it makes sense to at least raise an issue with grpcio. My impression is there have been no issues filed against grpcio for this issue in the last year of it being a problem. We should give an example Dockerfile that reproduces the problem and start a conversation, hence why I'm working on a repro. I'd even be happy to send the PR to make a fix once we understand it.

So, when comparing architectures, one data point is that on amd64 I do see a different set of linkages. Updating the above example Dockerfile:

FROM ghcr.io/home-assistant/amd64-base-python:3.8-alpine3.12

ENV WHEELS_LINKS=https://wheels.home-assistant.io/alpine-3.12/amd64/
...

Then building:

$ docker build --tag grpcio-bug .
$ docker run grpcio-bug
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
Hello world
[cmd] /bin/ash exited 0
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
$ docker run -it --name grpcio --entrypoint sh grpcio-bug 
# apk add binutils
# readelf -d /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cp
ython-38-x86_64-linux-gnu.so | grep Shared
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.musl-x86_64.so.1]
# readelf -s /usr/local/lib/python3.8/site-packages/grpc/_cython/cygrpc.cp
ython-38-x86_64-linux-gnu.so | grep stftime

So no references to __strftime_l at all. I'll look closer at the grpcio build.

My issue here is that it works fine on other armv7 images as shown above. Is there a specific base image you can recommend that you would expect them to support? e.g. arm32v7/python:3.8

[allenporter]

Also, the other variable here is that this is a home assistant built wheel, i think? The grpcio build process has environment variables it supports, including setting compatibility options, so we can see if passing those helps. I may be able to try to get a build of this package going locally, if i can figure out the right command.

This is the smallest Dockerfile that shows the problem with the home assistant built wheel that i'll now play with:

FROM arm32v7/alpine:3.12

RUN apk add --no-cache python3 py3-pip

WORKDIR /code
COPY requirements.txt .
COPY example.py .

ENV WHEELS_LINKS=https://wheels.home-assistant.io/alpine-3.12/armv7/
RUN \
    pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links ${WHEELS_LINKS} \
        -r requirements.txt

CMD python3 ./example.py

I think the next step is to include the process of building the wheel into the repro steps.

[allenporter]

I came across grpc/grpc#26620 which proposes building grpcio wheels for musl and alpine now that auditwheel supports musl linux.

[allenporter]

I've filed grpc/grpc#27512 against grpcio with a simpler repro case.

[allenporter]

I'm still looking into this.