Dependency issues building gtfs-etl image locally on ARM architecture

[ShayAdler]

Ran docker-compose -f docker-compose.yaml -f docker-compose-dev.yaml build gtfs-etl locally, and encountered several dependencies issues -

• gdal -
  #10 21.46 Collecting fiona>=1.8 #10 21.54 Downloading Fiona-1.8.21.tar.gz (1.0 MB) #10 21.58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 32.2 MB/s eta 0:00:00 #10 21.65 Preparing metadata (setup.py): started #10 21.75 Preparing metadata (setup.py): finished with status 'error' #10 21.76 error: subprocess-exited-with-error #10 21.76 #10 21.76 × python setup.py egg_info did not run successfully. #10 21.76 │ exit code: 1 #10 21.76 ╰─> [2 lines of output] #10 21.76 Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config' #10 21.76 A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable. #10 21.76 [end of output]
  Editing the dockerfile to resolved those errors -
  Dockerfile.txt

• Up to the next one - plyvel, got this error which I haven't had time to resolve yet, seems like a dependencies version incompatibility issue-
  plyvel_error.txt

[OriHoch]

It works for me, because this fails on the Docker build, then it could be either something with the OS / environment you are using, or due to some local changes you have in gtfs-etl code

here are the full reproduction steps when I run it from my PC:

• OS: Ubuntu 20.04
• Docker version 20.10.14
• Change to directory containing a clone of hasadna/open-bus-gtfs-etl repository
• Ensure on main branch and no local changes:

open-bus-gtfs-etl$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

• Run git pull origin main to make sure latest changes from main branch are updated locally
• Check the commit the branch is currently on

open-bus-gtfs-etl$ git log
commit db3b714edcf4b0176132e2150b3134e1a72390f4 (HEAD -> main, origin/main, origin/HEAD)

• Run the docker build

open-bus-gtfs-etl$ docker build --no-cache .
Sending build context to Docker daemon  994.8kB
Step 1/12 : FROM python:3.8@sha256:c7706b8d1b1e540b9dd42ac537498d7f3138e4b8b89fb890b2ee4d2c0bccc8ea
 ---> e7d3be492e61
Step 2/12 : RUN pip install --upgrade pip
 ---> Running in a1f75de5d0c4
Requirement already satisfied: pip in /usr/local/lib/python3.8/site-packages (21.1.2)
Collecting pip
  Downloading pip-22.1.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.2
    Uninstalling pip-21.1.2:
      Successfully uninstalled pip-21.1.2
Successfully installed pip-22.1.2
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
Removing intermediate container a1f75de5d0c4
 ---> 01c3c80290df
Step 3/12 : WORKDIR /srv
 ---> Running in 6fab3963ae17
Removing intermediate container 6fab3963ae17
 ---> d6af5d8648ab
Step 4/12 : RUN pip install -r https://raw.githubusercontent.com/hasadna/open-bus-stride-db/main/requirements.txt &&    git clone https://github.com/hasadna/open-bus-stride-db.git &&    pip install -e open-bus-stride-db
 ---> Running in 72f89744c4a8
Collecting alembic==1.6.0
  Downloading alembic-1.6.0-py2.py3-none-any.whl (164 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.2/164.2 kB 1.5 MB/s eta 0:00:00
Collecting psycopg2==2.8.6
  Downloading psycopg2-2.8.6.tar.gz (383 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 383.8/383.8 kB 6.2 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting SQLAlchemy>=1.3.0
  Downloading SQLAlchemy-1.4.37-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 15.3 MB/s eta 0:00:00
Collecting python-dateutil
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 20.2 MB/s eta 0:00:00
Collecting python-editor>=0.3
  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)
Collecting Mako
  Downloading Mako-1.2.0-py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 kB 22.1 MB/s eta 0:00:00
Collecting greenlet!=0.4.17
  Downloading greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (156 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.6/156.6 kB 26.9 MB/s eta 0:00:00
Collecting MarkupSafe>=0.9.2
  Downloading MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py): started
  Building wheel for psycopg2 (setup.py): finished with status 'done'
  Created wheel for psycopg2: filename=psycopg2-2.8.6-cp38-cp38-linux_x86_64.whl size=501038 sha256=11f54aed7f27378f271f3b33e6f4a06ae1e1eca30fd56d92210e18934d69aa0c
  Stored in directory: /root/.cache/pip/wheels/70/5e/69/8a020d78c09043156a7df0b64529e460fbd922ca065c4b795c
Successfully built psycopg2
Installing collected packages: python-editor, six, psycopg2, MarkupSafe, greenlet, SQLAlchemy, python-dateutil, Mako, alembic
Successfully installed Mako-1.2.0 MarkupSafe-2.1.1 SQLAlchemy-1.4.37 alembic-1.6.0 greenlet-1.1.2 psycopg2-2.8.6 python-dateutil-2.8.2 python-editor-1.0.4 six-1.16.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Cloning into 'open-bus-stride-db'...
Obtaining file:///srv/open-bus-stride-db
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Installing collected packages: open-bus-stride-db
  Running setup.py develop for open-bus-stride-db
Successfully installed open-bus-stride-db-1655034922.5502179
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 72f89744c4a8
 ---> 275336624777
Step 5/12 : COPY requirements.txt ./open-bus-gtfs-etl/requirements.txt
 ---> 1954ccecdc65
Step 6/12 : RUN pip install -r open-bus-gtfs-etl/requirements.txt
 ---> Running in 23219ef2bac0
Collecting partridge==0.11.0
  Downloading partridge-0.11.0-py2.py3-none-any.whl (15 kB)
Collecting tqdm==4.62.1
  Downloading tqdm-4.62.1-py2.py3-none-any.whl (76 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.2/76.2 kB 435.1 kB/s eta 0:00:00
Collecting gtfs_kit==5.1.4
  Downloading gtfs_kit-5.1.4-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.3/62.3 kB 3.8 MB/s eta 0:00:00
Collecting pandas==1.3.1
  Downloading pandas-1.3.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 31.6 MB/s eta 0:00:00
Collecting jsonschema==3.2.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.3/56.3 kB 14.5 MB/s eta 0:00:00
Collecting pydantic==1.8.2
  Downloading pydantic-1.8.2-cp38-cp38-manylinux2014_x86_64.whl (13.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.7/13.7 MB 49.7 MB/s eta 0:00:00
Collecting pytz==2021.1
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 510.8/510.8 kB 59.9 MB/s eta 0:00:00
Collecting click==7.1.2
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.8/82.8 kB 9.0 MB/s eta 0:00:00
Collecting setuptools==56.0.0
  Downloading setuptools-56.0.0-py3-none-any.whl (784 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 784.9/784.9 kB 28.7 MB/s eta 0:00:00
Collecting numpy==1.21.2
  Downloading numpy-1.21.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.8/15.8 MB 47.7 MB/s eta 0:00:00
Collecting psutil==5.9.0
  Downloading psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 283.8/283.8 kB 46.4 MB/s eta 0:00:00
Collecting kvfile==0.0.13
  Downloading kvfile-0.0.13-py2.py3-none-any.whl (6.5 kB)
Collecting plyvel==1.4.0
  Downloading plyvel-1.4.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (743 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 743.8/743.8 kB 59.6 MB/s eta 0:00:00
Collecting chardet
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 178.7/178.7 kB 24.4 MB/s eta 0:00:00
Collecting networkx>=2.0
  Downloading networkx-2.8.3-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 83.8 MB/s eta 0:00:00
Collecting pycountry<20
  Downloading pycountry-19.8.18.tar.gz (10.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/10.0 MB 58.5 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting requests<3
  Downloading requests-2.28.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 14.0 MB/s eta 0:00:00
Collecting rtree<1
  Downloading Rtree-0.9.7-cp38-cp38-manylinux2010_x86_64.whl (994 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 994.7/994.7 kB 60.0 MB/s eta 0:00:00
Collecting shapely<2
  Downloading Shapely-1.8.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 64.5 MB/s eta 0:00:00
Collecting geopandas<1
  Downloading geopandas-0.10.2-py2.py3-none-any.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 59.3 MB/s eta 0:00:00
Collecting utm<1
  Downloading utm-0.7.0.tar.gz (8.7 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting json2html<2
  Downloading json2html-1.3.0.tar.gz (7.0 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting folium<1
  Downloading folium-0.12.1.post1-py2.py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.0/95.0 kB 21.3 MB/s eta 0:00:00
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/site-packages (from pandas==1.3.1->-r open-bus-gtfs-etl/requirements.txt (line 4)) (2.8.2)
Collecting pyrsistent>=0.14.0
  Downloading pyrsistent-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (119 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 119.8/119.8 kB 28.9 MB/s eta 0:00:00
Collecting attrs>=17.4.0
  Downloading attrs-21.4.0-py2.py3-none-any.whl (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.6/60.6 kB 6.6 MB/s eta 0:00:00
Requirement already satisfied: six>=1.11.0 in /usr/local/lib/python3.8/site-packages (from jsonschema==3.2.0->-r open-bus-gtfs-etl/requirements.txt (line 5)) (1.16.0)
Collecting typing-extensions>=3.7.4.3
  Downloading typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting isodate
  Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.7/41.7 kB 5.0 MB/s eta 0:00:00
Collecting cachetools
  Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting jinja2>=2.9
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 31.5 MB/s eta 0:00:00
Collecting branca>=0.3.0
  Downloading branca-0.5.0-py3-none-any.whl (24 kB)
Collecting fiona>=1.8
  Downloading Fiona-1.8.21-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.6/16.6 MB 47.4 MB/s eta 0:00:00
Collecting pyproj>=2.2.0
  Downloading pyproj-3.3.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.7/7.7 MB 40.6 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.0/139.0 kB 28.6 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2022.5.18.1-py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.2/155.2 kB 38.7 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.3-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 16.1 MB/s eta 0:00:00
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting munch
  Downloading munch-2.5.0-py2.py3-none-any.whl (10 kB)
Collecting click-plugins>=1.0
  Downloading click_plugins-1.1.1-py2.py3-none-any.whl (7.5 kB)
Collecting cligj>=0.5
  Downloading cligj-0.7.2-py3-none-any.whl (7.1 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/site-packages (from jinja2>=2.9->folium<1->gtfs_kit==5.1.4->-r open-bus-gtfs-etl/requirements.txt (line 3)) (2.1.1)
Building wheels for collected packages: json2html, pycountry, utm
  Building wheel for json2html (setup.py): started
  Building wheel for json2html (setup.py): finished with status 'done'
  Created wheel for json2html: filename=json2html-1.3.0-py3-none-any.whl size=7608 sha256=89773d4145127dd38d7c50ef3f58fe3341cb24a2142f8d6b38e704d74e7ca68a
  Stored in directory: /root/.cache/pip/wheels/34/1a/a9/0d39b9e11fc97dd947ca4df7f0f0c1f4f2e25727d3c63b7739
  Building wheel for pycountry (setup.py): started
  Building wheel for pycountry (setup.py): finished with status 'done'
  Created wheel for pycountry: filename=pycountry-19.8.18-py2.py3-none-any.whl size=10627377 sha256=be8b7d4bc9d8fcb7d6c2317f039a6526ed2387186159b8f96ae75d321c8617ed
  Stored in directory: /root/.cache/pip/wheels/30/63/8c/165da7379ab28a48fdc297d0bb1823bbe06b06405566db3204
  Building wheel for utm (setup.py): started
  Building wheel for utm (setup.py): finished with status 'done'
  Created wheel for utm: filename=utm-0.7.0-py3-none-any.whl size=6107 sha256=7085a5f906eff5b78ca05a9daf95d6bcf38cdf33f859c5fd0293724e78011d71
  Stored in directory: /root/.cache/pip/wheels/65/e2/d8/878a8cc986641056fbfebefc4d8eb64238a7b6d3426e86b447
Successfully built json2html pycountry utm
Installing collected packages: utm, rtree, pytz, pycountry, plyvel, json2html, urllib3, typing-extensions, tqdm, shapely, setuptools, pyrsistent, psutil, numpy, networkx, munch, jinja2, isodate, idna, click, charset-normalizer, chardet, certifi, cachetools, attrs, requests, pyproj, pydantic, pandas, kvfile, jsonschema, cligj, click-plugins, branca, partridge, folium, fiona, geopandas, gtfs_kit
  Attempting uninstall: setuptools
    Found existing installation: setuptools 57.0.0
    Uninstalling setuptools-57.0.0:
      Successfully uninstalled setuptools-57.0.0
Successfully installed attrs-21.4.0 branca-0.5.0 cachetools-5.2.0 certifi-2022.5.18.1 chardet-4.0.0 charset-normalizer-2.0.12 click-7.1.2 click-plugins-1.1.1 cligj-0.7.2 fiona-1.8.21 folium-0.12.1.post1 geopandas-0.10.2 gtfs_kit-5.1.4 idna-3.3 isodate-0.6.1 jinja2-3.1.2 json2html-1.3.0 jsonschema-3.2.0 kvfile-0.0.13 munch-2.5.0 networkx-2.8.3 numpy-1.21.2 pandas-1.3.1 partridge-0.11.0 plyvel-1.4.0 psutil-5.9.0 pycountry-19.8.18 pydantic-1.8.2 pyproj-3.3.1 pyrsistent-0.18.1 pytz-2021.1 requests-2.28.0 rtree-0.9.7 setuptools-56.0.0 shapely-1.8.2 tqdm-4.62.1 typing-extensions-4.2.0 urllib3-1.26.9 utm-0.7.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 23219ef2bac0
 ---> efe74d5b18cf
Step 7/12 : COPY setup.py ./open-bus-gtfs-etl/setup.py
 ---> 975a95dea308
Step 8/12 : COPY open_bus_gtfs_etl ./open-bus-gtfs-etl/open_bus_gtfs_etl
 ---> 00bdbecf6015
Step 9/12 : RUN pip install -e open-bus-gtfs-etl
 ---> Running in f9ac5bd6eb45
Obtaining file:///srv/open-bus-gtfs-etl
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Installing collected packages: open-bus-gtfs-etl
  Running setup.py develop for open-bus-gtfs-etl
Successfully installed open-bus-gtfs-etl-1655034957.698335
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container f9ac5bd6eb45
 ---> 3e0c374d6a2a
Step 10/12 : ENV PYTHONUNBUFFERED=1
 ---> Running in 0ac0e311e2a4
Removing intermediate container 0ac0e311e2a4
 ---> e0a1e4a2db5f
Step 11/12 : ENTRYPOINT ["open-bus-gtfs-etl"]
 ---> Running in 7b6eca8384f0
Removing intermediate container 7b6eca8384f0
 ---> 14cca0c11101
Step 12/12 : CMD ["--help"]
 ---> Running in 3dd743ae48a4
Removing intermediate container 3dd743ae48a4
 ---> 2afadd31f336
Successfully built 2afadd31f336

[OriHoch]

Please try to run with the reproduction steps as I wrote above, and provide full output and reproduction steps

[OriHoch]

@ShayAdler the problem is that the base Python Docker image is multiplatform and you are running with ARM architecture which requires compilation of some Python libraries.

I modified all the Docker images to specify linux/amd64 architecture which is the chitecture they were built with and which we use on our infrastructure. It might work as-is, but if not then specific changes will be required to support ARM architecture