Skip to content

cmdstan directory mismatch causes build failure on linux/arm64 #630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
arestifo opened this issue Oct 26, 2022 · 5 comments
Closed

cmdstan directory mismatch causes build failure on linux/arm64 #630

arestifo opened this issue Oct 26, 2022 · 5 comments

Comments

@arestifo
Copy link

Summary:

I am trying to build prophet Python wheel on linux/arm64, but am getting an error during cmdstanpy build (apologies for the syntax highlighting, this is from docker output):

#0 483.8   Building wheel for prophet (pyproject.toml): started
#0 490.8   Building wheel for prophet (pyproject.toml): finished with status 'error'
#0 490.8   error: subprocess-exited-with-error
#0 490.8
#0 490.8   × Building wheel for prophet (pyproject.toml) did not run successfully.
#0 490.8   │ exit code: 1
#0 490.8   ╰─> [109 lines of output]
#0 490.8       running bdist_wheel
#0 490.8       running build
#0 490.8       running build_py
#0 490.8       creating build
#0 490.8       creating build/lib.linux-aarch64-cpython-38
#0 490.8       creating build/lib.linux-aarch64-cpython-38/prophet
#0 490.8       creating build/lib.linux-aarch64-cpython-38/prophet/stan_model
#0 490.8       Compiling cmdstanpy model
#0 490.8       Installing cmdstan to /tmp/pip-install-xw4x12ur/prophet_bfa27437f3df487f8b3241bfe9e946bd/build/lib.linux-aarch64-cpython-38/prophet/stan_model/cmdstan-2.26.1
#0 490.8       Installing CmdStan version: 2.26.1
#0 490.8       Install directory: /tmp/pip-install-xw4x12ur/prophet_bfa27437f3df487f8b3241bfe9e946bd/build/lib.linux-aarch64-cpython-38/prophet/stan_model
#0 490.8       Downloading CmdStan version 2.26.1
#0 490.8
#0 490.8       0.00B [00:00, ?B/s]
#0 490.8         0%|          | 0.00/40.4M [00:00<?, ?B/s]
#0 490.8         2%|▏         | 792k/40.4M [00:00<00:06, 6.76MB/s]
#0 490.8         4%|▍         | 1.77M/40.4M [00:00<00:04, 8.61MB/s]
#0 490.8         7%|▋         | 2.86M/40.4M [00:00<00:04, 9.09MB/s]
#0 490.8        10%|▉         | 3.92M/40.4M [00:00<00:03, 9.78MB/s]
#0 490.8        12%|█▏        | 5.02M/40.4M [00:00<00:03, 9.73MB/s]
#0 490.8        15%|█▍        | 6.01M/40.4M [00:00<00:03, 9.93MB/s]
#0 490.8        18%|█▊        | 7.20M/40.4M [00:00<00:03, 10.7MB/s]
#0 490.8        20%|██        | 8.23M/40.4M [00:00<00:03, 9.99MB/s]
#0 490.8        23%|██▎       | 9.30M/40.4M [00:01<00:03, 9.87MB/s]
#0 490.8        26%|██▌       | 10.4M/40.4M [00:01<00:03, 10.1MB/s]
#0 490.8        28%|██▊       | 11.4M/40.4M [00:01<00:03, 9.93MB/s]
#0 490.8        31%|███       | 12.5M/40.4M [00:01<00:02, 10.4MB/s]
#0 490.8        33%|███▎      | 13.5M/40.4M [00:01<00:02, 10.0MB/s]
#0 490.8        36%|███▌      | 14.6M/40.4M [00:01<00:02, 10.1MB/s]
#0 490.8        39%|███▉      | 15.7M/40.4M [00:01<00:02, 9.97MB/s]
#0 490.8        41%|████▏     | 16.7M/40.4M [00:01<00:02, 10.2MB/s]
#0 490.8        44%|████▍     | 17.7M/40.4M [00:01<00:02, 9.89MB/s]
#0 490.8        47%|████▋     | 18.8M/40.4M [00:01<00:02, 10.2MB/s]
#0 490.8        49%|████▉     | 19.8M/40.4M [00:02<00:02, 9.98MB/s]
#0 490.8        52%|█████▏    | 20.9M/40.4M [00:02<00:02, 10.2MB/s]
#0 490.8        54%|█████▍    | 21.9M/40.4M [00:02<00:01, 10.0MB/s]
#0 490.8        57%|█████▋    | 22.9M/40.4M [00:02<00:01, 10.3MB/s]
#0 490.8        59%|█████▉    | 24.0M/40.4M [00:02<00:01, 10.0MB/s]
#0 490.8        62%|██████▏   | 25.0M/40.4M [00:02<00:01, 10.3MB/s]
#0 490.8        65%|██████▍   | 26.2M/40.4M [00:02<00:01, 10.4MB/s]
#0 490.8        67%|██████▋   | 27.2M/40.4M [00:02<00:01, 10.6MB/s]
#0 490.8        70%|███████   | 28.3M/40.4M [00:02<00:01, 10.3MB/s]
#0 490.8        73%|███████▎  | 29.3M/40.4M [00:03<00:01, 10.3MB/s]
#0 490.8        75%|███████▌  | 30.4M/40.4M [00:03<00:01, 10.1MB/s]
#0 490.8        78%|███████▊  | 31.4M/40.4M [00:03<00:00, 10.1MB/s]
#0 490.8        80%|████████  | 32.5M/40.4M [00:03<00:00, 10.4MB/s]
#0 490.8        83%|████████▎ | 33.5M/40.4M [00:03<00:00, 9.70MB/s]
#0 490.8        86%|████████▌ | 34.6M/40.4M [00:03<00:00, 10.1MB/s]
#0 490.8        88%|████████▊ | 35.5M/40.4M [00:03<00:00, 9.86MB/s]
#0 490.8        91%|█████████ | 36.7M/40.4M [00:03<00:00, 10.5MB/s]
#0 490.8        93%|█████████▎| 37.7M/40.4M [00:03<00:00, 9.95MB/s]
#0 490.8        96%|█████████▌| 38.8M/40.4M [00:04<00:00, 10.3MB/s]
#0 490.8        98%|█████████▊| 39.8M/40.4M [00:04<00:00, 9.87MB/s]
#0 490.8
#0 490.8       02:08:37 - cmdstanpy - WARNING - CmdStan installation failed.
#0 490.8       Failed to unpack file /tmp/tmp8ugoxrp0, error:
#0 490.8           tarfile should contain top-level dir cmdstan-2.26.1,but found dir cmdstan-2.26.1-linux-arm64 instead.
#0 490.8       CmdStan installation failed.
#0 490.8       Failed to unpack file /tmp/tmp8ugoxrp0, error:
#0 490.8           tarfile should contain top-level dir cmdstan-2.26.1,but found dir cmdstan-2.26.1-linux-arm64 instead.
#0 490.8       Download successful, file: /tmp/tmp8ugoxrp0
#0 490.8       Extracting distribution
#0 490.8       Traceback (most recent call last):
#0 490.8         File "/opt/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
#0 490.8           main()
#0 490.8         File "/opt/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
#0 490.8           json_out['return_val'] = hook(**hook_input['kwargs'])
#0 490.8         File "/opt/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
#0 490.8           return _build_backend().build_wheel(wheel_directory, config_settings,
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 412, in build_wheel
#0 490.8           return self._build_with_temp_dir(['bdist_wheel'], '.whl',
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
#0 490.8           self.run_setup()
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
#0 490.8           exec(code, locals())
#0 490.8         File "<string>", line 226, in <module>
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
#0 490.8           return distutils.core.setup(**attrs)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
#0 490.8           return run_commands(dist)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#0 490.8           dist.run_commands()
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
#0 490.8           self.run_command(cmd)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
#0 490.8           super().run_command(command)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
#0 490.8           cmd_obj.run()
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
#0 490.8           self.run_command('build')
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
#0 490.8           self.distribution.run_command(command)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
#0 490.8           super().run_command(command)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
#0 490.8           cmd_obj.run()
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run
#0 490.8           self.run_command(cmd_name)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
#0 490.8           self.distribution.run_command(command)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
#0 490.8           super().run_command(command)
#0 490.8         File "/tmp/pip-build-env-ij1yem4o/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
#0 490.8           cmd_obj.run()
#0 490.8         File "<string>", line 140, in run
#0 490.8         File "<string>", line 128, in build_models
#0 490.8         File "<string>", line 106, in build_cmdstan_model
#0 490.8         File "<string>", line 90, in install_cmdstan_deps
#0 490.8       RuntimeError: CmdStan failed to install in repackaged directory
#0 490.8       [end of output]
#0 490.8
#0 490.8   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 490.8   ERROR: Failed building wheel for prophet

Additional Information:

We are installing prophet 1.1 using pip install prophet==1.1 in a fresh python:3.8-slim Docker container. It seems like the proper directory names are not being respected on linux/arm64:

cmdstanpy/cmdstanpy/utils/cmdstan.py

Lines 53 to 56 in e7bb366

if arch and arch.lower() != "false":
url_end = f'v{version}/cmdstan-{version}-linux-{arch}.tar.gz'
else:
url_end = f'v{version}/cmdstan-{version}.tar.gz'

cmdstanpy/cmdstanpy/install_cmdstan.py

Lines 501 to 505 in e7bb366

cmdstan_dir = f'cmdstan-{version}'
if top_dir != cmdstan_dir:
raise CmdStanInstallError(
'tarfile should contain top-level dir {},'
'but found dir {} instead.'.format(cmdstan_dir, top_dir)

Maybe cmdstan_dir doesn't account for the top-level directory on linux/arm64?

Current Version:

As shown in the log output, we are trying to build cmdstanpy 2.26.1
@WardBrian
Copy link
Member

Hm, I think this is caused by #616. I didn’t realize that the folder created would also have a different name than we were expecting.

This should be an easy fix which we can hopefully roll out next week.

In the meantime, if you’re able to manually install CmdStan (or use an option like Conda), you can then set the environment variable “PROPHET_REPACKAGE_CMDSTAN” to “false” on your machine and then install it.

Sorry for the inconvenience!

@arestifo
Copy link
Author

Thanks for the quick reply! We are working around the issue right now by using prophet 1.1.1, which has pre-built wheels for linux/arm64. This shouldn't cause an issue in our workflows until those pre-built wheels get removed as our version gets older.

Really glad to see it's an easy fix, it will be really nice to have the option to build from source🎉

@WardBrian
Copy link
Member

So, this is tricky. It turns out that the non-x86s tarballs do unpack to have the same names as the x86 tarball for recent versions. 2.26.1 was the first version which featured a non-x86 tarball, and it appears it was mispackaged.

For example, if you look at cmdstan-2.27.0-linux-arm64.tar.gz, it contains a folder called cmdstan-2.27.0, not cmdstan-2.27.0-linux-arm64.

I'm going to ping our release manager @serban-nicusor-toptal on this: Were we aware this one version is inconsistent? It seems silly to update an old tarball, but also it would be nice if this wasn't a special case.

@serban-nicusor-toptal
Copy link
Contributor

Hello Brian, Alex,

I wasn't aware of this inconsistency, tho I went ahead and fixed the directory name inside the archive. Would you mind giving it a try to confirm everything is fine now ?
I went ahead and checked all releases for the -linux-arm64 ones and it seems fine, worth noting that Rok created a script to standardize and automate this that I use when releasing so we should not encounter it again.
Thanks for reporting this Alex!

@WardBrian
Copy link
Member

Thanks @serban-nicusor-toptal - I can confirm that now works with our installation script.

@arestifo let us know if anything else comes up! Thanks for reporting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@arestifo @WardBrian @serban-nicusor-toptal