Skip to content

Add information about lack of possibility of cross compile Poetry #1416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jul 9, 2023
Merged

Add information about lack of possibility of cross compile Poetry #1416

Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5be1d17
add information about lack of possibility of cross compile Poetry pro…
Czaki Feb 10, 2023
cc0bb5d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 10, 2023
96cffc2
Update docs/faq.md
Czaki Feb 10, 2023
c7643f0
update information about poetry state
Czaki Feb 14, 2023
567fe2a
Apply suggestions from code review
Czaki Feb 14, 2023
ed32041
fix note text
Czaki Feb 16, 2023
983347f
Add reference to the new 'wheel tags' command
joerick Jun 18, 2023
3192cdc
Copy pass on this text
joerick Jun 18, 2023
1ce2091
Copy edit of the whole section, make it a bit easier to follow, add s…
joerick Jun 18, 2023
8f569c4
Merge remote-tracking branch 'origin/main' into poerty_problem
joerick Jun 23, 2023
f44d278
Update docs/faq.md
joerick Jul 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,13 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co

### Building macOS wheels for Apple Silicon {: #apple-silicon}

`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+:
`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners.

!!! note
Currently it is impossible to cross-compile project that uses Poetry as a build backend.
Reladed issue [here](https://github.com/python-poetry/poetry/issues/7107)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Team, with regards to #1415 that caused the current PR, I would like to clarify that I can to cross-compile project that uses Poetry as a build backend. Perhaps it is my understanding but this documentation change is a little confusing (at least to me)

To elaborate, I can compile

  1. arm64 compatible wheel on MacOS (x86_64) runner
  2. arm64 compatible wheel on Linux (x86_64) runner

In case of#1, the compiled wheel filename just contained x86_64 in it, instead of arm64, which is why I raised the ticket. But, a Apple Silicon M1 mac client was successfully able to download, install and execute the wheel in another Poetry-based project.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is put in the section about cross-compiling MacOS wheels. Cross-compilation of Linux ARM works differently than cross-compilation on MacOS.

And the wrong name of the produced wheel is, for me something that makes the building wheel not work.

If you have instructions that will allow you to build the wheel in a way that allows you to upload them directly from CI to PyPI without manual intervention, then this warning such be replaced.

At this moment, I do not have such a project and M1 machine that will allow validating such instruction.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps my comment intention was not clear and I could have articulated myself more clearly - I did not mean that the warning should not be added at all. Thank you for clarifying your thoughts and intention.

Copy link

@TeoZosa TeoZosa Feb 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have instructions that will allow you to build the wheel in a way that allows you to upload them directly from CI to PyPI without manual intervention, then this warning such be replaced.

Chiming in since I maintain a project which came across this same issue.

What are we defining as "manual intervention"? Looking at #1415, it looks like @azagniotov included instructions to build the wheel which would allow them to be uploaded from CI to PyPI. Namely, by setting the CIBW_REPAIR_WHEEL_COMMAND_MACOS env var like so (which is exactly how my project resolves this issue as well):

...
    steps:
      - uses: actions/checkout@v3
      - name: Build wheels
        uses: pypa/[email protected]
        env:
          CIBW_REPAIR_WHEEL_COMMAND_MACOS: >
            delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel} &&
            for file in {dest_dir}/*.whl ; do mv $file ${file//x86_64/arm64} ; done
          ...
          CIBW_ARCHS_MACOS: "arm64"
          ...
...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This solution is really bad as it skip whole delocate steep (as delocate will look fot x86 libs and dependencies)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Czaki you raise a good point re To have the correct workflow you need to first rename the wheel file and then execute delocate on renamed file. <= thank you for your guidance and observation, I will look into this 👍 Much appreciated

Copy link

@azagniotov azagniotov Feb 14, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Czaki I followed your pointer and swapped the order of operations where:

  1. The wheel was renamed first
  2. The execute delocate command was on the renamed wheel
CIBW_REPAIR_WHEEL_COMMAND_MACOS: |
    echo "Target delocate archs: {delocate_archs}"

    ORIGINAL_WHEEL={wheel}

    echo "Running delocate-listdeps to list linked original wheel dependencies"
    delocate-listdeps --all $ORIGINAL_WHEEL

    if [ "${{ matrix.cibuildwheel_platform_id }}" == "macosx_arm64" ];
    then
      echo "Renaming .whl file when architecture is 'macosx_arm64'"
      RENAMED_WHEEL=${ORIGINAL_WHEEL//x86_64/arm64}

      echo "Wheel will be renamed to $RENAMED_WHEEL"
      mv $ORIGINAL_WHEEL $RENAMED_WHEEL
    else
      RENAMED_WHEEL=$ORIGINAL_WHEEL
    fi

    echo "Running delocate-wheel command on $RENAMED_WHEEL"
    delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v $RENAMED_WHEEL

    echo "Running delocate-listdeps to list linked wheel dependencies"
    WHEEL_SIMPLE_FILENAME="${RENAMED_WHEEL##*/}"
    delocate-listdeps --all {dest_dir}/$WHEEL_SIMPLE_FILENAME

    echo "DONE."

The output is as follows:

Repairing wheel...
  
Target delocate archs: arm64
Running delocate-listdeps to list linked original wheel dependencies
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/usr/lib/libSystem.B.dylib
/usr/lib/libobjc.A.dylib
Renaming .whl file when architecture is 'macosx_arm64'
Wheel will be renamed to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-6h5tsf4q/cp310-macosx_arm64/built_wheel/my_project-0.0.1-cp310-cp310-macosx_12_0_arm64.whl
Running delocate-wheel command on /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-6h5tsf4q/cp310-macosx_arm64/built_wheel/my_project-0.0.1-cp310-cp310-macosx_12_0_arm64.whl
Fixing: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-6h5tsf4q/cp310-macosx_arm64/built_wheel/my_project-0.0.1-cp310-cp310-macosx_12_0_arm64.whl
Running delocate-listdeps to list linked wheel dependencies
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/usr/lib/libSystem.B.dylib
/usr/lib/libobjc.A.dylib
DONE.

The repairing wheel step completed successfully also in this case, and the compiled wheel was successfully installed on Apple M1 and invoked at runtime.

Thoughts? 🙇🏼

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.
For now I do not know if just link to this response or embed whole script in the documentation.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @Czaki .

I personally lean towards a more defensive approach and I would just include a link to this discussion where appropriate as oppose to including my code snippet in the official docs.

But, that being said, I defer to you and your team regarding this. Please do what you think is best for the Poetry-based community and such a widely used library (cibuildwheel).

Thank you for your time on this, regardless.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated the description to be more correct. Did it look better?


With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+:

#### `x86_64`

Expand Down