Add a basic `pyproject.toml` to allow building from source distribution #5

anoadragon453 · 2022-05-30T11:57:10Z

Adds a bare-minimum pyproject.toml file as detailed here to allow pip to build the rust jaeger reporter from source. This allows this package to be installed alongside python interpreters other than CPython (i.e. Pyston).

PS: I'm not sure if any other options are necessary, though testing building locally, installing and then running with a local Synapse works.

Version bounds are based on maturin's current version of v0.12.18.

reivilibre · 2022-05-30T12:28:05Z

This fails to compile for me; unsure why from first glance.

$ gh pr checkout 5
$ python -m build --sdist

...
Successfully built rust_python_jaeger_reporter-0.1.20.tar.gz

$ tar tvf dist/rust_python_jaeger_reporter-0.1.20.tar.gz 
-rw-r--r-- 0/0            1056 1970-01-01 01:00 rust_python_jaeger_reporter-0.1.20/Cargo.toml
-rw-rw-r-- 1000/1000      1462 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/.github/workflows/tests.yaml
-rw-rw-r-- 1000/1000        42 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/.gitignore
-rw-rw-r-- 1000/1000      1825 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/README.md
-rw-rw-r-- 1000/1000        10 2022-05-30 13:25 rust_python_jaeger_reporter-0.1.20/dist/rust_python_jaeger_reporter-0.1.20.tar.gz
-rw-rw-r-- 1000/1000        75 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/pyproject.toml
-rw-rw-r-- 1000/1000     21476 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/src/lib.rs
-rw-rw-r-- 1000/1000     11640 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/src/thrift_gen/agent.rs
-rw-rw-r-- 1000/1000     43311 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/src/thrift_gen/jaeger.rs
-rw-rw-r-- 1000/1000       202 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/src/thrift_gen/mod.rs
-rw-rw-r-- 1000/1000     30510 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/src/thrift_gen/sampling.rs
-rw-rw-r-- 1000/1000     40824 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/src/thrift_gen/zipkincore.rs
-rw-rw-r-- 1000/1000      2248 2022-05-30 13:23 rust_python_jaeger_reporter-0.1.20/test.py
-rw-r--r-- 0/0            2431 1970-01-01 01:00 rust_python_jaeger_reporter-0.1.20/PKG-INFO

$ python -m venv .venvtmp
$ . .venvtmp/bin/activate
$ pip install dist/rust_python_jaeger_reporter-0.1.20.tar.gz 
Processing ./dist/rust_python_jaeger_reporter-0.1.20.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: rust-python-jaeger-reporter
  Building wheel for rust-python-jaeger-reporter (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for rust-python-jaeger-reporter (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [95 lines of output]
      Running `maturin pep517 build-wheel -i /home/rei/work/rust-jaeger-python-client/.venvtmp/bin/python --compatibility off`
      warning: Patch `pyo3 v0.16.5 (https://github.com/PyO3/pyo3?branch=main#f84c740e)` was not used in the crate graph.
      Check that the patched package version and available features are compatible
      with the dependency requirements. If the patch has a different version from
      what is locked in the Cargo.lock file, run `cargo update` to use the new
      version. This may also occur with an optional dependency that is not enabled.
         Compiling proc-macro2 v1.0.39
         Compiling unicode-ident v1.0.0
         Compiling autocfg v1.1.0
         Compiling syn v1.0.95
         Compiling cfg-if v1.0.0
         Compiling libc v0.2.126
         Compiling proc-macro-hack v0.5.19
         Compiling parking_lot_core v0.8.5
         Compiling unindent v0.1.9
         Compiling crossbeam-utils v0.8.8
         Compiling log v0.4.17
         Compiling scopeguard v1.1.0
         Compiling smallvec v1.8.0
         Compiling inventory v0.1.11
         Compiling pyo3 v0.13.2
         Compiling lazy_static v1.4.0
         Compiling either v1.6.1
         Compiling cfg-if v0.1.10
         Compiling byteorder v1.4.3
         Compiling integer-encoding v3.0.3
         Compiling instant v0.1.12
         Compiling try_from v0.3.2
         Compiling itertools v0.10.3
         Compiling num-traits v0.2.15
         Compiling lock_api v0.4.7
         Compiling crossbeam-channel v0.5.4
         Compiling quote v1.0.18
         Compiling num_cpus v1.13.1
         Compiling paste-impl v0.1.18
         Compiling threadpool v1.8.1
         Compiling parking_lot v0.11.2
         Compiling paste v0.1.18
         Compiling ordered-float v1.1.1
         Compiling ordered-float v2.10.0
         Compiling thrift v0.14.1 (https://github.com/apache/thrift?tag=v0.14.1#f6fa1794)
         Compiling pyo3-macros-backend v0.13.2
         Compiling indoc-impl v0.3.6
         Compiling inventory-impl v0.1.11
         Compiling ctor v0.1.22
         Compiling ghost v0.1.4
         Compiling pyo3-macros v0.13.2
         Compiling indoc v0.3.6
         Compiling rust-python-jaeger-reporter v0.1.20 (/tmp/pip-req-build-9y1edii7)
      error: could not compile `rust-python-jaeger-reporter` due to 5 previous errors
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `cargo rustc --manifest-path Cargo.toml --message-format json --release --lib --`
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.10 at /home/rei/work/rust-jaeger-python-client/.venvtmp/bin/python
      error: cannot find attribute `pyo3` in this scope
        --> src/lib.rs:70:3
         |
      70 | #[pyo3(text_signature = "()")]
         |   ^^^^
         |
         = note: `pyo3` is in scope, but it is a crate, not an attribute
      
      
      error: cannot find attribute `pyo3` in this scope
         --> src/lib.rs:182:7
          |
      182 |     #[pyo3(text_signature = "($self, service_name, tags, max_length, /)")]
          |       ^^^^
          |
          = note: `pyo3` is in scope, but it is a crate, not an attribute
      
      
      error: cannot find attribute `pyo3` in this scope
         --> src/lib.rs:203:7
          |
      203 |     #[pyo3(text_signature = "($self, span, /)")]
          |       ^^^^
          |
          = note: `pyo3` is in scope, but it is a crate, not an attribute
      
      
      error: cannot find attribute `pyo3` in this scope
         --> src/lib.rs:220:7
          |
      220 |     #[pyo3(text_signature = "($self, /)")]
          |       ^^^^
          |
          = note: `pyo3` is in scope, but it is a crate, not an attribute
      
      
      error: aborting due to 4 previous errors
      
      
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/rei/work/rust-jaeger-python-client/.venvtmp/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for rust-python-jaeger-reporter
Failed to build rust-python-jaeger-reporter
ERROR: Could not build wheels for rust-python-jaeger-reporter, which is required to install pyproject.toml-based projects

Edit: the warning about outdated lockfile / the patch being ignored seems suspicious
Especially the part that sounds like it's ignoring your feature set — probably why the macros are failing since they're often in a separate feature.

Maybe it'd be worth upgrading to a later version so using a patched git source isn't needed?

rust-jaeger-python-client/Cargo.toml

Lines 29 to 35 in f64f30e

    
           [dependencies.pyo3] 
        
           version = "0.13.2" 
        
           features = ["extension-module"] 
        
           # There's a memory leak in v0.13.2 when using u128's 
        
           [patch.crates-io] 
        
           pyo3 = { git = 'https://github.com/PyO3/pyo3', branch = "main" }

(Edit: I also note that the dist dir is included in the sdist ... so if you keep building them, you'll get sdist-ception :/)

anoadragon453 · 2022-05-30T13:22:59Z

@reivilibre here's my build output. Are you using a nightly version of rustc? I'm on 1.61.0.

(venv-pyston2.3.3) [work@izzy rust-jaeger-python-client]$ pip install -v ~/Downloads/rust-jaeger-python-client-anoa-pyproject.zip 
Using pip 22.1.1 from /home/work/code/venv-pyston2.3.3/lib/python3.8-pyston2.3/site-packages/pip (python 3.8)
Processing /home/work/Downloads/rust-jaeger-python-client-anoa-pyproject.zip
  Running command pip subprocess to install build dependencies
  Collecting maturin<0.13,>=0.12
    Using cached maturin-0.12.18-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.musllinux_1_1_x86_64.whl (9.6 MB)
  Collecting tomli>=1.1.0
    Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
  Installing collected packages: tomli, maturin
  Successfully installed maturin-0.12.18 tomli-2.0.1
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  🔗 Found pyo3 bindings
  🐍 Found CPython 3.8 at /home/work/code/venv-pyston2.3.3/bin/pyston
  rust_python_jaeger_reporter-0.1.20.dist-info
  Checking for Rust toolchain....
  Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-xh_ph6ig --interpreter /home/work/code/venv-pyston2.3.3/bin/pyston`
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: rust-python-jaeger-reporter
  Running command Building wheel for rust-python-jaeger-reporter (pyproject.toml)
  Running `maturin pep517 build-wheel -i /home/work/code/venv-pyston2.3.3/bin/pyston --compatibility off`
     Compiling pyo3-build-config v0.14.0-alpha.0 (https://github.com/PyO3/pyo3?branch=main#b081e37d)
     Compiling proc-macro2 v1.0.27
     Compiling cfg-if v1.0.0
     Compiling unicode-xid v0.2.2
     Compiling libc v0.2.96
     Compiling syn v1.0.72
     Compiling proc-macro-hack v0.5.19
     Compiling autocfg v1.0.1
     Compiling smallvec v1.6.1
     Compiling log v0.4.14
     Compiling crossbeam-utils v0.8.5
     Compiling unindent v0.1.7
     Compiling scopeguard v1.1.0
     Compiling lazy_static v1.4.0
     Compiling cfg-if v0.1.10
     Compiling either v1.6.1
     Compiling byteorder v1.4.3
     Compiling integer-encoding v3.0.2
     Compiling instant v0.1.9
     Compiling try_from v0.3.2
     Compiling lock_api v0.4.4
     Compiling num-traits v0.2.14
     Compiling itertools v0.10.0
     Compiling crossbeam-channel v0.5.1
     Compiling quote v1.0.9
     Compiling num_cpus v1.13.0
     Compiling parking_lot_core v0.8.3
     Compiling pyo3-macros-backend v0.13.2 (https://github.com/PyO3/pyo3?branch=main#b081e37d)
     Compiling pyo3 v0.13.2 (https://github.com/PyO3/pyo3?branch=main#b081e37d)
     Compiling threadpool v1.8.1
     Compiling parking_lot v0.11.1
     Compiling paste-impl v0.1.18
     Compiling ordered-float v1.1.1
     Compiling ordered-float v2.5.1
     Compiling thrift v0.14.1 (https://github.com/apache/thrift?tag=v0.14.1#f6fa1794)
     Compiling paste v0.1.18
     Compiling indoc-impl v0.3.6
     Compiling pyo3-macros v0.13.2 (https://github.com/PyO3/pyo3?branch=main#b081e37d)
     Compiling indoc v0.3.6
     Compiling rust-python-jaeger-reporter v0.1.20 (/tmp/pip-req-build-z0830t_0)
      Finished release [optimized + debuginfo] target(s) in 14.11s
  🔗 Found pyo3 bindings
  🐍 Found CPython 3.8 at /home/work/code/venv-pyston2.3.3/bin/pyston
  📦 Built wheel for CPython 3.8 to /tmp/pip-req-build-z0830t_0/target/wheels/rust_python_jaeger_reporter-0.1.20-cp38-cp38-linux_x86_64.whl
  /tmp/pip-req-build-z0830t_0/target/wheels/rust_python_jaeger_reporter-0.1.20-cp38-cp38-linux_x86_64.whl
  Building wheel for rust-python-jaeger-reporter (pyproject.toml) ... done
  Created wheel for rust-python-jaeger-reporter: filename=rust_python_jaeger_reporter-0.1.20-cp38-cp38-linux_x86_64.whl size=2769879 sha256=b3680d5f0adfd8dae8c0c9945f31156b849a2d999069a342a08a9e4c0d94438c
  Stored in directory: /home/work/.cache/pip/wheels/d2/e9/a4/4f8e60a8b29246047ac44668593f563c4d6b398ef49e649311
Successfully built rust-python-jaeger-reporter
Installing collected packages: rust-python-jaeger-reporter
Successfully installed rust-python-jaeger-reporter-0.1.20
(venv-pyston2.3.3) [work@izzy rust-jaeger-python-client]$

reivilibre · 2022-05-30T16:12:06Z

@reivilibre here's my build output. Are you using a nightly version of rustc? I'm on 1.61.0.

(venv-pyston2.3.3) [work@izzy rust-jaeger-python-client]$ pip install -v ~/Downloads/rust-jaeger-python-client-anoa-pyproject.zip 
Using pip 22.1.1 from /home/work/code/venv-pyston2.3.3/lib/python3.8-pyston2.3/site-packages/pip (python 3.8)
Processing /home/work/Downloads/rust-jaeger-python-client-anoa-pyproject.zip
  Running command pip subprocess to install build dependencies
  Collecting maturin<0.13,>=0.12
    Using cached maturin-0.12.18-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.musllinux_1_1_x86_64.whl (9.6 MB)
  Collecting tomli>=1.1.0
    Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
  Installing collected packages: tomli, maturin
  Successfully installed maturin-0.12.18 tomli-2.0.1
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  🔗 Found pyo3 bindings
  🐍 Found CPython 3.8 at /home/work/code/venv-pyston2.3.3/bin/pyston
  rust_python_jaeger_reporter-0.1.20.dist-info
  Checking for Rust toolchain....
  Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-xh_ph6ig --interpreter /home/work/code/venv-pyston2.3.3/bin/pyston`
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: rust-python-jaeger-reporter
  Running command Building wheel for rust-python-jaeger-reporter (pyproject.toml)
  Running `maturin pep517 build-wheel -i /home/work/code/venv-pyston2.3.3/bin/pyston --compatibility off`
     Compiling pyo3-build-config v0.14.0-alpha.0 (https://github.com/PyO3/pyo3?branch=main#b081e37d)
     Compiling proc-macro2 v1.0.27
     Compiling cfg-if v1.0.0
     Compiling unicode-xid v0.2.2
     Compiling libc v0.2.96
     Compiling syn v1.0.72
     Compiling proc-macro-hack v0.5.19
     Compiling autocfg v1.0.1
     Compiling smallvec v1.6.1
     Compiling log v0.4.14
     Compiling crossbeam-utils v0.8.5
     Compiling unindent v0.1.7
     Compiling scopeguard v1.1.0
     Compiling lazy_static v1.4.0
     Compiling cfg-if v0.1.10
     Compiling either v1.6.1
     Compiling byteorder v1.4.3
     Compiling integer-encoding v3.0.2
     Compiling instant v0.1.9
     Compiling try_from v0.3.2
     Compiling lock_api v0.4.4
     Compiling num-traits v0.2.14
     Compiling itertools v0.10.0
     Compiling crossbeam-channel v0.5.1
     Compiling quote v1.0.9
     Compiling num_cpus v1.13.0
     Compiling parking_lot_core v0.8.3
     Compiling pyo3-macros-backend v0.13.2 (https://github.com/PyO3/pyo3?branch=main#b081e37d)
     Compiling pyo3 v0.13.2 (https://github.com/PyO3/pyo3?branch=main#b081e37d)
     Compiling threadpool v1.8.1
     Compiling parking_lot v0.11.1
     Compiling paste-impl v0.1.18
     Compiling ordered-float v1.1.1
     Compiling ordered-float v2.5.1
     Compiling thrift v0.14.1 (https://github.com/apache/thrift?tag=v0.14.1#f6fa1794)
     Compiling paste v0.1.18
     Compiling indoc-impl v0.3.6
     Compiling pyo3-macros v0.13.2 (https://github.com/PyO3/pyo3?branch=main#b081e37d)
     Compiling indoc v0.3.6
     Compiling rust-python-jaeger-reporter v0.1.20 (/tmp/pip-req-build-z0830t_0)
      Finished release [optimized + debuginfo] target(s) in 14.11s
  🔗 Found pyo3 bindings
  🐍 Found CPython 3.8 at /home/work/code/venv-pyston2.3.3/bin/pyston
  📦 Built wheel for CPython 3.8 to /tmp/pip-req-build-z0830t_0/target/wheels/rust_python_jaeger_reporter-0.1.20-cp38-cp38-linux_x86_64.whl
  /tmp/pip-req-build-z0830t_0/target/wheels/rust_python_jaeger_reporter-0.1.20-cp38-cp38-linux_x86_64.whl
  Building wheel for rust-python-jaeger-reporter (pyproject.toml) ... done
  Created wheel for rust-python-jaeger-reporter: filename=rust_python_jaeger_reporter-0.1.20-cp38-cp38-linux_x86_64.whl size=2769879 sha256=b3680d5f0adfd8dae8c0c9945f31156b849a2d999069a342a08a9e4c0d94438c
  Stored in directory: /home/work/.cache/pip/wheels/d2/e9/a4/4f8e60a8b29246047ac44668593f563c4d6b398ef49e649311
Successfully built rust-python-jaeger-reporter
Installing collected packages: rust-python-jaeger-reporter
Successfully installed rust-python-jaeger-reporter-0.1.20
(venv-pyston2.3.3) [work@izzy rust-jaeger-python-client]$

I've also now tried Rustc 1.61.0; are you building from the sdist? (The name isn't what I would conventionally expect of an sdist — looks like a repository zip if I'm being honest ;)).
(Ah, well it can't be an sdist if it's a zip!)

Note that the sdist doesn't have a Cargo.lock in it.

reivilibre · 2022-05-30T16:16:43Z

Note that the sdist doesn't have a Cargo.lock in it.

I think this is the reason why.
The later versions of PyO3 don't have the same feature flags. There is a patch in Cargo.toml to use the git main branch. Cargo.lock will contain a specific revision (until cargo update is called later), which will make this issue not appear locally/in CI, but the source distribution doesn't have Cargo.lock in it and so will try to pull the latest main branch.

anoadragon453 · 2022-05-30T17:08:13Z

@reivilibre aha, I see. It sounds like we should try updating (and pinning) the PyO3 version and subsequently updating the build flags everywhere. Looks like it was originally pinned due to a memory leak, which has presumably now been fixed if the fix was in main.

reivilibre · 2022-05-31T09:08:27Z

Pinning is optional (though perhaps we want to lock & use Cargo.lock if we're so interested in pinned deps these days — maybe not a big deal for our Jaeger reporter, but if we write more bits of Synapse in Rust), but it should be a versioned release (which follows semver by definition) rather than a floating git branch controlled by someone else! :)

erikjohnston · 2022-06-01T09:24:47Z

Yeah, I think we can move PyO3 back to using proper cargo releases and removing the git stuff.

erikjohnston · 2022-06-01T09:25:30Z

And I think if we're distributing the source then we should include the Cargo.lock. Initially I didn't bother as we used it to build wheels

pyproject.toml

erikjohnston · 2022-06-06T17:28:18Z

#6 bumps the deps

Co-authored-by: Erik Johnston <[email protected]>

anoadragon453 · 2022-07-14T13:25:59Z

I've simply accepted the suggested change from yourself @erikjohnston. Not sure if there's anything more to do?

Add a basic pyproject.toml to allow building from source distribution

8c80fae

erikjohnston reviewed Jun 6, 2022

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

Update pyproject.toml

cb2063b

Co-authored-by: Erik Johnston <[email protected]>

anoadragon453 requested a review from erikjohnston July 14, 2022 13:25

erikjohnston approved these changes Jul 14, 2022

View reviewed changes

erikjohnston merged commit 27754b5 into erikjohnston:master Jul 14, 2022

This was referenced Jul 26, 2022

Instrument /messages for understandable traces in Jaeger matrix-org/synapse#13368

Merged

Draft: Try to use use rust-jaeger-python-client matrix-org/synapse#13399

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add a basic `pyproject.toml` to allow building from source distribution #5

Add a basic `pyproject.toml` to allow building from source distribution #5

Uh oh!

anoadragon453 commented May 30, 2022

Uh oh!

reivilibre commented May 30, 2022 •

edited

Loading

Uh oh!

anoadragon453 commented May 30, 2022

Uh oh!

reivilibre commented May 30, 2022 •

edited

Loading

Uh oh!

reivilibre commented May 30, 2022 •

edited

Loading

Uh oh!

anoadragon453 commented May 30, 2022

Uh oh!

reivilibre commented May 31, 2022 •

edited

Loading

Uh oh!

erikjohnston commented Jun 1, 2022

Uh oh!

erikjohnston commented Jun 1, 2022

Uh oh!

Uh oh!

erikjohnston commented Jun 6, 2022

Uh oh!

anoadragon453 commented Jul 14, 2022

Uh oh!

Uh oh!

Add a basic pyproject.toml to allow building from source distribution #5

Add a basic pyproject.toml to allow building from source distribution #5

Uh oh!

Conversation

anoadragon453 commented May 30, 2022

Uh oh!

reivilibre commented May 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anoadragon453 commented May 30, 2022

Uh oh!

reivilibre commented May 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reivilibre commented May 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anoadragon453 commented May 30, 2022

Uh oh!

reivilibre commented May 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erikjohnston commented Jun 1, 2022

Uh oh!

erikjohnston commented Jun 1, 2022

Uh oh!

Uh oh!

erikjohnston commented Jun 6, 2022

Uh oh!

anoadragon453 commented Jul 14, 2022

Uh oh!

Uh oh!

Add a basic `pyproject.toml` to allow building from source distribution #5

Add a basic `pyproject.toml` to allow building from source distribution #5

reivilibre commented May 30, 2022 •

edited

Loading

reivilibre commented May 30, 2022 •

edited

Loading

reivilibre commented May 30, 2022 •

edited

Loading

reivilibre commented May 31, 2022 •

edited

Loading