Skip to content

Migrate bwa-mem2 from zlib to zlib-ng #56407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

brainstorm
Copy link
Member

Companion PR to #56242, let's see how much more perf is there with this switch.

@dslarm
Copy link
Contributor

dslarm commented May 28, 2025

Here's some new data for this - I ran with LD_LIBRARY_PATH pointing to installations of cloudflare zlib (github.com/cloudflare/zlib ) and zlib-ng (github.com/zlib-ng/zlib-ng) - quite easy to install - so we could check this is going to be worth doing:

TL;DR - it's worth it, for all platforms. At 64 threads, it's looking like a 5-10% performance improvement - although no benefit at say 8 or 16 threads.

MIN of time   threads      
platform combo 8 16 32 64
c7a.24xlarge (AMD Genoa) llvm 196.87 121.24 83.44 84.45
  llvm-cloudflare 198.65 115.93 82.88 78.33
  llvm-zlib-ng 202.4 109.05 84.18 78.33
  orig 220.57 121.82 84.71 85.97
c7a.24xlarge Total   196.87 109.05 82.88 78.33
c7g.16xlarge (AWS Graviton3) llvm 216.03 122.35 82.93 82.86
  llvm-cloudflare 218.6 119.82 82.06 79.91
  llvm-zlib-ng 218.84 119.04 81.67 78.46
  llvm-zlib-ng-clang19 220.56 118.56 84.97 78.8
  orig 241.2 130.41 87.1 84.78
  zlib-ng 218.39 119.88 82.02 82.94
c7g.16xlarge Total   216.03 118.56 81.67 78.46
c7i.24xlarge-smt-off (Intel Sapphire Rapids, SMT off) llvm 163.25 95.45 82.14 88.31
  llvm-cloudflare 163.33 95.2 82.4 82.19
  llvm-zlib-ng 163.38 93.96 76.99 82.1
  orig 190.84 105.04 82.53 91.65
c7i.24xlarge-smt-off Total   163.25 93.96 76.99 82.1
c7i.24xlarge-smt-on (Intel Sapphire Rapids, SMT on) llvm 163.85 94.91 83.9 81.82
  llvm-cloudflare 163.51 93.89 77.08 78.31
  llvm-zlib-ng 162.73 95.35 77.34 78.01
  orig 189.21 106.26 87.55 86.37
c7i.24xlarge-smt-on Total   162.73 93.89 77.08 78.01
c8g.24xlarge (AWS Graviton4) llvm 182.5 100.56 77.84 78.8
  llvm-cloudflare 188.31 100.47 78.47 83.46
  llvm-cloudflare-clang19 191.96 100.55 78.5 78.32
  llvm-zlib-ng 185.84 100.74 79.19 76.3
  llvm-zlib-ng-clang19 189.28 100.22 75.2 82.92
  orig 200.74 110.87 83.43 78.37
c8g.24xlarge Total   182.5 100.22 75.2 76.3

orig = current gcc build in bioconda, and the rest are your llvm artifacts with the LD_LIBRARY_PATH set to the other compression libraries. FWIW - it's not worth using LLVM for the compression library itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants