rust-lld: error: duplicate symbol in ci error: duplicate symbol

[yihong0618]

Hi I just try to use features = ["zlib-rs"] in greptimedb

but it failed in ci:
patch link: https://github.com/GreptimeTeam/greptimedb/pull/5706/files

ci failed link: https://github.com/GreptimeTeam/greptimedb/actions/runs/13848925444/job/38752767800

some error message:

  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: rust-lld: error: duplicate symbol: crc32
          >>> defined at crc32.c:1016 (src/zlib/crc32.c:1016)
          >>>            0dc752f03a07a721-crc32.o:(crc32) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:99 (src/lib.rs:99)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.crc32+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: crc32_combine
          >>> defined at crc32.c:1029 (src/zlib/crc32.c:1029)
          >>>            0dc752f03a07a721-crc32.o:(crc32_combine) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:134 (src/lib.rs:134)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.crc32_combine+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: adler32
          >>> defined at adler32.c:128 (src/zlib/adler32.c:128)
          >>>            0dc752f03a07a721-adler32.o:(adler32) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:164 (src/lib.rs:164)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.adler32+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: adler32_combine
          >>> defined at adler32.c:158 (src/zlib/adler32.c:158)
          >>>            0dc752f03a07a721-adler32.o:(adler32_combine) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:199 (src/lib.rs:199)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.adler32_combine+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: uncompress
          >>> defined at uncompr.c:83 (src/zlib/uncompr.c:83)
          >>>            0dc752f03a07a721-uncompr.o:(uncompress) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:269 (src/lib.rs:269)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.uncompress+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflate
          >>> defined at inflate.c:590 (src/zlib/inflate.c:590)
          >>>            0dc752f03a07a721-inflate.o:(inflate) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:324 (src/lib.rs:324)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflate+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateEnd
          >>> defined at inflate.c:1266 (src/zlib/inflate.c:1266)
          >>>            0dc752f03a07a721-inflate.o:(inflateEnd) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:348 (src/lib.rs:348)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateEnd+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateBackInit_
          >>> defined at infback.c:27 (src/zlib/infback.c:27)
          >>>            0dc752f03a07a721-infback.o:(inflateBackInit_) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:382 (src/lib.rs:382)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateBackInit_+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateBack
          >>> defined at infback.c:243 (src/zlib/infback.c:243)
          >>>            0dc752f03a07a721-infback.o:(inflateBack) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:402 (src/lib.rs:402)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateBack+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateBackEnd
          >>> defined at infback.c:621 (src/zlib/infback.c:621)
          >>>            0dc752f03a07a721-infback.o:(inflateBackEnd) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:429 (src/lib.rs:429)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateBackEnd+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateCopy
          >>> defined at inflate.c:1439 (src/zlib/inflate.c:1439)
          >>>            0dc752f03a07a721-inflate.o:(inflateCopy) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:458 (src/lib.rs:458)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateCopy+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateMark
          >>> defined at inflate.c:1510 (src/zlib/inflate.c:1510)
          >>>            0dc752f03a07a721-inflate.o:(inflateMark) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:493 (src/lib.rs:493)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateMark+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateSync
          >>> defined at inflate.c:1375 (src/zlib/inflate.c:1375)
          >>>            0dc752f03a07a721-inflate.o:(inflateSync) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:527 (src/lib.rs:527)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateSync+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateSyncPoint
          >>> defined at inflate.c:1431 (src/zlib/inflate.c:1431)
          >>>            0dc752f03a07a721-inflate.o:(inflateSyncPoint) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:544 (src/lib.rs:544)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateSyncPoint+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateInit_
          >>> defined at inflate.c:219 (src/zlib/inflate.c:219)
          >>>            0dc752f03a07a721-inflate.o:(inflateInit_) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:578 (src/lib.rs:578)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateInit_+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateInit2_
          >>> defined at inflate.c:179 (src/zlib/inflate.c:179)
          >>>            0dc752f03a07a721-inflate.o:(inflateInit2_) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:611 (src/lib.rs:611)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateInit2_+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflatePrime
          >>> defined at inflate.c:223 (src/zlib/inflate.c:223)
          >>>            0dc752f03a07a721-inflate.o:(inflatePrime) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:672 (src/lib.rs:672)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflatePrime+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateReset
          >>> defined at inflate.c:130 (src/zlib/inflate.c:130)
          >>>            0dc752f03a07a721-inflate.o:(inflateReset) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:698 (src/lib.rs:698)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateReset+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateReset2
          >>> defined at inflate.c:141 (src/zlib/inflate.c:141)
          >>>            0dc752f03a07a721-inflate.o:(inflateReset2) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:725 (src/lib.rs:725)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateReset2+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: duplicate symbol: inflateSetDictionary
          >>> defined at inflate.c:1299 (src/zlib/inflate.c:1299)
          >>>            0dc752f03a07a721-inflate.o:(inflateSetDictionary) in archive /greptimedb/target/debug/deps/liblibz_sys-3df317acd34adec9.rlib
          >>> defined at lib.rs:764 (src/lib.rs:764)
          >>>            libz_rs_sys-4a09f9934c9134bc.libz_rs_sys.9761070cd71d0df0-cgu.0.rcgu.o:(.text.inflateSetDictionary+0x0) in archive /greptimedb/target/debug/deps/liblibz_rs_sys-4a09f9934c9134bc.rlib
          
          rust-lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
          collect2: error: ld returned 1 exit status

[Byron]

Thanks for reporting.

Here @folkertdev might know what may be causing this.

[folkertdev]

Well the problem is that the zlib symbols are included in your test binary twice. I suspect the culprit is this

│   │   │       │   │       │   │   │   │   │   │   │   └── shadow-rs v0.38.1
│   │   │       │   │       │   │   │   │   │   │   │       ├── const_format v0.2.34 (*)
│   │   │       │   │       │   │   │   │   │   │   │       ├── git2 v0.20.0
│   │   │       │   │       │   │   │   │   │   │   │       │   ├── bitflags v2.9.0 (*)
│   │   │       │   │       │   │   │   │   │   │   │       │   ├── libc v0.2.171
│   │   │       │   │       │   │   │   │   │   │   │       │   ├── libgit2-sys v0.18.0+1.9.0
│   │   │       │   │       │   │   │   │   │   │   │       │   │   ├── libc v0.2.171
│   │   │       │   │       │   │   │   │   │   │   │       │   │   └── libz-sys v1.1.22 (*)

you can see these diagrams with cargo tree. The above is a dev dependency, and I suspect it'll be hard to remove.

We run into this problem in zlib-rs itself when testing versus another implementation. We have two workarounds

custom-prefix = [] # use the LIBZ_RS_SYS_PREFIX to prefix all exported symbols
testing-prefix = [] # prefix all symbols with LIBZ_RS_SYS_TEST_ for testing

These are feature flags of libz-rs-sys. You might be able to use a fake dependency on libz-rs-sys to force the right behavior, e.g.

flate2 = { version = "1.1.0", default-features = false, features = ["zlib-rs"] }
# prefix the zlib symbols so they don't clash with other dependencies. update libz-rs-sys in lockstep with flate2
libz-rs-sys = { version = "0.4.2", default-features = false, features = ["testing-prefix"] }

That testing prefix is an internal feature really, but might get you unstuck. We will add something more dedicated in the next release.

[yihong0618]

Well the problem is that the zlib symbols are included in your test binary twice. I suspect the culprit is this

│   │   │       │   │       │   │   │   │   │   │   │   └── shadow-rs v0.38.1
│   │   │       │   │       │   │   │   │   │   │   │       ├── const_format v0.2.34 (*)
│   │   │       │   │       │   │   │   │   │   │   │       ├── git2 v0.20.0
│   │   │       │   │       │   │   │   │   │   │   │       │   ├── bitflags v2.9.0 (*)
│   │   │       │   │       │   │   │   │   │   │   │       │   ├── libc v0.2.171
│   │   │       │   │       │   │   │   │   │   │   │       │   ├── libgit2-sys v0.18.0+1.9.0
│   │   │       │   │       │   │   │   │   │   │   │       │   │   ├── libc v0.2.171
│   │   │       │   │       │   │   │   │   │   │   │       │   │   └── libz-sys v1.1.22 (*)

you can see these diagrams with cargo tree. The above is a dev dependency, and I suspect it'll be hard to remove.

We run into this problem in zlib-rs itself when testing versus another implementation. We have two workarounds

custom-prefix = [] # use the LIBZ_RS_SYS_PREFIX to prefix all exported symbols
testing-prefix = [] # prefix all symbols with LIBZ_RS_SYS_TEST_ for testing

These are feature flags of libz-rs-sys. You might be able to use a fake dependency on libz-rs-sys to force the right behavior, e.g.

flate2 = { version = "1.1.0", default-features = false, features = ["zlib-rs"] }

prefix the zlib symbols so they don't clash with other dependencies. update libz-rs-sys in lockstep with flate2

libz-rs-sys = { version = "0.4.2", default-features = false, features = ["testing-prefix"] }
That testing prefix is an internal feature really, but might get you unstuck. We will add something more dedicated in the next release.

Thank you very much

[folkertdev]

Oh btw the latest version of zlib-rs (part of the latest release of flate2) no longer exports the symbols (by default), so the workaround with the testing-prefix is no longer needed, and this issue should be resolved at this point.

[yihong0618]

Thank you