regression: cargo 1.88 nondeterministically hangs with nested parallel builds

[sunshowers]

Problem

Hi there --

I've encountered a regression in Cargo 1.88 in a situation where if:

• a lot of Cargo builds are run in parallel in a workspace
• one of the crates has a build script that does its own Cargo build

Then cargo build can nondeterministically hang. Cargo 1.87 does not have this bug.

A minimized reproduction is at https://github.com/nextest-rs/test-cargo-stress. On my 16c/32t Ryzen 7950x this bug reproduces within a couple of seconds.

I did a bisect of Rust nightlies and found that the regression occurred between nightly-2025-04-28 and nightly-2025-04-29.

% cargo +nightly-2025-04-28 -V
cargo 1.88.0-nightly (d811228b1 2025-04-15)

% cargo +nightly-2025-04-29 -V
cargo 1.88.0-nightly (7918c7eb5 2025-04-27)

The range of Cargo commits is https://github.com/rust-lang/cargo/compare/d811228b1..7918c7eb5.

(This was originally nextest-rs/nextest#2463.)

Steps

1. Clone https://github.com/nextest-rs/test-cargo-stress and cd into it
2. cargo run

Possible Solution(s)

No response

Notes

The bug does not appear to be OS-specific. I've seen reproductions on Linux, Windows and illumos.

Version

% cargo -v -V
cargo 1.88.0 (873a06493 2025-05-10)
release: 1.88.0
commit-hash: 873a0649350c486caf67be772828a4f36bb4734c
commit-date: 2025-05-10
host: x86_64-unknown-linux-gnu
libgit2: 1.9.0 (sys:0.20.0 vendored)
libcurl: 8.12.1-DEV (sys:0.4.80+curl-8.12.1 vendored ssl:OpenSSL/3.4.1)
ssl: OpenSSL 3.4.1 11 Feb 2025
os: Ubuntu 24.4.0 (noble) [64-bit]

[epage]

Can you check against master?

We had a deadlock fixed in #15698 which was backported to 1.89 in #15699.

[sunshowers]

Out right now, will check later. Thanks for the pointer!

If that is indeed the bug, is there a workaround we can do within nextest?

[epage]

If its within that commit range, the main thing I can think of that causes Cargo to hit it is the stabilization of GC. If thats the case, then you might be able to avoid it if you pass in the env variable to disable GC

[sunshowers]

Ah that was my suspicion, but I think I did manage to get a repro with CARGO_CACHE_AUTO_CLEAN_FREQUENCY=never. (I want to double check this, though.)

Notably, I could only get the bug to repro with the nested build.

[sunshowers]

Quick update:

• The bug repros with CARGO_CACHE_AUTO_CLEAN_FREQUENCY=never.
• It doesn't repro with 1.89.0-beta.4 or nightly.
• The bug was fixed in nightly-2025-06-26:

% cargo +nightly-2025-06-25 -V
cargo 1.90.0-nightly (84709f085 2025-06-22)

% cargo +nightly-2025-06-26 -V
cargo 1.90.0-nightly (409fed7dc 2025-06-23)

The range of commits is 84709f0...409fed7 -- it does seem like #15698 fixes this.

Not sure what to do about working around this bug yet, unfortunately.

[mati865]

In a real project (test in https://github.com/davidlattimore/wild) I've found that tinkering with kernel schedulers can increase or even make the problem go away, it doesn't help with stress test though. Just lowers or increases the time it takes.

On my 5950X and https://github.com/sched-ext/scx version 1.0.14 using schedulers: bpfland (depending on the test order this one can also make it worse), lavd or p2dq works around the problem with Wild's tests, and schedulers: flash, rustland and rusty make it hang more often.

Far from an easy to deploy workaround and not a perfectly reliable, but still might be helpful to someone.

[sunshowers]

Thanks @mati865 for the investigation. I happen to use bpfland, and as you said it doesn't help with stress tests. On nextest's tests it's definitely rarer for me than it seems to be in some other Linux environments, as well as other platforms like illumos and Windows.