Skip to content

Commit 5dad2d4

Browse files
authored
Turbopack: avoid negative uppers and follower and use retry loop instead (#79451)
### Why? Multiple graph modifications can happen concurrently. Each modification only locks one or two tasks in the graph at a single step, but a full graph modification could affect many tasks in graph. So modifications are executed step by step only affecting a maximum of two tasks on each step. Because of this operation mode modifications bubble through the graph and multiple modifications can bubble through the graph concurrently, even at different speeds. One important case is that an "add" modification could be partially applied, while a "remove" modification for the same item is triggered. This "remove" modification could be faster and would try to remove items from tasks where they have not been added yet (as the "add" modification is still in progress). To account for this problem we used to allow negative counts on each item, so the "remove" modification would temporarily cause negative item counts until the "add" modification returns the count to zero. But some decisions (e. g. if a upper or follower should be removed) are based on the values of these counts, so this approach doesn't work correctly in some rare race condition edge cases. So this change uses a different approach. ### What? We want to avoid negative counts, so when the "remove" modification tries to remove an non-existent item, it should wait for the "add" modification to catch up before continuing. There are two cases here: Either the "add" modification is done by a different thread, or it's done by the same thread and is in the queue. The first case is handled by a busy loop yielding. The second case is handled by re-enqueuing the job at the end of the queue. There is a limit of 10s for retries after that we panic. This allows to detect invalid graph scenarios and points out bugs in the graph implementation. Let's hope we never see this panic. In the old implementation this problem was silently ignored: Count got negative, but that was just accepted and has let to weird side effects (like errors/collectibles not disappearing or hanging compilation).
1 parent 7903a57 commit 5dad2d4

File tree

2 files changed

+330
-193
lines changed

2 files changed

+330
-193
lines changed

0 commit comments

Comments
 (0)