[mutex.cc : 2331] RAW: Check pw->next == w failed: pw not w's predecessor

[jpfejes]

Hi,

running one of the test suites of our product we run into the above error message quite often.

The problem is in Mutex::UnlockSlow when a waiter is found in the previous iteration and it's predecessor is unknown (pw == nullptr) we assume that it's 'h', the head of the circular waiter list.

However in our test runs we see waiters can be inserted between the head and 'w' (the waiter selected), so 'w's predecessor won't be 'h'.

Any opinion?

Thanks,
JP

[derekmauro]

Can you share a test to reproduce the problem?

[jpfejes]

Maybe I can prepare one, but now it's in a complex test system, running microservices in docker containers.

[jpfejes]

Not succeeded to create a test that shows the fault, but I can provide some logs from the test case I mentioned above. Maybe this can help figure out what's the problem:

Here we can see:
[UnlockSlow] sets up walk list, w_walk: 0x7fcbbee59b00 then releases the spinlock, walk the list and selects w == 0x7fcbbee59b00, which is the head in the circular waiter list (it holds 1 element)
[LockSlowLoop] acquires spinlock, enqueues 0x7fcbbee58700 right after the head, releases spinlock
[UnlockSlow] decides to wake up w 0x7fcbbee59b00, pw is unset so it sets to the list head, but pw->next won't be w as LockSlowLoop inserted another thread after it

[mutex.cc : 2145] RAW: >[UnlockSlow]:00000016:0x51300000e148
[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x0c]->0x7fcbbee59b00
[mutex.cc : 2002] RAW: >[LockSlowLoop]:00000013:0x51300000e148
[mutex.cc : 2196] RAW:  [UnlockSlow]:00000016:0x51300000e148 {spinlock-acquired}
[mutex.cc : 2301] RAW:  [UnlockSlow]:00000016:0x51300000e148 {first-waiter-has-a-condition-or-is-a-reader}
[mutex.cc : 2015] RAW: +[LockSlowLoop]:00000013:0x51300000e148 v: [0x4c]->0x7fcbbee59b00
[mutex.cc : 2329] RAW:  [UnlockSlow]:00000016:0x51300000e148 {setup-walk-list}
[mutex.cc : 2350] RAW:  [UnlockSlow]:00000016:0x51300000e148 {walk-list-set-up} pw_walk: (nil), w_walk: 0x7fcbbee59b00
[mutex.cc : 2037] RAW:  [LockSlowLoop]:00000013:0x51300000e148 {need-to-access-waiters}
[mutex.cc : 2353] RAW:  [UnlockSlow]:00000016:0x51300000e148 {spinlock-released}
[mutex.cc : 2015] RAW: +[LockSlowLoop]:00000013:0x51300000e148 v: [0x0c]->0x7fcbbee59b00
[mutex.cc : 2400] RAW:  [UnlockSlow]:00000016:0x51300000e148 {walk-done} w: 0x7fcbbee59b00, pw: (nil), pw_walk: 0x7fcbbee59b00, w_walk: 0x7fcbbee59b00, h: 0x7fcbbee59b00, old_h: 0x7fcbbee59b00

[mutex.cc : 2037] RAW:  [LockSlowLoop]:00000013:0x51300000e148 {need-to-access-waiters}
[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x0c]->0x7fcbbee59b00
[mutex.cc : 2096] RAW:  [LockSlowLoop]:00000013:0x51300000e148 {spinlock-acquired}
[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x4c]->0x7fcbbee59b00
[mutex.cc : 2098] RAW:  [LockSlowLoop]:00000013:0x51300000e148 {Enqueue} head: 0x7fcbbee59b00, thread: 0x7fcbbee58700
[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x4c]->0x7fcbbee59b00
[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x4c]->0x7fcbbee59b00
[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x4c]->0x7fcbbee59b00
[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x4c]->0x7fcbbee59b00
[mutex.cc : 1055] RAW:  [Enqueue]:00000013 thread: 0x7fcbbee58700 -> head: 0x7fcbbee59b00, head->next: 0x7fcbbee58700, head->next->next: 0x7fcbbee59b00

[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x4c]->0x7fcbbee59b00
[mutex.cc : 2113] RAW:  [LockSlowLoop]:00000013:0x51300000e148 {spinlock-released}
[mutex.cc : 2174] RAW: +[UnlockSlow]:00000016:0x51300000e148 v: [0x0c]->0x7fcbbee59b00
[mutex.cc : 2196] RAW:  [UnlockSlow]:00000016:0x51300000e148 {spinlock-acquired}
[mutex.cc : 2290] RAW:  [UnlockSlow]:00000016:0x51300000e148 {its-a-writer-or-eof} wake: 0x7fcbbee59b00
[mutex.cc : 2295] RAW:  [UnlockSlow]:00000016:0x51300000e148 {pw == nullptr} w: 0x7fcbbee59b00, h: 0x7fcbbee59b00, h->next: 0x7fcbbee58700, h->next->next: 0x7fcbbee59b00
[mutex.cc : 2404] RAW:  [UnlockSlow]:00000016:0x51300000e148 {its-coming} h: 0x7fcbbee59b00, w: 0x7fcbbee59b00, pw: 0x7fcbbee59b00, pw->next: 0x7fcbbee58700
[mutex.cc : 2406] RAW: Check pw->next == w failed: pw not w's predecessor