Jobs stuck in "unknown" state after runtime crash

[matthieu-beteille]

I'm experiencing an issue where after a runtime crash (typically when the event loop gets hogged for a while followed by an OOM) some jobs end up in what appears to be an "unknown" state in Redis.

These affected jobs:

• Have a deduplication key set
• Cannot be re-added/started due to the deduplication mechanism
• Are not running anymore
• Don't get stalled and retried or moved to the failed state
• Cannot be found using queue pagination methods like getJobs()
• Can only be retrieved using queue.getJob() with their specific job ID

Question

I understand this behavior is likely related to my process crashing during job execution and is not an expected case, but I'm trying to understand:

1. Are there any known edge cases where a job can end up in this limbo state without triggering the stalled job mechanism?
2. If yes, what's the recommended approach for handling such "ghost" jobs or to avoid such situations?

Right now the only way I can get out of this limbo state is by getting each individual job using .getJob and removing them individually (which is not convenient since i need to know all "ghost" job ids in advance)

I've attempted to reproduce this locally but have been unsuccessful so far (will try further). I understand this is not a complete bug report, but wanted to check if anyone has encountered similar behavior or if there are known circumstances that could lead to this.

[manast]

There are no known cases where this could happen, and we are very interested to know if they are. When you say OOM, do you mean your process went out of memory or the Redis instance itself? If it is the second, then maybe there could be some unknown edge case where atomicity is not preserved by Redis. In general when Redis hits OOM, it is quite difficult to even clean the queues as many commands stop working.

[matthieu-beteille]

Yes, the process went out of memory not the redis instance itself.

[matthieu-beteille]

yes that's what happens the OOM crashes my node process, it restarts but the jobs are in an "unknown" state and they do not stall (so they are not restarted or failed as i would expect).

[manast]

"unknown" state? that should not be possible. If your worker is processing a job, and it crashes, that job will be active for a while even though the worker has died. Later on, a new worker will detect this job is not being processed anymore and it will be moved back to wait status. If you can create a case where this does not hold true, we will certainly look into it.

[matthieu-beteille]

I understand the stalling mechanism should work, but I'm seeing this issue in a small number of edge cases, making it hard to reproduce consistently.

I thought you might have insight into potential race conditions that could theoretically cause this.

Maybe it has nothing to do with the stalling mechanism, for instance, if I am sequentially adding many new jobs to a queue at the exact moment the node process is crashing, could some of these newly added jobs end up in an ambiguous state where they don't have a proper status tracked?

I am still trying to create a repro case.

[manast]

all operations in BullMQ are atomic, so if you manage to send the "add" command to Redis, it will either succeed or fail, if the process crashed it would not affect the outcome, either it managed to send it or not.

[manast]

Btw, If we knew any theoretical case where this atomicity would not hold, then we would solve it so that it was not theoretically possible :)