Replies: 5 comments 2 replies
-
|
If they are failing, check the error message of the job to know why they are failing. |
Beta Was this translation helpful? Give feedback.
-
|
@manast The failing job are all pointing to similar error that the database object on which the job is trying to process does not exist even though the data is present. When we try to rerun the job manully again, at times it is getting processed successfully. Below is an example: There is job name : autoProcessVendorPayoutTask failing with below error: 
What is strange to me is that no worker is picking this job and following are the queue events: `Job delayed: 98784866-4965-44e6-abea-c024b0daf4fd, Delay: 1742221820144, queueName: P3 Job waiting in the queue: 98784866-4965-44e6-abea-c024b0daf4fd, queueName: P3, Previous state: waiting Job failed: 98784866-4965-44e6-abea-c024b0daf4fd, Reason: No vendor Transaction Payout Exist, queueName: P3, job: {"name":"autoProcessVendorPayoutTask","data":23343,"opts":{"attempts":0,"backoff":{},"removeOnFail":{"age":86400,"count":1000},"jobId":"98784866-4965-44e6-abea-c024b0daf4fd","removeOnComplete":{"age":3600,"count":1000},"delay":15000},"id":"98784866-4965-44e6-abea-c024b0daf4fd","progress":0,"returnvalue":null,"stacktrace":["Error: No vendor Transaction Payout Exist\n at VendorPayoutService.processedVendorPayout (/usr/src/app/app/services/vendorPayoutService/VendorPayoutService.ts:835:19)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async AutoProcessVendorPayoutTask.processTask (/usr/src/app/app/task/vendorPayout/autoProcessVendorPayoutTask.ts:19:35)\n at async JobProcessor.execute (/usr/src/app/app/task/base/jobProcessor.ts:32:26)\n at async Worker.worker.bullmq_1.Worker.autorun [as processFn] (/usr/src/app/app/task/base/bullWorker.ts:26:34)\n at async /usr/src/app/node_modules/bullmq/src/classes/worker.ts:910:26\n at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/src/classes/worker.ts:1174:16)"],"delay":0,"priority":0,"attemptsStarted":1,"attemptsMade":1,"timestamp":1742221805144,"queueQualifiedName":"bull:P3","finishedOn":1742221820177,"processedOn":1742221820156,"failedReason":"No vendor Transaction Payout Exist"}` The job is getting processed 15 sec after record is getting created, so I don't see a way why it would not find the record. Below are cloud run logs. Typically we see worker picking up the job and hence job move to active state but in this case job is not even coming in active state 
|
Beta Was this translation helpful? Give feedback.
-
|
Since the job is failing with an error produced by your code, thats where I would put the debugging efforts. The job has started but some issue in the processor code raises an exception and the job is marked as failed. |
Beta Was this translation helpful? Give feedback.
-
|
I will move this as a discussion since there is no indication of being a bug in BullMQ. |
Beta Was this translation helpful? Give feedback.
-
|
@manast Is it possible for queue to not emit 'Active' Event and emit directly the "failed" event? Also, can it happen that the job is failed without worker picking it up? We are seeing this happening randomly with different task and worker logs are not coming for those jobs. |
Beta Was this translation helpful? Give feedback.
-
|
How are your workers running on Cloud Run? With a service or a job? If using a service and min instances to keep it awake, ensure you disable the "only allocate CPU during request processing", since workers are not a typical fit for Cloud Run services since they are usually used for request/response services. |
Beta Was this translation helpful? Give feedback.
-
|
We are running as a service with minimum count and we have disabled only allocate CPU during request processing. Above problems was very random: We had to use another instance of redis to get rid of this problem. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Version
v5.43.1
Platform
NodeJS
What happened?
Randomly some of the jobs are moving directly to failed status without being picked by the worker on cloud run (GCP) I am saying so because worker event logs are not coming.
Using Redis 6.X available on GCP.
When i tried running the same docker contrainer with local redis it works fine but on testing environment, randomly some events are getting failed with error like invalid id on db even though id exist in the database.
I am getting proper worker logs and other jobs are getting processed but some job are getting impacted randomly.
Please suggest how should I proceed to debug what could be wrong? Is it some config or some application code?
How to reproduce.
No response
Relevant log output
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions