Safe queue or job versioning

[saschabratton]

I am trying to version queues (or jobs) so that when new versions of the application are deployed, the old workers can finish out the old type of job, but not pickup the new ones.

I was going to use a queue name, e.g. queue-type/v${version}. So when I launch the new deployment, it will start putting new events to the new version of queue and the new workers will pick them up only. Because the message router is no longer using the old version of queue, no new jobs will be added, and the old workers can complete any jobs remaining and then shutdown the instances.

However, is this a problem or antipattern? Would this potentially create some fragments in the memory store that aren't cleaned very well? Or is there a better pattern for handling this?

[manast]

I haven't seen this pattern used before but in order to completely clean a queue without leaving any keys you would need to use the "obliterate" method for the queue. The amount of data left by a queue depends quite much on the settings you have for the completed and failed jobs, if you do not keep any jobs in completed or failed status the amount of memory consumed by the queue is quite minimal.

[saschabratton]

I see, so I would need to figure out some way to run obliterate against the old version of queues, once the old version jobs are completely drained. It might be awkward to incorporate this into deployment pipeline, but doable.

But it sounds like the overhead may be so minimal it is not a "real" issue; however, I do keep some recent cache of complete/failed jobs for real-time debugging. I suppose if I set a TTL on those, they would expire naturally without extra work. (I think this wouldn't clean like meta keys, etc.?)

@manast Is there a better pattern for queue or job versioning that you have seen?

In most cases, new version of jobs is probably just handling some new parameter, so old version worker likely won't choke on it, but if old workers pick it up during soak time after new version deployment, we are not getting the new functionality from new jobs, which is undesirable.

But in some cases, it is possible old workers might fail on new job versions (and vice versa).

Another option I considered is using a new database in the memory store for new deployment? I preferred using the queue name versioning because then I do not have to change an environment configuration value on the deployments, since the version can be taken from the new package.

[saschabratton]

@manast any thoughts on a "best practice" pattern here?