Separate settings for each queue

[nilpoona]

Are there any plans to make it possible to configure SQS Active Job settings on a per queue basis, as shown below?

queue1:
  max_messages: 10
  visibility_timeout: 30
queue2:
  max_messages: 5
  visibility_timeout: 100

[alextwoods]

Since the poller currently runs independently per queue, it would be relatively straightforward to accept per queue configuration. I'll mark this as a feature request and we'd be happy to take a PR for it.

In the mean time - when running workers you can specify the max_messages and visibility_timeout per queue via arguments, eg: RAILS_ENV=development bundle exec aws_sqs_active_job --queue default --max_messages 5 --visibility_timeout 100 (run bundle exec aws_sqs_active_job --help for a complete list of arguments).

[nilpoona]

Thanks. I will send you a PR later.
In the meantime, I will try to deal with the method you suggested of specifying the poller argument directly.

[mullermp]

@nilpoona We are looking to do a minor release of this gem soon. Did you still intend to send a PR for this?

[nilpoona]

@mullermp
We did not intend to make it in time for that release. We are not in a hurry.

[januszm]

Any chance to have the poller poll from all queues from the config file? That would mimic the behaviour of Sidekiq/Shoryuken task queues that are widely adopted already. I would rather avoid having to run multiple pollers just to poll from different queues, and instead run a single background process.

[mullermp]

@januszm Can you be more specific about your use case and why you need a single process for multiple queues, rather than one queue and only one process? Managing multiple processes outside of the ruby environment (and out of ruby code) reduces a large amount of complexity and is more performant.

[januszm]

@mullermp it seems to me that this is the simplest use case you can imagine, i.e. we want to have one Worker tier, which is easier to manage, and this worker tier is supposed to process tasks from the default queue + possibly a few other optional queues. I know that it is better to have more worker tiers if the tasks require completely different servers, e.g. CPU intensive and those that require a lot of RAM. But for most simple tasks, one is enough. Use case: I have a separate team in the company that has its own SNS topic to which it sends tasks to be executed by my system. I would like to have the possibility for my worker tier to also process messages from the queue subscribed to this SNS topic, so I would add it to config/aws_sqs_active_job.yml and expect that one poller will also handle this additional queue.

[alextwoods]

With the 1.0 release of aws-activejob-sqs we've added separate configuration for queues and added the ability to run the poller for multiple/all queues. Thanks for the suggestions!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.