Skip to content

feat: Configure micoservices queues using env variable #17675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

macbury
Copy link

@macbury macbury commented Apr 17, 2025

Description

This PR introduces fine-grained control over microservices queues using two new environment variables:

  • IMMICH_MICROSERVICES_QUEUES_INCLUDE: Specify which queues to process
  • IMMICH_MICROSERVICES_QUEUES_EXCLUDE: Specify which queues to ignore

Why

Many friends that I know run Immich on NAS devices like QNAP and Synology, which typically lack the computing power necessary for resource-intensive tasks such as video encoding and machine learning. This leads them to deploy the machine-learning container on a separate, more powerful machine.
In my personal setup, I run Immich on a QNAP NAS and initially tried moving all microservices to a more powerful homelab server with an AMD Zen9 processor. I mounted the QNAP share via Samba using autofs. While this approach significantly improved video transcoding performance, it introduced issues with external library watching and scanning due to inherent limitations of Samba shares.
The new queue-specific environment variables allow for perfect distribution:

NAS

services:
  immich-server:
    ...
    environment:
      IMMICH_WORKERS_INCLUDE: 'api'
  
  immich-microservices:
    ...
    environment:
      IMMICH_WORKERS_INCLUDE: 'microservices'
+     IMMICH_MICROSERVICES_QUEUES_EXCLUDE: 'videoConversion,thumbnailGeneration'

Server with GPU

services:
  immich-machine-learning:
    ...
  immich-microservices:
    ...
    environment:
      IMMICH_WORKERS_INCLUDE: 'microservices'
+     IMMICH_MICROSERVICES_QUEUES_INCLUDE: 'videoConversion,thumbnailGeneration'

Fixes # (issue)

How Has This Been Tested?

I implemented some unit test. Also I did perform manual test:

I tested this locally by running localy server twice:

# run first process without video transcoding
cd server/
export IMMICH_WORKERS_INCLUDE=microservices
export IMMICH_MICROSERVICES_QUEUES_EXCLUDE='videoConversion,thumbnailGeneration'
npm start
# run second process with video transcoding
cd server/
export IMMICH_WORKERS_INCLUDE=microservices
export IMMICH_MICROSERVICES_QUEUES_INCLUDE='videoConversion,thumbnailGeneration'
npm start

Then I just opened web ui, found video run transcoding tasks for all. I should see video transcoding only in second process. All other tasks still worked as before.

Checklist:

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation if applicable
  • I have no unrelated changes in the PR.
  • I have confirmed that any new dependencies are strictly necessary.
  • I have written tests for new code (if applicable)
  • I have followed naming conventions/patterns in the surrounding code
  • All code in src/services/ uses repositories implementations for database calls, filesystem operations, etc.
  • All code in src/repositories/ is pretty basic/simple and does not have any immich specific logic (that belongs in src/services/)

@macbury macbury requested a review from danieldietzler as a code owner April 17, 2025 18:24
@github-actions github-actions bot added documentation Improvements or additions to documentation 🗄️server labels Apr 17, 2025
Copy link
Contributor

github-actions bot commented Apr 17, 2025

Label error. Requires exactly 1 of: changelog:.*. Found: documentation, 🗄️server. A maintainer will add the required label.

@macbury macbury changed the title [feature] configure microsrvices queues using env variable [feature] Configure micoservices queues using env variable Apr 17, 2025
@macbury macbury changed the title [feature] Configure micoservices queues using env variable feat: Configure micoservices queues using env variable Apr 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation 🗄️server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant