PDF files do not upload to Minio

[bytersproblem]

I have trouble with uploading PDF files to Omnivore. It reports that the file cannot be uploaded and as the error in the browser it dipslays Minio URL (docker-compose setup) localhost:1010. I tried to change the port in docker-compose.yml to 9000, amended all the instances of minio to :9000, but still it doesn't work.
I don't know it nginx config should be changed or something in the docker-compose.yml or .env.
Thanks!

[bytersproblem]

I found this in omnivore-api logs:

{
    "replicationMode": "master",
    "parameters": [
        "d7",
        "e849",
        "SUCCEEDED",
        "https://omnivore.app/attachments/u/9058f7e5-150d-4026-9247-5ce36b24e9a6/3kalendarrad202425osnovneakademskestudije.pdf",
        "u-9058-f-7-e-5-150-d-4026-9247-5-ce-36-b-24-e-9-a-6-3-kalendarra-1955c204242",
        "3 kalendar rad 2024 25 osnovne akademske studije",
        0,
        "FILE",
        "9058f7e5-150d-4026-9247-5ce36b24e9a6",
        "PDF",
        ""
    ]
}
[app@13:09:32] COMMIT
{
    "replicationMode": "master"
}
[app@13:09:33] Thumbnail job created
{
    "id": "28559"
}

It is the https://omnivore.app and not my tld.

[Podginator]

Hey - That omnivore.app thing is quite misleading, don't worry about it too much. I think it is harc coded, I will create an issue to remove it, but as far as I know this is not actually used in the App.

The .env that you need to change is the following:

LOCAL_MINIO_URL=https://omnivore-bucket.podginator.com

In the guide it should have you point your minio instance to the net. I realise that this is confusing due to the LOCAL minio url. I will update the guide to make this clearer.

Minio generates a URL for you, and this URL provides a signed url to upload to. This is the same way that GCS and S3 works, and how it used to work on the cloud instance. Therefore the LOCAL_MINIO_URL has to point to a service that accepts that.

If you're running it locally, it'll be localhost:1010, but if you have it hosted it'll be whatever you have pointed the minio instance to.

[bytersproblem]

Now I receive error 400. I am sending my enviroment.

.env

# Postgres & Migrate

POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=omnivore
PGPASSWORD=postgres
POSTGRES_USER=postgres
PG_REPLICATION=false

PG_HOST=postgres
PG_PASSWORD=app_pass
PG_DB=omnivore
PG_USER=app_user
PG_PORT=5432
PG_POOL_MAX=20

# API

API_ENV=local
IMAGE_PROXY_SECRET=xxx
JWT_SECRET=xxx
SSO_JWT_SECRET=xxx
GATEWAY_URL=http://api:8080/api
CONTENT_FETCH_URL=http://content-fetch:8080/?token=some_token
GCS_USE_LOCAL_HOST=true
GCS_UPLOAD_BUCKET=omnivore
AUTO_VERIFY=true
AWS_ACCESS_KEY_ID=minio   # Used for Minio S3 Client
AWS_SECRET_ACCESS_KEY=miniominio
AWS_REGION=us-east-1
CONTENT_FETCH_QUEUE_ENABLED=true

IMAGE_PROXY_URL=https://omnivore.mydomain.com/images # Need to change this for NGINX
CLIENT_URL=https://omnivore.mydomain.com # Need to change this when using NGINX
LOCAL_MINIO_URL=https://omnivore.mydomain.com/bucket

# Redis
REDIS_URL=redis://redis:6379/0

#MAIL
WATCHER_API_KEY=xxx
LOCAL_EMAIL_DOMAIN=domain.tld
LOCAL_EMAIL_DOMAIN=domain.tld
SNS_ARN=arn_of_sns #for if you use SES and SNS for Email.

# Web
APP_ENV=prod
BASE_URL=https://omnivore.mydomain.com # Front End - Need to change this when using NGINX
SERVER_BASE_URL=https://omnivore.mydomain.com # API Server, need to change this when using NGINX
HIGHLIGHTS_BASE_URL=https://omnivore.mydomain.com # Front End - Need to change this when using NGINX

# Content Fetch

VERIFICATION_TOKEN=some_token
REST_BACKEND_ENDPOINT=http://api:8080/api
SKIP_UPLOAD_ORIGINAL=true

# Minio
MINIO_ACCESS_KEY=minio
MINIO_SECRET_KEY=miniominio
AWS_S3_ENDPOINT_URL=http://minio:9000

# Export
EXPORT_QUEUE_NAME=omnivore-backend-queue

# Youtube
YOUTUBE_TRANSCRIPT_PROMPT=Reformat the supplied transcript data adding formatting, puncuation, and paragraphs, but do not summarize or change the content. Format the output as markdown.
YOUTUBE_MAXIMUM_VIDEO_DURATION_TRANSCRIPT=1801

nginx/server.conf

     upstream omnivore_web {
    	ip_hash;
    	server 192.168.0.22:3000;
     }

     upstream omnivore_backend {
    	ip_hash;
    	server 192.168.0.22:4000;
     }

     upstream omnivore_imageproxy {
    	ip_hash;
    	server 192.168.0.22:7070;
     }

    upstream omnivore_bucket {
    	ip_hash;
    	server 192.168.0.22:9000;
    }
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;

`nginx/conf.d/domain.conf
}

server {
    listen 80;
    listen [::]:80;
    server_name omnivore;

...etc

/nginx/conf.d/domain/d/omnivore.conf

 # Override for authentication on the frontend
        location /api/client/auth {
            proxy_pass http://omnivore_web;
        }

        # Save Route (Frontend)
        location /api/save {
            proxy_pass http://omnivore_web;
        }

        # API
        location /api {
            proxy_pass http://omnivore_backend;
        }

        # Minio
        location /bucket {
            proxy_pass http://omnivore_bucket;
        }

        # ImageProxy
        location /images {
            rewrite ^/images/(.*)$ /$1 break;
            proxy_pass http://omnivore_imageproxy;
        }

        # FrontEnd application
        location / {
            proxy_pass http://omnivore_web;
        }

        # Mail Proxy
        location /mail {
            proxy_pass http://localhost:4398/mail;
        }

docker-compose.yml

version: '3'
x-postgres:
  &postgres-common
  image: "ankane/pgvector:v0.5.1"
  user: postgres
  healthcheck:
    test: "exit 0"
    interval: 2s
    timeout: 12s
    retries: 3

services:
  postgres:
    <<: *postgres-common
    container_name: "omnivore-postgres"
    volumes:
      - pgdata:/var/lib/postgresql/data
    env_file:
      - .env

  migrate:
    image: "ghcr.io/omnivore-app/sh-migrate:latest"
    container_name: "omnivore-migrate"
    command: '/bin/sh ./packages/db/setup.sh' # Also create a demo user with email: demo@omnivore.app, password: demo_password
    env_file:
      - .env
    depends_on:
      postgres:
        condition: service_healthy

  api:
    image: "ghcr.io/omnivore-app/sh-backend:latest"
    container_name: "omnivore-api"
    ports:
      - "4000:8080"
    healthcheck:
      test: ["CMD-SHELL", "nc -z 0.0.0.0 8080 || exit 1"]
      interval: 15s
      timeout: 90s
      retries: 6
    env_file:
      - .env
    depends_on:
      migrate:
        condition: service_completed_successfully

  queue-processor:
    image: "ghcr.io/omnivore-app/sh-queue-processor:latest"
    container_name: "omnivore-queue-processor"
    env_file:
      - .env
    depends_on:
      api:
        condition: service_started

  web:
    image: "ghcr.io/omnivore-app/sh-web:latest"
    container_name: "omnivore-web"
    ports:
      - "3000:8080"
    env_file:
      .env
    depends_on:
      api:
        condition: service_healthy

  image-proxy:
    image: "ghcr.io/omnivore-app/sh-image-proxy:latest"
    container_name: "omnivore-image-proxy"
    ports:
      - "7070:8080"
    env_file:
      - .env

  content-fetch:
    image: "ghcr.io/omnivore-app/sh-content-fetch:latest"
    container_name: "omnivore-content-fetch"
    ports:
      - "9090:8080"
    environment:
      - USE_FIREFOX=true # Using Firefox here because the official chrome version seems to freeze a lot in Docker.
    env_file:
      - .env
    depends_on:
      redis:
        condition: service_healthy
      api:
        condition: service_healthy

  redis:
    image: "redis:7.2.4"
    container_name: "omnivore-redis"
    expose:
      - 6379
    ports:
      - "6379:6379"
    healthcheck:
      test: [ "CMD", "redis-cli", "--raw", "incr", "ping" ]
    volumes:
      - redis_data:/data

  minio:
    image: minio/minio
    expose:
      - 9000
    ports:
      - "9000:9000"
    healthcheck:
      test: [ "CMD", "mc", "ready", "local" ]
      interval: 5s
      timeout: 1s
    env_file:
      - .env
    command: server /data
    volumes:
      - minio_data:/data

  createbuckets:
    image: minio/mc
    environment:
      - MINIO_ACCESS_KEY=minio
      - MINIO_SECRET_KEY=miniominio
      - BUCKET_NAME=omnivore
      - ENDPOINT=http://minio:9000
      - AWS_S3_ENDPOINT_URL=http://minio:9000
    depends_on:
      - minio
    entrypoint: >
      /bin/bash -c "
      sleep 5;
      until (/usr/bin/mc config host add myminio http://minio:9000 minio miniominio) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc mb myminio/omnivore;
      /usr/bin/mc policy set public myminio/omnivore;
      exit 0;
      "
  mail-watch-server:
    image: "ghcr.io/omnivore-app/sh-local-mail-watcher:latest"
    container_name: "omnivore-mail-watch-server"
    ports:
      - "4398:8080"
    env_file:
      - .env
    depends_on:
      redis:
        condition: service_healthy
volumes:
  pgdata:
  redis_data:
  minio_data:

[Podginator]

Can you try it with

rewrite ^/bucket/(.*)$ /$1 break;

in this section of the nginx file?

  # Minio
        location /bucket {
            proxy_pass http://omnivore_bucket;
        }

Above the proxy pass

[bytersproblem]

It is still error 400 in the console. It seems that the request does not reach Minio docker, since there is no activity in the logs of the container. Also nothing in the web container.

[Podginator]

I think I might know the issue, but you'll have to give me some time to investigate it a bit further. I think it's the way the signing is done, stripping the /bucket causes the signature to be wrong. That's my theory at least. I will have to test it a bit further.

[cerbaire]

Hey,
I wonder if there is some guidelines to configure the Synology reverse proxy? I see some rewriting configuration in the Nginx configuration examples. Not sure how to apply this on Synology.