Cannot delete scanners. Get 504 Gateway Time-out.

[Aslan-Liu]

Hi all,

I tried to delete trivy scanner which is registered in Habor before. However, I cannot delete it via Harbor UI or even using Harbor restful api. Always get 504 Gateway Time-out.

I also try to see logs from harbor-core container, but I cannot see any error messages from that.
Harbor now is running on K8S, and use NFS to mount all necessary folders in containers.

• harbor version: 2.5.1
• containerd version: 1.7.13

Here are pods in my harbor namespace.

NAME                                          READY   STATUS    RESTARTS        AGE
harbor-chartmuseum-7884fcd9c4-b2jwq           1/1     Running   0               6d
harbor-core-8bc64f4c-v4465                    1/1     Running   0               6d
harbor-jobservice-6b5b69f8fb-26k5z            1/1     Running   0               6d
harbor-jobservice-6b5b69f8fb-4kqmh            1/1     Running   0               6d
harbor-jobservice-6b5b69f8fb-f4k7n            1/1     Running   0               6d
harbor-nginx-c7dfb8899-jvrk9                  1/1     Running   0               6d
harbor-portal-79dd59d6fb-c6nhg                1/1     Running   0               6d
harbor-portal-79dd59d6fb-gxqdf                1/1     Running   0               6d
harbor-portal-79dd59d6fb-pvz4l                1/1     Running   0               6d
harbor-postgres-0                             1/1     Running   0               7d4h
harbor-postgres-1                             1/1     Running   0               7d4h
harbor-postgres-2                             1/1     Running   0               7d4h
harbor-registry-66c74767f9-9q6hz              2/2     Running   0               6d
harbor-registry-66c74767f9-bxbc9              2/2     Running   0               6d
harbor-registry-66c74767f9-g6xnl              2/2     Running   0               6d
harbor-trivy-scanner-harbor-scanner-trivy-0   1/1     Running   0               7d4h
harbor-trivy-scanner-harbor-scanner-trivy-1   1/1     Running   0               7d4h
harbor-trivy-scanner-harbor-scanner-trivy-2   1/1     Running   0               7d4h
harbor-trivy-scanner-harbor-scanner-trivy-3   1/1     Running   0               7d4h
harbor-trivy-scanner-harbor-scanner-trivy-4   1/1     Running   0               7d4h
redis-ha-haproxy-5db577f4-7kljq               1/1     Running   1 (7d7h ago)    182d
redis-ha-haproxy-5db577f4-gdv9z               1/1     Running   0               12d
redis-ha-haproxy-5db577f4-nkbhf               1/1     Running   30 (77d ago)    307d
redis-ha-server-0                             3/3     Running   1 (7d6h ago)    12d
redis-ha-server-1                             3/3     Running   108 (28d ago)   307d
redis-ha-server-2                             3/3     Running   33 (28d ago)    63d

Does anyone can teach me how to debug or remove trivy scanner in Harbor?

[Aslan-Liu]

I am still struggling with this issue. I found one thing. I think maybe too many records in my database which stores data on NFS storage is the cause. If Harbor try to clean all relatived records when removing trivy-scanner, it may cause time out due to database performance issue.

harbor=# select count(*) from vulnerability_record;
 417416

harbor=# select count(*) from report_vulnerability_record;
 3235781

Dose anyone knonw how to fix this issue?

[Vad1mo]

• can you share the logs?
• what entries do you have in the db
• Please upgrade Harbor to latest version

[stonezdj]

If it is a build-in trivy scanner, it can not be removed.

[Aslan-Liu]

Sorry for the late reply.

@Vad1mo
Actually, I don't know how to find useful logs. I used "kubectl logs -f" to redirect all logs from "harbor-core" into a file and send the delete api via Harbor UI at the same time. After getting 504 Gateway Time-out response, I tried to grep the log file with my scanner id. But I cannot see anything in it. Here are my commands

kubectl -n harbor logs -f harbor-core-8bc64f4c-v4465 > harbor.log
cat harbor.log | grep e38291db-d472-11ee-9fe7-92a278d09ce6     # (output nothing)

That's the reason why I stuck at here for many days. Did I do the wrong way for log collecting?
I know upgrading to the latest version maybe a way to check this issue, but this is in the production environment.
I cannot upgrade it as I want.

@stonezdj
No, this is not build-in trivy. I installed trivy scanner by using trivy helm charts.

[stonezdj]

I am still struggling with this issue. I found one thing. I think maybe too many records in my database which stores data on NFS storage is the cause. If Harbor try to clean all relatived records when removing trivy-scanner, it may cause time out due to database performance issue.

harbor=# select count(*) from vulnerability_record;
 417416

harbor=# select count(*) from report_vulnerability_record;
 3235781

Dose anyone knonw how to fix this issue?

When deleting trivy scanner, it will not delete record in these two tables.