Dependency Check is slow

[MMKonstantin]

I configured my OWASP Dependency Check on Jenkins to use the latest version 12.1.0. Yesterday it took 20 hours to pull just 7% of the NVD database. Today I created API KEY and added it to the configuration. For 2 hours the progress is 32%. The first 30 minutes it pulled 32% and it got stucked to 32%. Do you have any suggestions? I have configured a folder and it will not download everything for every single run, but I need at least one time to download everything which seems not possible for right now. I am thinking to manually download the NVD database by ssh-ing on the Jenkins instance but I need some steps of doing that. I am not sure is it even posible.

Update: I started it once again, and this time it got stuck at 21% with no further progress for 2 hours and 12 minutes.

[AntoineLange]

I tried to run it 10 times yesterday without any success. It always got stuck before the end

[Mike3285]

Mine too, it gets stuck at some point and never completes the download.
It sometimes outputs this warning:
[WARN] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=36000 : 3rd time

[danshome]

I think this is a duplicate of #7418. What seems to be happening is the responses from NVD stop sending data in the middle of a response and since there is no read timeout set for the http5 client the default is 0, which means to wait indefinitely, hence the reason things hang and never progress once it gets in this state. We've had to diable the plugin for now because we haven't been able to complete a download since last week. @jeremylong

[aikebah]

As @danshome correctly signals this is a duplicate of #7418

[aikebah]

Be aware that you are a lot less likely to not run into these issues if you behave like a responsible consumer of the NVD API and don't re-initialize the CVE DB from scratch by crawling the NVD API on every build.

Cache your CVEDB on persistent storage connected to your CI environment so that subsequent builds only need to download the 'between 0 and a few K' new and updated CVEs since your last CI build instead of the full set of 280K+ entries on each and every build.
https://dependency-check.github.io/DependencyCheck/data/cacheh2.html

And/or perform the initial population of the CVE DB from your own local mirror of the NVD
https://dependency-check.github.io/DependencyCheck/data/mirrornvd.html

[MMKonstantin]

@aikebah I have caching configured and I am using NVD API KEY, but it is not able to finish the initial download. I just need NVD database to be pulled one time.

[aikebah]

In case you have someone that successfully initialized or updated their data in the recent past that you trust you would be able to get past the stall by taking a copy of their odc.mv.db as a kickstart into your cache. ODC would then only need to retrieve the new entries since their last successful update.

Other than that there's indeed not much of an option besides periodically trying until succeeding to download from an apparently again overloaded NVD API.

My gut feel is that still way too many people have their CI configured without using a shared persistent data-directory causing the intermittent overload on NVD API (unclear to me as an outsider to what extent the 'government efficiency' saga might be influencing available compute resources on the side of NVD API in present times)