NVD mirror preparation tasks

[terriko]

• related: breaking change: switching to use our NVD mirror by default #3153

I've got a few things I'd like in cve-bin-tool in conjunction with the NVD mirror work:

1. Minor directory structure changes for mirrors
   • As @warthog9 mentioned in today's meeting: the micro mirror project may eventually want to also mirror OSV, Redhat and others if licensing allows. I don't think we're actually too excited about doing that yet since NVD is the only service that has a history of not working when we need it, but future-proofing for it basically just means "make some directories" so let's do it.
   • Current setup is here: https://github.com/sec-data/mirror-sandbox but I propose:
     • /nvd/json - holds all NVD json files (currently in json_feed). These will be exact copies of the json feeds from https://nvd.nist.gov/vuln/data-feeds for now; we may eventually need to generate these from the API if they turn off the feeds as planned in September.
     • /cve-bin-tool - holds all our pre-parsed files. So we'd move cve_severity and the others into the cve-bin-tool directory.
     • Then if we ever set up mirrors for other data sources, we'd have /osv, /redhat etc.
2. Ability to use NVD JSON files from a mirror
   • currently we expect our pre-parsed files on a mirror, but we have the code to use the JSON from NVD, so it should be a matter of wiring these two abilities together.
   • Probably the workflow should go "if there's pre-parsed stuff, use that. If there's not, load the full JSON files and do the parsing. If neither of those exist, fail with a message about the mirror not looking right"
3. Make sure we're sending an appropriate User-Agent header with requests
   • Right now I imagine we're sending whatever python requests does by default, but it might be nice to send cve-bin-tool and a version number, which would allow us to gauge usage on the mirrors if we wanted.
   • It looks pretty easy to change this using requests https://docs.python-requests.org/en/latest/user/advanced/#request-and-response-objects
   • Our json code may be old enough that it's not using requests but rather urllib directly.
   • We should also figure out how to document this so people can have reasonable privacy expectations and understanding.
4. Provide a mirroring script for "grab the json and validate it" since we have the code for that but it's hard to trigger on its own.
   • Validation will mostly mean checking the metadata they provide and making sure file size/sha256 sum matches. Might have to watch for race conditions.
   • We can also provide the ability to validate the jsonschema, but we know from experience that it doesn't always validate, so we probably want to make sure this doesn't block anything.
   • @b31ngd3v may have this working on https://github.com/sec-data/mirror-sandbox/ already (I know the files are there, but I don't know how we're doing validation off the top of my head)

I think we're pretty close to turning on the replication once we firm up a directory structure.

I'm going to finish reading up on requests and user-agent setting and see if I can get a PR out shortly, but I need to do some PR review first and I may fall down a rathole of old code that doesn't use requests.

@b31ngd3v -- you've done most of the work on the mirrors, did you want to work on the directory changes or any of the rest of this?

[warthog9]

Ok I've made a couple of adjustments to how the mirroring script works on cveb.in effectively in preparation for this:

• Moved the existing data to /cve-bin-tool/ as suggested above
• /nvd/ now exists as well, this has both the json and xml directories and is effectively just a quick wget and rsync into place

You can see the results of this at https://mirror.cveb.in/ which is the not mirrorbits. That being said mirrorbits seems to have done DELIGHTFULLY exactly the right thing so things like https://cveb.in/get/nvd/json/cve/1.1/nvdcve-1.1-2002.json.gz works as expected.

I think this is starting to look like what we are wanting / expecting. Thoughts?

[warthog9]

Also note right now the user agent we are using in cve-bin-tool is "Python/3.10 aiohttp/3.8.5" so we likely need to set a custom one, which shouldn't be too bad as it likely looks mostly like:

headers = {
     'User-Agent': 'cve-bin-tool/<version> - see <url for explanation>'
}

and passing that headers into the request object

[terriko]

This means that if we set up for the 3.3 release with our default mirror url as
https://cveb.in/get/cve-bin-tool

then it should work right now? (Note that we're just specifically pointing at our stuff, not the full mirror. And also I haven't actually tried it yet.) After I run some tests we might want to merge that into main this week and have it be the default on the dev branch.

We probably still want to set things up so it can fail over to the nvd mirror or understand it if we give it https://cveb.in/get/nvd instead of the cve-bin-tool url. And I do still think we could stand to update our request headers before we do a release. But I think we're almost ready for this to be in the dev tree and start switching over our CI to use it.

[warthog9]

https://cveb.in/get/cve-bin-tool would be the base url you'd expect as a default config

[terriko]

Okay, first pass on a User-Agent PR is here: #3183

The current User-Agent string shows as

User-Agent: cve-bin-tool/3.2.2dev0 (https://github.com/intel/cve-bin-tool/)

I did a bit of reading and it seems like User-Agent can be anything, but name/version (url) is common enough that people wouldn't find it surprising. We could conceivably add stuff like os/python version data as well (also common in user-agent strings) but I'm going to err on the side of privacy since I don't think we need that data for anything.

Reviews appreciated.

Edit: I've now added the header to all the data_source requests so as well as the mirrors, nvd, redhat, osv, etc. will see this header. I haven't gone digging for other requests yet and I probably won't do that in this PR because I'd like to let all the tests pass and because I don't think anyone but our own mirrors particularly care at this point.

[terriko]

Okay, user-agent is merged, directory changes have been made on the mirror. I don't believe we've got any import for unmodified NVD files yet, but I don't think we need to wait on that.

@b31ngd3v -- Are we good to enable the cveb.in mirrors as our default NVD data source?

[b31ngd3v]

Downloading nvd json feed from cveb.in works but the --use-mirror flag with cveb.in doesn't work!
so I was thinking should we change the use mirror function to download nvd json feed instead of the current method which downloads exported cve.db json data?

[terriko]

As discussed today: current plan is to switch --use-mirror to work with the unparsed NVD json feed because there's a problem with the parsed one. Once that's working, we should be able to turn on the mirror as default. I'm particularly interested in having it be the default for our tests for the next little while so we can keep an eye on it before the planned 3.3. release (in September)

[terriko]

I think with #3265 merged we've got the mirrors enabled and this particular issue can be closed, but I'll point to two remaining issues that are still nice to have:

• It would be nice to have a script for generating "old" style JSON files from API2: see feat: Convert NVD API 2.0 to JSON files for mirroring #3199
• We never got around to number 4 above so I've filed it separately as feat: Add validation script for NVD JSON files #3275