bug: tests failing after NVD server update

[terriko]

There are currently 20 tests failing (details below). Since we're a few days after NVD has done their server move and updates, I strongly suspect that a data change has caused these and all these tests will need to be updated with new exptected data, or that our cache got somehow corrupted and will need to be refreshed. I haven't actually investigated them yet and will likely disable them temporarily while we debug.

=========================== short test summary info ============================
FAILED test/test_csv2cve.py::TestCSV2CVE::test_csv2cve_valid_file - AssertionError: assert ('cve_bin_tool', 20, 'There are 2 products with known CVEs detected') in [('cve_bin_tool', 20, 'CVE Binary Tool v3.3'), ('cve_bin_tool', 20, 'This product uses the NVD API but is not endorsed or certified by the NVD.'), ('cve_bin_tool.CVEDB', 20, 'Using cached CVE data (<24h old). Use -u now to update immediately.'), ('cve_bin_tool.CVEDB', 20, 'There are 17109 CVE entries in the database'), ('cve_bin_tool.CVEDB', 20, 'There are 17109 CVE entries from REDHAT in the database'), ('cve_bin_tool', 20, 'CVE database contains CVEs from National Vulnerability Database (NVD), Open Source Vulnerability Database (OSV), Gitlab Advisory Database (GAD) and RedHat'), ...]
 +  where [('cve_bin_tool', 20, 'CVE Binary Tool v3.3'), ('cve_bin_tool', 20, 'This product uses the NVD API but is not endorsed or certified by the NVD.'), ('cve_bin_tool.CVEDB', 20, 'Using cached CVE data (<24h old). Use -u now to update immediately.'), ('cve_bin_tool.CVEDB', 20, 'There are 17109 CVE entries in the database'), ('cve_bin_tool.CVEDB', 20, 'There are 17109 CVE entries from REDHAT in the database'), ('cve_bin_tool', 20, 'CVE database contains CVEs from National Vulnerability Database (NVD), Open Source Vulnerability Database (OSV), Gitlab Advisory Database (GAD) and RedHat'), ...] = <_pytest.logging.LogCaptureFixture object at 0x7ff4abb82c50>.record_tuples
FAILED test/test_exploits.py::TestExploitScanner::test_exploit_checker[True-exploits_list0-product_info0-triage_info0-CRITICAL-EXPLOIT] - IndexError: list index out of range
FAILED test/test_exploits.py::TestExploitScanner::test_exploit_checker[False-exploits_list1-product_info1-triage_info1-CRITICAL] - IndexError: list index out of range
FAILED test/test_exploits.py::TestExploitScanner::test_exploit_checker[True-exploits_list2-product_info2-triage_info2-CRITICAL] - IndexError: list index out of range
FAILED test/test_exploits.py::TestExploitScanner::test_exploit_checker[False-exploits_list3-product_info3-triage_info3-CRITICAL] - IndexError: list index out of range
FAILED test/test_language_scanner.py::TestLanguageScanner::test_java_package[/home/runner/work/cve-bin-tool/cve-bin-tool/test/language_data/pom.xml-product_list0] - UnboundLocalError: cannot access local variable 'product_info' where it is not associated with a value
FAILED test/test_language_scanner.py::TestLanguageScanner::test_python_package[/home/runner/work/cve-bin-tool/cve-bin-tool/test/language_data/PKG-INFO] - UnboundLocalError: cannot access local variable 'product_info' where it is not associated with a value
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx-spdx_parsed_data0] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='apache-jena', version='3.12.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.rdf-spdx_parsed_data1] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.json-spdx_parsed_data2] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='apache-jena', version='3.12.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.xml-spdx_parsed_data3] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='jena', version='3.12.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.yml-spdx_parsed_data4] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='apache-jena', version='3.12.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_test.spdx.yaml-spdx_parsed_data5] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='apache-jena', version='3.12.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_spdx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/spdx_mixed_test.spdx.json-spdx_parsed_data6] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='apache-jena', version='3.12.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='saxon', version='8.8'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_test.xml-cyclonedx_parsed_data0] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='acme application', version='9.1.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat-catalina', version='9.0.14'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_test.json-cyclonedx_parsed_data1] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='acme application', version='9.1.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat-catalina', version='9.0.14'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_test2.json-cyclonedx_parsed_data2] - AssertionError: assert ProductInfo(vendor='ubuntu', product='ubuntu', version='22.04') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='acme application', version='9.1.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat-catalina', version='9.0.14'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='ubuntu', version='22.04'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_cyclonedx_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/cyclonedx_mixed_test.json-cyclonedx_parsed_data3] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='acme application', version='9.1.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat-catalina', version='9.0.14'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='invalid_purl_package', version='1.1.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_sbom.py::TestSBOM::test_valid_swid_file[/home/runner/work/cve-bin-tool/cve-bin-tool/test/sbom/swid_test.xml-swid_parsed_data0] - AssertionError: assert ProductInfo(vendor='gnu', product='glibc', version='2.11.1') in defaultdict(<class 'dict'>, {ProductInfo(vendor='UNKNOWN', product='windows embedded standard 7', version='6.1.7601'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='windows embedded standard 7 with sp1 patches', version='3.0'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='sql 2005 express', version='9.00.5000.00,SP4'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='.net frame work', version='2.1.21022.8,SP2'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='glibc', version='2.11.1'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='java 8', version='1.8'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='tomcat 9', version='9.037'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}, ProductInfo(vendor='UNKNOWN', product='spring framework', version='4.7'): {'default': {'remarks': <Remarks.NewFound: 1>, 'comments': '', 'severity': ''}, 'paths': {''}}})
FAILED test/test_triage.py::test_triage - assert 0 >= 1
 +  where 0 = len([])
=========== 20 failed, 1905 passed, 26 skipped in 915.56s (0:15:15) ============

[joydeep049]

I would like to work on solving this.
How do i proceed?
@terriko

[mastersans]

I would like to work on solving this. How do i proceed? @terriko

I mainly delved into the 'test_sbom' area but didn't discover anything that should have caused an AssertionError in this instance. I also ran tests on the container randomly, and they passed without issues. I have a suspicion that there might be a problem with the cache, but refreshing the cache may solve it or give us further hints about whether it needs more investigation.

[terriko]

The cache refreshes every day, so it had refreshed many many times before I went ahead and disabled these tests: https://github.com/intel/cve-bin-tool/actions/workflows/update-cache.yml

So... I don't think that's the problem exactly, but it's possible that there was some data change that means all new updates of the cache have the same issue?

@crazytrain328 The first step here is for you to make a branch, choose one of these tests, enable it, and run the test:

• make sure to update your local cache if you haven't recently
• If the test fails at any step, debug it.
• If it doesn't fail on your local branch, try pushing your branch into github and running the tests there on your fork.
• If if doesn't fail on your fork, make a PR and run the tests again on there.

If it never fails any of those steps and you have a clean PR with all tests running, it's safe to re-enable that test and you can move on to the test one of the 20 tests I have temporarily disabled. If it fails at any step, use the error message to figure out what needs to change: it may be the test data, it may be something in our code. It may not be a recent change: some of these tests were unintentionally disabled for months.

[joydeep049]

In our script, test in the test_exploits.py file is currently disabled. I enabled it and ran it in my local machine and all tests passed.
However , the test_language_scanner failed when i ran it and gave mainly two errors.
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host mirror.cveb.in:443 ssl:default [None]
ConnectionAbortedError: SSL handshake is taking longer than 60.0 seconds: aborting the connection
In this error due my branch being not updated ?
@terriko

[terriko]

That error says you can't connect to mirror.cveb.in -- is it blocked on your end for some reason or might you have been having network errors? can you visit https://mirror.cveb.in/ in your web browser?

I can't remember if any of those tests can be run as if they're in offline mode easily, but you can probably run the test files through cve-bin-tool --offline [filename] and see what you're getting in them.

[joydeep049]

Hello , I came across something interesting while debugging the above error and testing the test_language_scanner file :) .

It shows that a non-existent file cve_bin_tool/checkers/expat is being imported.
Is this an error on my local repo?
@terriko

[mastersans]

Hello , I came across something interesting while debugging the above error and testing the test_language_scanner file :) .

It shows that a non-existent file cve_bin_tool/checkers/expat is being imported.
Is this an error on my local repo?
@terriko

This is unrelated to your local repo expat checker is present with different name libexpat so its unable to import it, probably a issue can be raised..

[joydeep049]

Hello , I came across something interesting while debugging the above error and testing the test_language_scanner file :) .

It shows that a non-existent file cve_bin_tool/checkers/expat is being imported.
Is this an error on my local repo?
@terriko

This is unrelated to your local repo expat checker is present with different name libexpat so its unable to import it, probably a issue can be raised..

Yes, I'll look for other problems in the tests then.

[terriko]

edit: nevermind, I figured out what was going wrong on my end.