Quiet mode broken in main branch?

[terriko]

___________________________ TestCLI.test_quiet_mode ____________________________

self = <test.test_cli.TestCLI object at 0x7fa413663d00>
capsys = <_pytest.capture.CaptureFixture object at 0x7fa4111e6f10>
caplog = <_pytest.logging.LogCaptureFixture object at 0x7fa41114ce50>

    def test_quiet_mode(self, capsys, caplog):
        """Test that an quite mode isn't generating any output"""
    
        with tempfile.NamedTemporaryFile(
            "w+b", suffix="strong-swan-4.6.3.out", delete=False
        ) as f:
            signatures = [b"\x7f\x45\x4c\x46\x02\x01\x01\x03\n", b"strongSwan 4.6.3"]
            f.writelines(signatures)
            filename = f.name
    
        main(["cve-bin-tool", "-q", filename, "-u", "now"])
        # clean up temporary file.
        os.remove(filename)
    
        # Make sure log is empty
        assert not caplog.records
    
        # Make sure nothing is getting printed on stdout or stderr
        captured = capsys.readouterr()
>       assert not (captured.out or captured.err)
E       AssertionError: assert not ('Updating CVEs from NVD...                                            0% -:--:--\n')
E        +  where 'Updating CVEs from NVD...                                            0% -:--:--\n' = CaptureResult(out='Updating CVEs from NVD...                                            0% -:--:--\n', err='').out

test/test_cli.py:303: AssertionError
=============================== warnings summary ===============================

Looks like something is ignoring the quiet mode flag. Probably an easy fix if you grep for "Updating CVEs from NVD" and make sure it respects the quiet flag, but I haven't looked at it deeply myself.

[BenL-github]

Hi, I am new to open source and tried to look into this issue. I ran pytest test/test_cli.py::TestCLI::test_quiet_mode but could not reproduce this result (the test passed). Has this issue been fixed or am I running the wrong test?

[terriko]

This one has been transient. I haven't been able to duplicate it locally either, although you might be able to get the effect if you somehow disconnected from the internet partway through a test run?

If you look through recent tests on main you can see it happened on a few pull requests:
https://github.com/intel/cve-bin-tool/commits/main

You'll need look for the red x marks and click those to see the GitHub Actions reports where the tests failed.

When we get transient errors, it's usually due to networking issues in the cloud instances where GitHub Actions runs, so this may be more of the same. (Ideally, quiet mode should be reporting an error code that indicates some sort of network error but not providing output in that case.)

What you should be looking for is the "Updating CVEs" line and seeing if there is any situation where it would print when quiet mode is enabled.