Merge branch 'intel:main' into master

Author: anthonyharrison

.github/actions/spelling/allow.txt

@@ -14,6 +14,7 @@ apk
 Args
 asn
 asottile
+atlassian
 autoescape
 autoextract
 autoextracts
@@ -233,6 +234,7 @@ nitishsaini
 nlk
 noopener
 noreferrer
+nosec
 nowdailynever
 nplurals
 ntp
@@ -286,6 +288,7 @@ Rahul
 readme
 readthedocs
 realpython
+rebasing
 refactoring
 regex
 Romi
@@ -339,6 +342,7 @@ unicode
 uniq
 unittest
 url
+urlopen
 usecase
 username
 usr

.github/workflows/pythonapp.yml

@@ -17,7 +17,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        tool: ['isort', 'black', 'pyupgrade', 'flake8', 'format_checkers']
+        tool: ['isort', 'black', 'pyupgrade', 'flake8', 'format_checkers', 'bandit']
     steps:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2

.pre-commit-config.yaml

@@ -14,12 +14,18 @@ repos:
     hooks:
     - id: pyupgrade
       args: ["--py37-plus"]
-      
+
 -   repo: https://github.com/pycqa/flake8
     rev: 4.0.1
     hooks:
     - id: flake8
 
+-   repo: https://github.com/PyCQA/bandit
+    rev: 1.7.1
+    hooks:
+    - id: bandit
+      args: ["-c", "bandit.conf"]
+
 -   repo: local
     hooks:
     - id: format_checkers

CONTRIBUTING.md

@@ -17,6 +17,7 @@ If you've already contributed to other open source projects, contributing to the
     - [Using pre-commit to run linters automatically](#using-pre-commit-to-run-linters-automatically)
     - [Running isort by itself](#running-isort-by-itself)
     - [Running black by itself](#running-black-by-itself)
+    - [Running bandit by itself](#running-bandit-by-itself)
     - [Other linting tools](#other-linting-tools)
   - [Making a new branch & pull request](#making-a-new-branch--pull-request)
     - [Commit message tips](#commit-message-tips)
@@ -150,7 +151,7 @@ The CVE Binary Tool has a set of tests that can be run using `pytest` command.
 
 We have done our best to make tests stable and ensure that they pass at all times, but occasionally tests may fail due to factors outside your control (common causes: internet connectivity, rate limiting by NVD or new vulnerability data changing our test expectations). If a test doesn't pass, you should look at it to see if any changes you made caused the failure.  If you're not sure, submit your code as a pull request and mention the issue and someone will try to help you sort it out.
 
-When you submit your code as a pull request, the whole test suite will be run on windows and linux using the versions of python we support, including longer tests. We don't expect you to do all that yourself; usually trying for one version of python on whatever local OS you have is good enough and you can let Github Actions do the rest!
+When you submit your code as a pull request, the whole test suite will be run on windows and linux using the versions of python we support, including longer tests. We don't expect you to do all that yourself; usually trying for one version of python on whatever local OS you have is good enough and you can let GitHub Actions do the rest!
 
 ## Running linters
 
@@ -160,6 +161,7 @@ CVE Binary Tool uses a few tools to improve code quality and readability:
 - `black` provides automatic style formatting.  This will give you basic [PEP8](https://www.python.org/dev/peps/pep-0008/) compliance. (PEP8 is where the default python style guide is defined.)
 - `flake8` provides additional code "linting" for more complex errors like unused imports.
 - `pyupgrade` helps us be forward compatible with new versions of python.
+- `bandit` is more of a static analysis tool than a linter and helps us find potential security flaws in the code.
 
 We provide a `dev-requirements.txt` file which includes all the precise versions of tools as they'll be used in GitHub Actions.  You an install them all using pip:
 
@@ -206,6 +208,26 @@ files you've changed because you won't have to scroll through a pile of
 auto-formatting changes to find your own modifications.  However, you can also
 specify a whole folder using ```./```
 
+### Running bandit by itself
+
+We have a configuration file for bandit called `bandit.conf` that you should use.  This disables a few of the checkers and disables scanning of the test directory.
+
+To run it on all the code we scan, use the following:
+
+```bash
+bandit -c bandit.conf -r cve_bin_tool/
+```
+
+You can also run it on individual files:
+
+```bash
+bandit -c bandit.conf filename.py
+```
+
+If you run it without the config file, it will run a few extra checkers and will run on test code, so you'll get additional warnings.
+
+Bandit helps you target manual code review, but bandit issues aren't always things that need to be fixed, just reviewed.  If you have a bandit finding that doesn't actually need a fix, you can mark it as reviewed using a `# nosec` comment.  If possible, include details as to why the bandit results are ok for future reviewers.  For example, we have comments like `#nosec uses static https url above` in cases where bandit prompted us to review the variable being passed to urlopen().
+
 ### Other linting tools
 
 As well as `black` for automatically making sure code adheres to the style guide, we use `flake8` to help us find things like unused imports.  The [flake8 documentation](https://flake8.pycqa.org/en/latest/user/index.html) covers what you need to know about running it.
@@ -230,6 +252,13 @@ git pull
 git checkout -b my_new_branch
 ```
 
+>Note: If you accidentally check something in to main and want to reset it to match our main branch, you can save your work using `checkout -b` and then do a `git reset` to fix it:
+>```
+>git checkout -b saved_branch
+>git reset --hard origin/main
+>```
+>You do not need to do the `checkout` step if you don't want to save the changes you made.
+
 When you're ready to share that branch to make a pull request, make sure you've checked in all the files you're working on.  You can get a list of the files you modified using `git status` and see what modifications you made using `git diff`
 
 Use `git add FILENAME` to add the files you want to put in your pull request, and use `git commit` to check them in.  Try to use [a clear commit message](https://chris.beams.io/posts/git-commit/) and use the [Conventional Commits](https://www.conventionalcommits.org/) format.  
@@ -266,7 +295,7 @@ Here's a quick checklist to help you make sure your pull request is ready to go:
 4. Have I added or updated any documentation if I changed or added a feature?
    - New features are often documented in [MANUAL.md](https://github.com/intel/cve-bin-tool/blob/main/doc/MANUAL.md).  (See [Making documentation](#making-documentation) for more information.)
 5. Have I used [Conventional Commits](https://www.conventionalcommits.org/) to format the title of my pull request?
-6. If I closed a bug, have I linked it using one of [Github's keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)? (e.g. include the text `fixed #1234`)
+6. If I closed a bug, have I linked it using one of [GitHub's keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)? (e.g. include the text `fixed #1234`)
 7. Have I checked on the results from GitHub Actions?
    - GitHub Actions will run all the tests, linters and a spell check for you.  If you can, try to make sure everything is running cleanly with no errors before leaving it for a human code reviewer!
    - As of this writing, tests take less than 20 minutes to run once they start, but they can be queued for a while before they start.  Go get a cup of tea or work on something else while you wait!
@@ -279,6 +308,8 @@ Someone will review your code and try to provide feedback in the comments on Git
 
 If something needs fixing or we have questions, we'll work back and forth with you to get that sorted.  We usually do most of the chatting directly in the pull request comments on GitHub, but if you're stuck you can also stop by our [Gitter chat room](https://gitter.im/cve-bin-tool/community) to talk with folk outside of the bug.
 
+>Another useful tool is `git rebase`, which allows you to change the "base" that your code uses.  We most often use it as `git rebase origin/main` which can be useful if a change in the main tree is affecting your code's ability to merge.  Rebasing is a bit much for an intro document, but [there's a git rebase tutorial here](https://www.atlassian.com/git/tutorials/rewriting-history/git-rebase) that you may find useful if it comes up.
+
 Once any issues are resolved, we'll merge your code.  Yay!
 
 In rare cases, the code won't work for us and we'll let you know.  Sometimes this happens because someone else has already submitted a fix for the same bug, (Issues marked [good first issue](https://github.com/intel/cve-bin-tool/labels/good%20first%20issue) can be in high demand!) or because you worked on a checker that didn't have a good signature. Don't worry, these things happens, no one thinks less of you for trying!

README.md

@@ -24,14 +24,14 @@ For more details, see our [documentation](https://cve-bin-tool.readthedocs.io/en
 - [CVE Binary Tool quick start / README](#cve-binary-tool-quick-start--readme)
   - [Installing CVE Binary Tool](#installing-cve-binary-tool)
   - [Most popular usage options](#most-popular-usage-options)
-    - [Using the tool offline](#using-the-tool-offline)
+  - [Using the tool offline](#using-the-tool-offline)
     - [Finding known vulnerabilities using the binary scanner](#finding-known-vulnerabilities-using-the-binary-scanner)
     - [Finding known vulnerabilities in a list of components](#finding-known-vulnerabilities-in-a-list-of-components)
     - [Scanning an SBOM file for known vulnerabilities](#scanning-an-sbom-file-for-known-vulnerabilities)
-    - [Output Options](#output-options)
+  - [Output Options](#output-options)
   - [Full option list](#full-option-list)
   - [Configuration](#configuration)
-  - [Using CVE Binary Tool in Github Actions](#using-cve-binary-tool-in-github-actions)
+  - [Using CVE Binary Tool in GitHub Actions](#using-cve-binary-tool-in-github-actions)
   - [Binary checker list](#binary-checker-list)
   - [Limitations](#limitations)
   - [Requirements](#requirements)
@@ -106,6 +106,8 @@ Usage:
                             choose method for getting CVE lists from NVD
       -u {now,daily,never,latest}, --update {now,daily,never,latest}
                             update schedule for NVD database (default: daily)
+      --nvd-api-key NVD_API_KEY
+                        specify NVD API key (used to improve NVD rate limit)
 
     Input:
       directory             directory to scan
@@ -178,7 +180,7 @@ in the terminal and provide it as an input by running `cve-bin-tool -L pkg-list`
 You can use `--config` option to provide configuration file for the tool. You can still override options specified in config file with command line arguments. See our sample config files in the
 [test/config](https://github.com/intel/cve-bin-tool/blob/main/test/config/)
 
-## Using CVE Binary Tool in Github Actions
+## Using CVE Binary Tool in GitHub Actions
 
 If you want to integrate cve-bin-tool as a part of your github action pipeline.
 You can checkout our example [github action](https://github.com/intel/cve-bin-tool/blob/main/doc/how_to_guides/cve_scanner_gh_action.yml).

bandit.conf

@@ -85,10 +85,19 @@
 tests:
 
 # (optional) list skipped test IDs here, eg '[B101, B406]':
-skips: ['B603', 'B607', 'B404']
+skips: ['B603', 'B607', 'B404', "B608"]
+# B603, B607 and B404 are all subprocess-related.
+# B608 should be re-enabled when multi-line issues can be marked with nosec
+
 # Explantion: cve-bin-tool is at heart a shell script that calls other processes.
 # Switching to pure python has significant performance impacts.
 
+exclude_dirs:
+  - "test/"
+  - "/test/"
+  - "./test/"
+  - "./build/lib/test/"
+
 ### (optional) plugin settings - some test plugins require configuration data
 ### that may be given here, per-plugin. All bandit test plugins have a built in
 ### set of sensible defaults and these will be used if no configuration is

cve_bin_tool/available_fix/debian_cve_tracker.py

@@ -44,33 +44,36 @@ def cve_info(
         """Produces the Backported fixes' info"""
 
         cve_data = format_output(all_cve_data)
-        check_json()
-        with open(DEB_CVE_JSON_PATH) as jsonfile:
-            json_data = load(jsonfile)
-            for cve in cve_data:
-                try:
-                    cve_fix = json_data[cve["product"]][cve["cve_number"]]["releases"][
-                        self.compute_distro()
-                    ]
-                    if cve_fix["status"] == "resolved":
-                        if self.is_backport:
-                            if cve_fix["fixed_version"].startswith(cve["version"]):
-                                LOGGER.info(
-                                    f'{cve["product"]}: {cve["cve_number"]} has backported fix in v{cve_fix["fixed_version"]} release.'
-                                )
-                            else:
-                                LOGGER.info(
-                                    f'{cve["product"]}: No known backported fix for {cve["cve_number"]}.'
-                                )
+        json_data = self.get_data()
+        for cve in cve_data:
+            try:
+                cve_fix = json_data[cve["product"]][cve["cve_number"]]["releases"][
+                    self.compute_distro()
+                ]
+                if cve_fix["status"] == "resolved":
+                    if self.is_backport:
+                        if cve_fix["fixed_version"].startswith(cve["version"]):
+                            LOGGER.info(
+                                f'{cve["product"]}: {cve["cve_number"]} has backported fix in v{cve_fix["fixed_version"]} release.'
+                            )
                         else:
                             LOGGER.info(
-                                f'{cve["product"]}: {cve["cve_number"]} has available fix in v{cve_fix["fixed_version"]} release.'
+                                f'{cve["product"]}: No known backported fix for {cve["cve_number"]}.'
                             )
-                except KeyError:
-                    if cve["cve_number"] != "UNKNOWN":
+                    else:
                         LOGGER.info(
-                            f'{cve["product"]}: No known fix for {cve["cve_number"]}.'
+                            f'{cve["product"]}: {cve["cve_number"]} has available fix in v{cve_fix["fixed_version"]} release.'
                         )
+            except KeyError:
+                if cve["cve_number"] != "UNKNOWN":
+                    LOGGER.info(
+                        f'{cve["product"]}: No known fix for {cve["cve_number"]}.'
+                    )
+
+    def get_data(self):
+        check_json()
+        with open(DEB_CVE_JSON_PATH) as jsonfile:
+            return load(jsonfile)
 
     def compute_distro(self):
         if self.distro_name == "ubuntu":

cve_bin_tool/available_fix/redhat_cve_tracker.py

@@ -74,8 +74,8 @@ def cve_info(
 
     def get_data(self, cve_number: str, product: str):
         try:
-            full_query = f"{RH_CVE_API}/{cve_number}.json"
-            response = request.urlopen(full_query).read().decode("utf-8")
+            full_query = f"{RH_CVE_API}/{cve_number}.json"  # static https url above
+            response = request.urlopen(full_query).read().decode("utf-8")  # nosec
             return loads(response)
         except error.HTTPError as e:
             LOGGER.debug(e)

cve_bin_tool/checkers/xml2.py

@@ -60,7 +60,8 @@ def guess_xml2_version(lines):
                 new_guess2 = match.group(1).strip()
                 if len(new_guess2) > len(new_guess):
                     new_guess = new_guess2
-        return new_guess
+        # If no version guessed, set version to UNKNOWN
+        return new_guess or "UNKNOWN"
 
     def get_version(self, lines, filename):
         version_info = super().get_version(lines, filename)

cve_bin_tool/cli.py

@@ -101,6 +101,12 @@ def main(argv=None):
         choices=["now", "daily", "never", "latest"],
         help="update schedule for NVD database (default: daily)",
     )
+    nvd_database_group.add_argument(
+        "--nvd-api-key",
+        action="store",
+        default="",
+        help="specify NVD API key (used to improve NVD rate limit)",
+    )
 
     input_group = parser.add_argument_group("Input")
     input_group.add_argument(
@@ -305,6 +311,7 @@ def main(argv=None):
         "backport_fix": "",
         "available_fix": "",
         "nvd": "api",
+        "nvd_api_key": "",
         "filter": [],
         "affected_versions": 0,
         "sbom": "spdx",
@@ -337,6 +344,23 @@ def main(argv=None):
     else:
         error_mode = ErrorMode.TruncTrace
 
+    # once logging is set, we can output the version and NVD notice
+    LOGGER.info(f"CVE Binary Tool v{VERSION}")
+    LOGGER.info(
+        "This product uses the NVD API but is not endorsed or certified by the NVD."
+    )
+
+    # If NVD API key is not set, check for environment variable (e.g. GitHub Secrets)
+    if not args["nvd_api_key"] and os.getenv("nvd_api_key"):
+        args["nvd_api_key"] = os.getenv("nvd_api_key")
+
+    # If you're not using an NVD key, let you know how to get one
+    if not args["nvd_api_key"] and not args["offline"]:
+        LOGGER.info("Not using an NVD API key. Your access may be rate limited by NVD.")
+        LOGGER.info(
+            "Get an NVD API key here: https://nvd.nist.gov/developers/request-an-api-key"
+        )
+
     if platform.system() != "Linux":
         warning_nolinux = """
                           **********************************************
@@ -400,6 +424,7 @@ def main(argv=None):
         error_mode=error_mode,
         nvd_type=args["nvd"],
         incremental_update=True if db_update == "latest" and args["nvd"] else False,
+        nvd_api_key=args["nvd_api_key"],
     )
 
     # if OLD_CACHE_DIR (from cvedb.py) exists, print warning

cve_bin_tool/cvedb.py

@@ -32,7 +32,7 @@
 )
 from cve_bin_tool.log import LOGGER
 from cve_bin_tool.nvd_api import NVD_API
-from cve_bin_tool.version import VERSION, check_latest_version
+from cve_bin_tool.version import check_latest_version
 
 logging.basicConfig(level=logging.DEBUG)
 
@@ -71,6 +71,7 @@ def __init__(
         error_mode=ErrorMode.TruncTrace,
         nvd_type="json",
         incremental_update=False,
+        nvd_api_key: str = "",
     ):
         self.feed = feed if feed is not None else self.FEED
         self.cachedir = cachedir if cachedir is not None else self.CACHEDIR
@@ -93,6 +94,9 @@ def __init__(
         self.incremental_update = incremental_update
         self.all_cve_entries = None
 
+        # store the nvd api key for use later
+        self.nvd_api_key = nvd_api_key
+
         if not os.path.exists(self.dbpath):
             self.rollback_cache_backup()
 
@@ -108,6 +112,7 @@ async def nist_fetch_using_api(self):
             logger=self.LOGGER,
             error_mode=self.error_mode,
             incremental_update=self.incremental_update,
+            api_key=self.nvd_api_key,
         )
         if self.incremental_update:
             await nvd_api.get_nvd_params(
@@ -256,7 +261,7 @@ async def refresh(self):
         # refresh the database
         if not os.path.isdir(self.cachedir):
             os.makedirs(self.cachedir)
-        self.LOGGER.info(f"CVE Binary Tool v{VERSION}")
+
         # check for the latest version
         if self.version_check:
             check_latest_version()