GSoC 2021 Project idea: Improved triage & tracking

[terriko]

In the 2.0 release, we've introduced some basic triage ability so people can at least disable display of CVEs they feel have been mitigated in their systems.

There's probably more to do in the triage & tracking space, and I think we could probably make a gsoc project out of related ideas

• Add --append option to append output of previous scan rather than creating new one. #823 talks about appending scans
• Add --append option to append output of previous scan rather than creating new one. #823 also has a comment about combining reports / reusing triage across projects
• Some users may want to be able to use old scan data to track time-to-fix information (for their own projects or for what they're seeing upstream)

I'm NOT super excited about interactive command-line triage because I don't see any indications that our users actually want this, but we might be able to make it possible to flag things in the HTML reports and save data back to a json/csv file from the browser. That would be mostly html/javascript work and not python, though.

Anyhow, this thread is open for brainstorming on the topic. I'm not convinced this is a winner of a project yet, but it's the only thing currently flagged as "future" that seemed to be the right size for one.

[imsahil007]

Thinking about this way, users might want to differentiate between the scan-data of multiple teams. Maybe, we can add some other metadata along with these scans.
User will be able to distinguish the sources of severity (For e.g, Multiple teams working on a large project where they might have done triage separately as mentioned in #823 )

[Niraj-Kamdar]

An approach to mark triage through html would be to create localserver and aiohttp is more than enough for our purpose. By creating localserver we don't need to write that much javascript and we can make it more interactive easily.

[terriko]

Unfortunately, the problem with running a localserver isn't a technical one exactly. The problem is that Intel's security rules for all projects releasing under github.com/intel would require a lot more validation for any interface that included a full webserver, even a python localserver. (Our security checklist adapts to the type of project being released.)

Merging and displaying data from triage done via a spreadsheet or text editor doesn't incur nearly as much additional validation (and some of it we already did for the 1.0 and 2.0 releases) so it's a lot more attractive to me but still a useful upgrade.

If I did want to add a more complete triage interface using localhost, I'd have to convince my management to put more hours on it, and they'd most likely have to come out of the hours I've currently earmarked for mentoring gsoc students. I'd rather be able to take on another gsoc student than I would a full interactive triage interface, given the choice.

[terriko]

And in case anyone's curious, here's the publicly available description of some of the things we do for software security: https://newsroom.intel.com/wp-content/uploads/sites/11/2020/10/sdl-2020-whitepaper.pdf -- Most of that happens behind the scenes and is done by me and @pdxjohnny for each release because this is a pretty small project, but a larger project would have dedicated validation and so on.

[imsahil007]

We can save intermediate reports using existing output_json. Then we can do 2 things:

1. Either create another argument -m --merge which will take the filename of these JSON(s) as input and merge them together.
2. Create an HTML utility page through which we can merge these files using vanilla Javascript. The page will be standalone.
   (Or maybe both)
   De-duplicating through JS will be fairly simple.

//De-duplicate scan dictionary
Array.prototype.unique = function() {
    var scan = this.concat();
    for(var i=0; i< scan.length; ++i) {
        for(var j=i+1; j< scan.length; ++j) {
            if(scan[i] === scan[j])
                scan.splice(j--, 1);
        }
    }

    return a;
};

//var scan1 = json loaded from scan1.json
//var scan2 = json loaded from scan2.json
var result = scan1.concat(scan2).unique();
/* Repeat same for n scans*/

[terriko]

Finished as part of GSoC 2021