engine: map more unknown_failure to a specified failure string

[hellais]

While doing some data analysis work I enumerated all the unknown failures found in the OONI dataset since Jan 1st 2022 and produced this list: https://gist.github.com/hellais/8b38d360c169e5a2a8ce8856bcf519ff.

I think it would be very useful to have some specific error strings defined for some of these so that we can use them as part of the analysis more easily.

I think the TLS related ones are especially interesting.

Several of these have been consolidated into a single failure string and replaced the variable part with [token] to make it easier to look at.

Here is the snipped used for doing that mapping:

import re

lookup_re = re.compile(r'(lookup [a-zA-Z0-9\.\-]*)')

def consolidate_failure(f):
    if f.startswith("unknown_failure: net/http: HTTP/1.x transport connection broken: malformed HTTP response"):
        return "unknown_failure: net/http: HTTP/1.x transport connection broken: malformed HTTP response [garbage]"
    if f.startswith("unknown_failure: net/http: HTTP/1.x transport connection broken: malformed HTTP status code"):
        return "unknown_failure: net/http: HTTP/1.x transport connection broken: malformed HTTP status code [garbage]"
    if f.startswith("unknown_failure: tls: oversized record received with length"):
        return "unknown_failure: tls: oversized record received with length [length]"
    if f.startswith("unknown_failure: net/http: HTTP/1.x transport connection broken: malformed MIME header: missing colon:"):
        return "unknown_failure: net/http: HTTP/1.x transport connection broken: malformed MIME header: missing colon: [garbage]"
    if f.startswith("unknown_failure: x509: certificate is valid for "):
        return "unknown_failure: x509: certificate is valid for [domain_1], [domain_2] not [domain_3]"
    return lookup_re.sub('lookup [domain_name]', f)

This list is a superset of what has been reported in these issues: #2412 #2411 #2410

[ohnorobo]

CensoredPlanet also had to do a similar analysis. The exact mapping will depend on the details of the networking library/stack used (in CensoredPlanet's case it's https://pkg.go.dev/net), but you may find the correspondences we ended up with useful, especially for the TLS/HTTP specific ones (some are specific to CensoredPlanet's internal infrastructure and you should ignore them):

• Algorithm: https://github.com/censoredplanet/censoredplanet-analysis/blob/master/pipeline/metadata/hyperquack_outcome.py
• Enumerated errors: https://github.com/censoredplanet/censoredplanet-analysis/blob/master/pipeline/metadata/test_hyperquack_outcome.py
• Documentation for the strings: https://github.com/censoredplanet/censoredplanet-analysis/blob/master/docs/outcome.md

[hellais]

We now have mappings for this inside of the data pipeline. I think doing this in the probe is probably not worth the effort, so I am closing this issue. I will link it though from the data repo since it contains very useful links from the censored planet pipeline (thanks @ohnorobo !)