Allow nominated tests to fail but still return a zero exit code #299
Description
Some systems are known to fail in specific ways. Having the conformance-test-harness fail every time due to this will hide the good work of all the things that are passing, including new things that turn up. Specifically, I would like to specify known failures so that if any new failures occur we can detect that. At the moment we'll simply fail in all cases and require human intervention to review the logs every time. It also means we can't gate releases based on the current exit code because some known failures will always gate the release.
Although I would like all systems to pass all tests, I accept that this is not always the case.