[th/tft-check] add "--check" option to tft.py to check the test results
#201
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thomas Haller <[email protected]>
A plain sys.exit() in the middle of the code is not nice. Also, this may or may not happen on a thread, where this may or may not do the right thing. Just raise an exception. We catch all exceptions at the highest level and error out with a backtrace. We either should make an effort to handle the error (e.g. continue the test run and record an error in the test results), or we can just die with an exception. Do the latter. But don't do it via sys.exit() directly. Also, `if not r or not r.success:` is redundant. Signed-off-by: Thomas Haller <[email protected]>
sys.exit(-1) really means sys.exit(255), which looks a bit like exiting due to signal 127 (but no such signal exists). This is odd. Don't do this. Instead, exit on error with a regular exit code (0-127). Choose 70 (EX_SOFTWARE) and 78 (EX_CONFIG) from <sysexits.h> for this. EX_SOFTWARE is also what ktoolbox.common.run_main() does and indicates a bug-like situation (or something which we intentionally didn't handle and are fine to abort as if it were a bug). Signed-off-by: Thomas Haller <[email protected]>
ktoolbox.common.run_main() runs the main function and logs a traceback on any Exception. This is what we want for all our tools, because our tools are not supposed to let exceptions escape, and if they do, we want a traceback in the log. The helper does that. Use it. Note that this changes the error code in some cases, as run_main() exits with 70 (EX_SOFTWARE). That is an intentional change in behavior. Signed-off-by: Thomas Haller <[email protected]>
It is useful to track the filename along the result. Add a property for that, we will use it later. Signed-off-by: Thomas Haller <[email protected]>
We will call print_results code from "tft.py". Refactor in preparation for that. Signed-off-by: Thomas Haller <[email protected]>
By default (and previously always), "tft.py" runs the tests and writes the results to file. It usually does not fail (exit with non-zero exit code), unless there is a serious problem while running the test. In particular, it always returns with success, regardless whether the test had any failures. Usually, you will want to run "print_results.py" afterwards, which pretty-prints the results and exits with error if any tests failed. Add an option "--check", which let's "tft.py" behave as if you were running "print_results.py" afterwards. In this case, the exit code will indicate whether there were any failed tests. Signed-off-by: Thomas Haller <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
sys.exit()print_results.pyto make it usable fromtft.py.--checkoption totft.py. When set, the behavior is as if thetft.pycall was automatically followed by aprint_results.pyrun. Note that with this, we log the pretty-printed results at the end oftft.pyand exit with code 1, if any tests failed. Otherwise,tft.pycommand usually succeeds, regardless whether any tests failed.