|
1 | | -# Deepcov |
2 | | - |
3 | | -[](https://pypi.org/project/deepcov) |
4 | | -[](https://pypi.org/project/deepcov) [](https://twitter.com/treebeardtech) |
5 | | - |
6 | | -**<div style="font-size:21px">🚧 under construction, do not attempt to use 🚧</div>** |
7 | | - |
8 | | -The goal of this project is to make it easier to identify the root cause of pytest failures. |
9 | | - |
10 | | -Deepcov lets you view coverage data for each individual test case in VSCode to identify which (if any) changes in your code caused failures. |
| 1 | +<p align="center"> |
| 2 | + <img width="150" src="docs/icon.png" style="border: 1px solid rgba(0,0,0,0.3)" /> |
| 3 | +</p> |
| 4 | +<p align="center">See test results for every line of Python code |
11 | 5 |
|
12 | | -## The Problem |
| 6 | +</p> |
| 7 | +<p align="center"> |
| 8 | + <a href="https://pypi.org/project/pytest-deepcov"><img src="https://img.shields.io/pypi/pyversions/pytest-deepcov?logo=python&logoColor=white" /></a> |
| 9 | + <a href="https://pypi.org/project/pytest-deepcov"><img src="https://img.shields.io/pypi/v/pytest-deepcov?logo=python&logoColor=white" /></a> |
| 10 | + <a href="https://twitter.com/treebeardtech"><img src="https://img.shields.io/static/v1?label=twitter&message=follow&color=blue&logo=twitter" /></a> |
| 11 | +</p> |
13 | 12 |
|
14 | | -You are working on a bug fix. Main branch is clean according to CI, so you create a branch then put in a few source changes. All config and dependencies remain unchanged. |
| 13 | +**Deepcov is a pytest plugin + VSCode extension which combines test status and code coverage data.** |
15 | 14 |
|
16 | | -Let's run some tests before opening a PR: |
17 | 15 |
|
18 | | -```log |
19 | | -FAILED workflow_handler/tests/test_model_r13n.py::TestR13n::test_when_data_not_fetched_then_no_regional_update |
20 | | -FAILED workflow_handler/tests/test_model_r13n.py::TestR13n::test_connect |
21 | | -FAILED workflow_handler/tests/test_model_r13n.py::TestR13n::test_when_data_fetched_then_update |
22 | | -====== 3 failed, 510 passed, 2 xfailed, 51 warnings in 140.83s (0:02:20) ======= |
23 | | -``` |
| 16 | +<p align="center"> |
| 17 | + <img width="550" src="docs/screen.png" style="border: 1px solid rgba(0,0,0,0.3)" /> |
| 18 | +</p> |
24 | 19 |
|
25 | | -These tests look unfamiliar. They may be new, and I don't think they are related to my change. The error logs are hard to decipher. |
| 20 | +## Making Sense of Test Runs |
26 | 21 |
|
27 | | -What could be wrong? |
| 22 | +Tests can often fail due to unreliable infrastructure, such as network dependencies. When this happens, it can be difficult figuring out if we should let it distract us from what we're working on. |
28 | 23 |
|
29 | | -1. False positive: The tests are not configured for my dev environment |
30 | | -2. False positive: The tests are flaky, due to a network dependency/race condition |
31 | | -3. True positive: My code has somehow modified these testcases |
32 | | -4. Some combination of the above |
| 24 | +What if we just wanted to pay attention to tests running through components we have changed? |
33 | 25 |
|
34 | | -There are various courses of action I can take to make sense of this. Re-running the suite, debugging individual tests or opening a PR to run CI can help, but are time consuming. |
| 26 | +**Pytest** can tell us if tests pass, but can't tell us if we changed their behaviour, whereas **coverage** data can tell us if tests are running through a component, but not if they are passing. |
35 | 27 |
|
36 | | -Deepcov proposes a way of identifying if your code change has impacted a test without re-running failures. |
37 | 28 |
|
38 | | -## The Solution |
| 29 | +| | pytest | pytest-cov | pytest-deepcov | |
| 30 | +|-----------------------------|--------|------------|----------------| |
| 31 | +| indicates code health | <span style="color: #00CE1C">✔</span> | <span style="color: #DF0E25">✖</span> | <span style="color: #00CE1C">✔</span> | |
| 32 | +| data for every line of code | <span style="color: #DF0E25">✖</span> | <span style="color: #00CE1C">✔</span> | <span style="color: #00CE1C">✔</span> | |
39 | 33 |
|
40 | | -Deepcov shows lines of code that were run by failing tests, so you can connect test failures with source changes. |
| 34 | +## Tutorial |
41 | 35 |
|
42 | | -### Pre-req: Capture test coverage and results when you run pytest |
| 36 | +We are going to debug a toy test suite using deepcov. |
43 | 37 |
|
44 | | -```zsh |
45 | | -pip install -U pytest pytest-cov |
| 38 | +<p align="center"> |
| 39 | + <img width="550" src="docs/demo.gif" style="border: 1px solid rgba(0,0,0,0.3)" /> |
| 40 | +</p> |
46 | 41 |
|
47 | | -pytest \ |
48 | | - --cov=src \ # Collect coverage data for your repo |
49 | | - --cov-context=test \ # Ensure coverage data is segmented per-test |
50 | | - --junit-xml=results.xml # Output test results |
| 42 | +Before starting, ensure you have python and VSCode installed. |
| 43 | +### Run tests with deepcov attached |
| 44 | +1. Fork and clone this repo |
| 45 | +1. `code pytest-deepcov/python-cli/tests/resources` |
| 46 | +1. Open `src/lib.py` and `src/test_lib.py` |
| 47 | +1. Create yourself a new terminal window in VSCode |
| 48 | +1. (optional) `virtualenv .venv; . .venv/bin/activate` |
| 49 | +1. `pip install pytest-deepcov` |
| 50 | +1. `pytest --cov=src` |
| 51 | +1. the tests should fail, and a new directory `.deepcov` should be created |
| 52 | + |
| 53 | +### Visualise deepcov data |
| 54 | +1. Install the [VSCode extension](https://marketplace.visualstudio.com/items?itemName=treebeardtech.deepcov). It will add a 'deepcov' button to the top right of your editor. Don't click it yet :) |
| 55 | +1. Configure your CLI location: Open VSCode settings, then search for `deepcov`. Follow the instruction to connect with the python package. |
| 56 | +1. Enable deepcov using the 'deepcov' button to view your pytest test results from above. |
| 57 | +1. `src/lib.py` and `src/test_lib.py` should have annotations on every line with test results. |
| 58 | +1. Disable deepcov using the same button |
| 59 | + |
| 60 | +### Troubleshooting |
| 61 | + |
| 62 | +1. If it complains that it can't find the CLI, you will need to (a) ensure the CLI is correct, (b) close and reopen VSCode to refresh the config |
| 63 | +1. Are you on Windows? If so, please try windows subsystem for linux. |
| 64 | +1. Please log an issue on this repo and I will help |
| 65 | +1. If you are feeling brave, try using the CLI to debug (see below) |
| 66 | +## Deepcov Internals |
| 67 | + |
| 68 | +The VSCode extension fetches data for each line of code from the deepcov python package. |
| 69 | + |
| 70 | +`deepcov /Users/a/git/treebeardtech/deepcov/python-cli/tests/resources/src/lib.py` |
| 71 | +```json |
| 72 | +{ |
| 73 | + "lines": { |
| 74 | + "3": { |
| 75 | + "passed": [], |
| 76 | + "failed": [] |
| 77 | + }, |
| 78 | + "1": { |
| 79 | + "passed": [ |
| 80 | + "ran on startup" |
| 81 | + ], |
| 82 | + "failed": [] |
| 83 | + }, |
| 84 | +... |
| 85 | + "5": { |
| 86 | + "passed": [ |
| 87 | + "src.test_lib.TestLib.test_divide2[3.0]", |
| 88 | + "src.test_lib.test_divide", |
| 89 | + "src.test_lib.test_divide2[3.0]" |
| 90 | + ], |
| 91 | + "failed": [ |
| 92 | + "src.test_lib.TestLib.test_divide2[0]", |
| 93 | + "src.test_lib.test_divide2[0]" |
| 94 | + ] |
| 95 | + } |
| 96 | + } |
| 97 | +} |
51 | 98 | ``` |
52 | 99 |
|
53 | | -### View the code ran by failing tests |
54 | | - |
55 | | -When coverage and test report data is captured, you can view test statuses for each line of code. |
| 100 | +(Note: this cli must be run inside the `.deepcov` output directory) |
56 | 101 |
|
57 | | -<p align="center"> |
58 | | - <img width="750" src="docs/editor.png" /> |
59 | | -</p> |
| 102 | +You can use this approach to create plugins for other text editors. |
60 | 103 |
|
61 | | -## See also |
| 104 | +## Limitations |
| 105 | +* Project structure: Your pytest root directory must be the same as your vscode root directory. |
| 106 | +* `coveragepy` is used to capture coverage data using the Python trace hook. This means: |
| 107 | + 1. You cannot use the debugger and deepcov at the same time. Deepcov detects when the debugger is active and disables itself. |
| 108 | + 1. Everything takes 30% longer. It is best to only use deepcov to rerun subsets of a failing test suite. |
| 109 | +## Diving Deeper |
62 | 110 |
|
63 | | -- https://arxiv.org/abs/1607.04347 |
64 | | -- https://github.com/saeg/jaguar |
65 | | -- https://arxiv.org/pdf/1607.04347.pdf |
| 111 | +Having test results for every line of code opens up the possibility of implementing *spectrum-based fault localisation* |
| 112 | +- https://arxiv.org/abs/1607.04347 contains a description of this technique and expected results |
| 113 | +- https://github.com/saeg/jaguar implements the described heuristics in java |
0 commit comments