Skip to content

Commit d34307b

Browse files
dorotat-nvdorotat-nv
authored
run pytest with or without docs and notebooks in run_pytest.sh (NVIDIA#569)
### Description We've streamlined our testing workflow to improve CI efficiency. By default, documentation tests and notebook validations are skipped to reduce pipeline duration. ### Type of changes <!-- Mark the relevant option with an [x] --> - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Refactor - [x] Documentation update - [x] Other (please describe): user guideline, updated to CI workflow ### CI Pipeline Configuration Check the boxes below to configure CI (continuous integration) behavior. These will automatically apply labels. > **Note:** Leave boxes unchecked by default, check only if you want to modify default behavior.] - [ ] [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/dorotat/pytest-nbval-on-demand/docs/docs/user-guide/contributing/contributing.md#skip_ci) Skip continuous integration pipeline - [x] [RUN_NOTEBOOKS](https://github.com/NVIDIA/bionemo-framework/blob/dorotat/pytest-nbval-on-demand/docs/docs/user-guide/contributing/contributing.md#run_notebooks) Run notebook validation tests (default: skipped) - [x] [RUN_DOCS](https://github.com/NVIDIA/bionemo-framework/blob/dorotat/pytest-nbval-on-demand/docs/docs/user-guide/contributing/contributing.md#run_docs) Run documentation tests (default: skipped) ### Usage When running tests locally, use these flags to control test execution: ```bash ./ci/scripts/run_pytest.sh --skip-docs ./ci/scripts/run_pytest.sh --no-nbval ./ci/scripts/run_pytest.sh --no-nbval --skip-docs ``` Control test pipeline execution by adding these labels to your PR: `RUN_DOCS` - Enables documentation testing `RUN_NOTEBOOKS_VALIDATION` - Enables notebook validation ### Pre-submit Checklist <!--- Ensure all items are completed before submitting --> - [x] I have tested these changes locally - [x] I have updated the documentation accordingly - [ ] I have added/updated tests as needed - [ ] All existing tests pass successfully --------- Signed-off-by: Dorota Toczydlowska <[email protected]> Signed-off-by: Dorota Toczydlowska <[email protected]> Signed-off-by: dorotat-nv <[email protected]>
1 parent f12a475 commit d34307b

File tree

4 files changed

+186
-54
lines changed

4 files changed

+186
-54
lines changed

‎.github/pull_request_template.md

Lines changed: 24 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,45 +1,34 @@
1-
(**NOTE:** _**delete** these instructional lines as you fill-out this PR template_)
1+
### Description
2+
<!-- Provide a detailed description of the changes in this PR -->
23

3-
(**NOTE:** _template is designed to be filled-in and used as the **squashed commit message for the entire PR**. _Italicized text_ is intended to be deleted as you fill in this template. Use the text between the `---`)
4+
### Type of changes
5+
<!-- Mark the relevant option with an [x] -->
46

5-
---
7+
- [ ] Bug fix (non-breaking change which fixes an issue)
8+
- [ ] New feature (non-breaking change which adds functionality)
9+
- [ ] Refactor
10+
- [ ] Documentation update
11+
- [ ] Other (please describe):
612

7-
_High level summary of changes. Try to keep this as short and informative as possible: less is more._
13+
### CI Pipeline Configuration
14+
Configure CI behavior by checking relevant boxes below. This will automatically apply labels.
815

9-
_Describe your changes. You can be more detailed and descriptive here. If it is a code change, Be sure to answer:_
10-
- _What is changing?_
11-
- _What is the new or fixed functionality?_
12-
- _Why or when would someone want to use these changes?_
13-
- _How can someone use these changes?_
14-
---
16+
- [ ] [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests
17+
- [ ] [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest
1518

16-
## Summary
17-
_High level summary of changes. Try to keep this as short and informative as possible: less is more._
19+
> [!NOTE]
20+
> By default, the notebooks validation tests are skipped unless explicitly enabled.
1821
19-
## Details
20-
_Describe your changes. You can be more detailed and descriptive here._
21-
22-
## Usage
23-
_How does a user interact with the changed code?_
22+
### Usage
23+
<!--- How does a user interact with the changed code -->
2424
```python
25-
python -m your.new.module -and -all -options
26-
```
27-
28-
## Testing
29-
_How do you prove that your code behaves the way you claim?_
30-
31-
Tests for these changes can be run via:
32-
```shell
33-
pytest -v tests/your/new/or/existing/test_functions.py::test_function
25+
TODO: Add code snippet
3426
```
3527

28+
### Pre-submit Checklist
29+
<!--- Ensure all items are completed before submitting -->
3630

37-
(**NOTE:** _also **delete** this checklist as you fill-out this PR template_)
38-
39-
**Most of the changes** to files with extensions `*.py`, `*.yaml`, `*.yml`, `Dockerfile*` or `requirements.txt` **DO REQUIRE both `pytest-` and `jet-` CI stages**.
40-
41-
- [ ] Did you review the [Before your PR is "Ready for review" section](https://github.com/NVIDIA/bionemo-framework/-/blob/dev/CONTRIBUTING.md?ref_type=heads#before-pr-ready) before asking for review?
42-
- [ ] Did you make sure your changes have tests? Did you test your changes locally?
43-
- [ ] Can you add [the `SKIP_CI` label](https://github.com/NVIDIA/bionemo-framework/-/blob/dev/CONTRIBUTING.md?ref_type=heads#skip-ci) to your PR?
44-
- [ ] Can you add [the `PYTEST_NOT_REQUIRED` label](https://github.com/NVIDIA/bionemo-framework/-/blob/dev/CONTRIBUTING.md?ref_type=heads#skip-pytest) to your PR?
45-
- [ ] Can you add [the `JET_NOT_REQUIRED` label](https://github.com/NVIDIA/bionemo-framework/-/blob/dev/CONTRIBUTING.md?ref_type=heads#skip-jet) to your PR?
31+
- [ ] I have tested these changes locally
32+
- [ ] I have updated the documentation accordingly
33+
- [ ] I have added/updated tests as needed
34+
- [ ] All existing tests pass successfully

‎.github/workflows/pr-labels.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: PR Label Management
2+
3+
on:
4+
pull_request:
5+
types: [opened, edited, synchronize]
6+
7+
jobs:
8+
manage-labels:
9+
runs-on: ubuntu-latest
10+
permissions:
11+
pull-requests: write
12+
13+
steps:
14+
- name: Check PR body and manage labels
15+
uses: actions/github-script@v6
16+
with:
17+
script: |
18+
const prBody = context.payload.pull_request.body;
19+
20+
const labelChecks = {
21+
'SKIP_CI': /\[x\]\s*\[SKIP_CI\]/i,
22+
'INCLUDE_NOTEBOOKS_TESTS': /\[x\]\s*\[INCLUDE_NOTEBOOKS_TESTS\]/i
23+
};
24+
25+
const currentLabels = new Set(
26+
context.payload.pull_request.labels.map(label => label.name)
27+
);
28+
29+
for (const [label, pattern] of Object.entries(labelChecks)) {
30+
const shouldHaveLabel = pattern.test(prBody);
31+
const hasLabel = currentLabels.has(label);
32+
33+
if (shouldHaveLabel && !hasLabel) {
34+
await github.rest.issues.addLabels({
35+
owner: context.repo.owner,
36+
repo: context.repo.repo,
37+
issue_number: context.payload.pull_request.number,
38+
labels: [label]
39+
});
40+
} else if (!shouldHaveLabel && hasLabel) {
41+
await github.rest.issues.removeLabel({
42+
owner: context.repo.owner,
43+
repo: context.repo.repo,
44+
issue_number: context.payload.pull_request.number,
45+
name: label
46+
});
47+
}
48+
}

‎ci/scripts/run_pytest.sh

Lines changed: 95 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -15,30 +15,107 @@
1515
# See the License for the specific language governing permissions and
1616
# limitations under the License.
1717

18-
set -xueo pipefail
19-
export PYTHONDONTWRITEBYTECODE=1
20-
# NOTE: if a non-nvidia user wants to run the test suite, just run `export BIONEMO_DATA_SOURCE=ngc` prior to this call.
21-
export BIONEMO_DATA_SOURCE="${BIONEMO_DATA_SOURCE:-pbss}"
22-
# flexible GPU memory management, reducing the risk of fragmentation-related CUDA OOM
23-
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
24-
source "$(dirname "$0")/utils.sh"
25-
26-
if ! set_bionemo_home; then
27-
exit 1
28-
fi
2918

30-
python -m coverage erase
19+
# Enable strict mode with better error handling
20+
set -ueo pipefail
21+
22+
# Function to display usage information
23+
usage() {
24+
cat << EOF
25+
Usage: $(basename "$0") [OPTIONS]
26+
27+
Options:
28+
--skip-docs Skip running tests in the docs directory
29+
--no-nbval Skip jupyter notebook validation tests
30+
31+
Note: Documentation tests (docs/) are only run when notebook validation
32+
is enabled (--no-nbval not set) and docs are not skipped
33+
(--skip-docs not set)
34+
-h, --help Display this help message
35+
EOF
36+
exit "${1:-0}"
37+
}
38+
39+
# Function to handle cleanup on script exit
40+
cleanup() {
41+
local exit_code=$?
42+
[ -n "${coverage_files[*]:-}" ] && rm -f "${coverage_files[@]:-}"
43+
exit "$exit_code"
44+
}
3145

46+
# Set default environment variables
47+
: "${BIONEMO_DATA_SOURCE:=pbss}"
48+
: "${PYTHONDONTWRITEBYTECODE:=1}"
49+
: "${PYTORCH_CUDA_ALLOC_CONF:=expandable_segments:True}"
50+
51+
# Export necessary environment variables
52+
export BIONEMO_DATA_SOURCE PYTHONDONTWRITEBYTECODE PYTORCH_CUDA_ALLOC_CONF
53+
54+
# Initialize variables
55+
declare -a coverage_files
56+
SKIP_DOCS=false
57+
NO_NBVAL=false
3258
error=false
33-
for dir in docs/ ./sub-packages/bionemo-*/; do
59+
60+
# Parse command line arguments
61+
while (( $# > 0 )); do
62+
case "$1" in
63+
--skip-docs) SKIP_DOCS=true ;;
64+
--no-nbval) NO_NBVAL=true ;;
65+
-h|--help) usage ;;
66+
*) echo "Unknown option: $1" >&2; usage 1 ;;
67+
esac
68+
shift
69+
done
70+
71+
# Set up trap for cleanup
72+
trap cleanup EXIT
73+
74+
# Source utility functions
75+
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
76+
source "$SCRIPT_DIR/utils.sh" || { echo "Failed to source utils.sh" >&2; exit 1; }
77+
78+
# Set up BioNeMo home directory
79+
set_bionemo_home || exit 1
80+
81+
# Clear previous coverage data
82+
python -m coverage erase
83+
84+
# Set up pytest options
85+
PYTEST_OPTIONS=(
86+
-v
87+
--durations=0
88+
--durations-min=60.0
89+
)
90+
[[ "$NO_NBVAL" != true ]] && PYTEST_OPTIONS+=(--nbval-lax)
91+
92+
# Define test directories
93+
TEST_DIRS=(./sub-packages/bionemo-*/)
94+
if [[ "$NO_NBVAL" != true && "$SKIP_DOCS" != true ]]; then
95+
TEST_DIRS+=(docs/)
96+
fi
97+
98+
echo "Test directories: ${TEST_DIRS[*]}"
99+
100+
# Run tests with coverage
101+
for dir in "${TEST_DIRS[@]}"; do
34102
echo "Running pytest in $dir"
35-
python -m coverage run --parallel-mode --source=bionemo \
36-
-m pytest -v --nbval-lax --durations=0 --durations-min=60.0 "$dir" || error=true
103+
coverage_file=".coverage.${dir//\//_}"
104+
coverage_files+=("$coverage_file")
105+
106+
if ! python -m coverage run \
107+
--parallel-mode \
108+
--source=bionemo \
109+
--data-file="$coverage_file" \
110+
-m pytest "${PYTEST_OPTIONS[@]}" "$dir"; then
111+
error=true
112+
fi
37113
done
38114

115+
# Combine and report coverage
39116
python -m coverage combine
40117
python -m coverage report --show-missing
41118

42-
if [ "$error" = true ]; then
43-
exit 1
44-
fi
119+
# Exit with appropriate status
120+
$error && exit 1
121+
exit 0

‎docs/docs/user-guide/contributing/contributing.md

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ repository (unless external constraints prevent it).
6464

6565
## Pull Request (PR) Guidelines
6666

67-
### Labeling Your PR
67+
### Labeling Your PR as External Contributor
6868

6969
If you are an external contributor (not an NVIDIA employee), please add the `contribution` label to your PR before submitting. Labels can be accessed in the right sidebar of the GitHub user interface when creating or editing a PR.
7070

@@ -135,6 +135,24 @@ If you are an external contributor (not an NVIDIA employee), please add the `con
135135
redistributed consistent with this project or the open source license(s) involved.
136136
```
137137
138+
### CI Pipeline Configuration Controls
139+
140+
CI pipeline behavior can be controlled via checkboxes in PR descriptions to optimize test execution:
141+
142+
Key behaviors:
143+
- Controls processed automatically on PR submit/update
144+
- Labels applied based on checkbox status
145+
- Invalid combinations default to most restrictive option
146+
147+
#### **SKIP_CI**
148+
- Skips entire CI pipeline
149+
- Use for documentation typos, README updates
150+
151+
#### **INCLUDE_NOTEBOOKS_TESTS**
152+
- Enables notebook validation tests
153+
- Use when modifying notebooks or notebook-related code
154+
- Disabled by default
155+
138156
### Developer workflows:
139157
140158
You should always carefully test your changes. Run `pytest ...` in your container locally. All tests are done via `pytest`.

0 commit comments

Comments
 (0)