Skip to content

Update test workflow to run lighter testing on draft PRs #720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Jun 20, 2025

Conversation

ehennestad
Copy link
Collaborator

@ehennestad ehennestad commented Jun 12, 2025

Motivation

This PR introduces a few changes to improve CI speed and efficiency:

  • Run tests only on the latest MATLAB release when a pull request is in draft mode (i.e., no version matrix).
  • Cache MATLAB setup and Python dependencies to reduce setup time.
  • All tests still run — none are skipped.

Testing across all supported MATLAB releases can occasionally take significantly longer than usual (e.g., 27 minutes vs. the typical 12), especially when workers are queued or run slower than usual. These changes aim to make draft PR iterations faster while retaining full test coverage once the PR is marked ready for review.

How to test the behavior?

N/A

Checklist

  • Have you ensured the PR description clearly describes the problem and solutions?
  • Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
  • If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

Updated workflow to run tests only on latest release of MATLAB for draft PRs
Fixed json formatting bug
Cline suggested fix for error on github runner
Cline suggested fix for error on github runner
Added step for caching pip dependencies
Minor fix for failing test
Copy link

codecov bot commented Jun 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.08%. Comparing base (16862b7) to head (6008c84).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #720      +/-   ##
==========================================
+ Coverage   95.05%   95.08%   +0.02%     
==========================================
  Files         161      162       +1     
  Lines        5868     5897      +29     
==========================================
+ Hits         5578     5607      +29     
  Misses        290      290              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ehennestad ehennestad marked this pull request as ready for review June 12, 2025 13:47
@ehennestad ehennestad requested a review from bendichter June 12, 2025 18:30
@bendichter
Copy link
Contributor

Is this to get the tests to run faster? What tests are being skipped? How long does it take to run the tests before and after the change?

@ehennestad
Copy link
Collaborator Author

ehennestad commented Jun 19, 2025

A few changes to speed things up:

  1. only run tests with latest matlab release (no matrix) if PR is in draft mode

  2. cache matlab setup and python dependencies

No tests are skipped

Sometimes testing on all releases can take more time than usual (ie if some workers are queued or for other reasons take longer to finish) last week a test run took 27 min compared to usual 12.

Caching saves about 1 minute

bendichter
bendichter previously approved these changes Jun 20, 2025
Does not have significant speedup gains
Risk having outdated versions in cache as the cache does not detect if dependencies are updated
@ehennestad
Copy link
Collaborator Author

ehennestad commented Jun 20, 2025

Based on discussion:

  • Added more comments in the set-matrix workflow job
  • Removed the part adding EOF before and after a json string
  • Removed caching for pip dependencies. No noticable speedup + risk of using oudated versions of dependencies

@bendichter bendichter merged commit da1aa27 into NeurodataWithoutBorders:main Jun 20, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants