Skip to content

CI: Flaky tests #632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
samuel-w opened this issue Sep 8, 2020 · 8 comments · Fixed by #648 or #788
Closed

CI: Flaky tests #632

samuel-w opened this issue Sep 8, 2020 · 8 comments · Fixed by #648 or #788
Assignees
@m3nu
Labels
priority:low Nice to have feature, minor improvement to functionality or usability type:task A TODO
Milestone
0.7.3

Comments

@samuel-w
Copy link
Contributor

samuel-w commented Sep 8, 2020 •
edited
Loading

Tests sometimes fail without any bad code, examples:
https://github.com/borgbase/vorta/pull/631/checks?check_run_id=1086248541
https://github.com/borgbase/vorta/pull/540/checks?check_run_id=1079839809
https://github.com/borgbase/vorta/pull/628/checks?check_run_id=1085059935
https://github.com/borgbase/vorta/pull/607/checks?check_run_id=1087127941
https://github.com/borgbase/vorta/pull/607/checks?check_run_id=1087128002

Since we can run the tests again, its just annoying.
git commit --allow-empty -m "Make tests go brrrrr"

Maybe fix with pytest-xdist and upping timeouts?

Edit:
https://github.com/borgbase/vorta/runs/1203963170
@samuel-w samuel-w added the priority:low Nice to have feature, minor improvement to functionality or usability label Sep 8, 2020
@samuel-w samuel-w changed the title CI BUG: Flaky tests CI: Flaky tests Sep 8, 2020
@samuel-w samuel-w added good first issue Simple change to start learning code base help wanted This issue is available, comment if you want to fix it labels Sep 8, 2020
@m3nu
Copy link
Contributor

m3nu commented Sep 9, 2020

It's always macOS machines failing. Sometimes there are errors regarding the GUI and then it's very slow. The limit for macOS is also more strict for Github workflows.

I will

  • increase the time a bit
  • Limit tests on macOS to only Python 3.7

@m3nu m3nu self-assigned this Sep 9, 2020
@m3nu m3nu closed this as completed in f6e7b54 Sep 9, 2020
m3nu added a commit to m3nu/vorta that referenced this issue Sep 9, 2020
@m3nu
Reduce tests on macOS and increase limit. Fixes borgbase#632
9d2e362
@samuel-w
Copy link
Contributor Author

samuel-w commented Sep 9, 2020 •
edited by m3nu
Loading

Some failures are database related though:
https://github.com/borgbase/vorta/pull/635/checks?check_run_id=1088827625

https://github.com/borgbase/vorta/runs/1100436520

@m3nu
Copy link
Contributor

m3nu commented Sep 9, 2020

You're correct. I noticed that before, but it got more frequent recently.

This looks like a race condition, where our BorgThread tries to save a log, while the next test already started and the said table is deleted. I'll look into this again. Just hard to reproduce.

@m3nu m3nu reopened this Sep 9, 2020
@m3nu m3nu added type:task A TODO and removed good first issue Simple change to start learning code base help wanted This issue is available, comment if you want to fix it labels Sep 9, 2020
@samuel-w
Copy link
Contributor Author

Could just add a gap in between tests https://stackoverflow.com/questions/47770037/interval-between-tests-in-pytest

@m3nu
Copy link
Contributor

m3nu commented Sep 13, 2020

I think the cleaner way is to ensure no background threads are running at the end of the test.

@m3nu m3nu mentioned this issue Sep 13, 2020
Merged
@m3nu
Copy link
Contributor

m3nu commented Sep 13, 2020 •
edited
Loading

See my PR to cancel background threads after each test. I ran this 11x and didn't encounter a false positive test failure.

Let me know how you find that PR.

10:30 - ok
10:32 - ok
10:35 - ok
10:40 - ok
10:43 - ok
10:47 - ok
10:51 - ok
10:56 - ok
11:00 - ok
11:04 - ok

@m3nu m3nu closed this as completed in #648 Sep 13, 2020
m3nu added a commit that referenced this issue Sep 13, 2020
@m3nu
Cancel Borg thread at end of test. (#648)
a86683e 
Maybe fixes #632
@m3nu m3nu reopened this Sep 13, 2020
@m3nu
Copy link
Contributor

m3nu commented Sep 13, 2020

Leave this open, since there are other unreliable tests left.

@m3nu m3nu added this to the 0.7.3 (Next minor release) milestone Jan 31, 2021
This was referenced Feb 11, 2021
Closed
Merged
@m3nu
Copy link
Contributor

m3nu commented Feb 15, 2021

Closing in favor of more detailed discussion on same topic: #748

@m3nu m3nu closed this as completed Feb 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@m3nu m3nu

Labels
priority:low Nice to have feature, minor improvement to functionality or usability type:task A TODO
Projects
None yet
Milestone
0.7.3
Development

Successfully merging a pull request may close this issue.

Cancel Borg thread at end of test. m3nu/vorta
Fix flaky tests m3nu/vorta
2 participants
@m3nu @samuel-w