ci: test on Python 3.11 #2419

Molkree · 2022-12-04T01:27:24Z

Took the liberty to also edit the readme to remove explicit mentions of Python versions. This way there's no need to update it twice a year (when the old version gets dropped and the new one gets added).

~~Temporarily skipped language scanner tests to showcase 3.11 hopefully passing in CI. Will revert this commit before merging.~~ Reverted this and merged from main as the test was fixed.

Discovered that asyncio.coroutine was removed in 3.11 so guarded its import with version check, without it tests fail. Should be good because it's only used on versions less than 3.8 in the code.

Had to slightly change one test test_exception because of 3.11 changes: 9d6a3d8.

Awaiting feedback on whether we should change Windows/long/manyfail Python versions. ~~And whether to add a PyPI classifier in this PR.~~ Python 3.11 classifier added: 6a21641.

At least 2 tests are flaky, unsure if it's related to 3.11, see below: #2419 (comment).

codecov-commenter · 2022-12-04T01:32:04Z

Codecov Report

Merging #2419 (1378aa5) into main (a93f003) will decrease coverage by 5.35%.
The diff coverage is 57.14%.

@@            Coverage Diff             @@
##             main    #2419      +/-   ##
==========================================
- Coverage   82.34%   76.99%   -5.36%     
==========================================
  Files         591      591              
  Lines        9729     9731       +2     
  Branches     1319     1321       +2     
==========================================
- Hits         8011     7492     -519     
- Misses       1372     1940     +568     
+ Partials      346      299      -47

Flag	Coverage Δ
longtests	`76.99% <57.14%> (-4.88%)`	⬇️
win-longtests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
test/test_extractor.py	`95.07% <50.00%> (-0.97%)`	⬇️
test/test_version.py	`96.29% <66.66%> (-3.71%)`	⬇️
cve_bin_tool/data_sources/curl_source.py	`36.45% <0.00%> (-58.34%)`	⬇️
cve_bin_tool/data_sources/nvd_source.py	`22.82% <0.00%> (-35.69%)`	⬇️
cve_bin_tool/cvedb.py	`42.23% <0.00%> (-31.06%)`	⬇️
cve_bin_tool/data_sources/osv_source.py	`34.11% <0.00%> (-28.51%)`	⬇️
test/test_cli.py	`65.10% <0.00%> (-22.67%)`	⬇️
cve_bin_tool/data_sources/gad_source.py	`49.36% <0.00%> (-21.52%)`	⬇️
test/test_json.py	`68.96% <0.00%> (-20.69%)`	⬇️
cve_bin_tool/data_sources/redhat_source.py	`49.66% <0.00%> (-13.25%)`	⬇️
... and 16 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Molkree · 2022-12-04T02:01:35Z

Looks like another change is failing test_version tests, specifically the test_exception case.

In this test we mock exception during GET request to PyPI and previously we got expected string or bytes-like object in response but now we get expected string or bytes-like object, got 'MagicMock'.

I don't think the real exception will have anything to do with MagicMock... So unsure how to fix this properly, any ideas?

terriko · 2022-12-07T05:42:36Z

The tests in main should be ok now (although I suppose it could change when the db updates tonight).
I think adding 3.11 to the setup file is good as long as it's working (and it seems to be!)
I don't know what to do about the mocker stuff. Maybe disable the test and file an issue for now?

terriko

Looks good. I've added a suggestion about linking directly to the testing config (since that's one of those things that's obvious to some developers but probably the majority of users wouldn't know where to find that).

I think you'll want to

add 3.11 to setup.py
remove the code temporarily blocking the tests

And then see how it goes. If you want to rebase and see it running you probably need to go against b17a185 instead of the tip of main right now.

test/README.md

Co-authored-by: Terri Oda <[email protected]>

Molkree · 2022-12-30T23:53:17Z

What fails now:

test/test_extractor.py::TestExtractFileIpk::test_extract_file_ipk
Unsure if related to Sporadic CI error on extraction tests #1839. It passes in my WSL and fails on Windows but for another reason Docs claim that ar is installed by default on Windows #2477.
test/test_scanner.py::TestScanner::test_version_in_package, specifically this entry http://archive.ubuntu.com/ubuntu/pool/main/libv/libvncserver/-libvncserver1_0.9.12+dfsg-9ubuntu0.3_amd64.deb-libvncserver-0.9.12.
Here's a funny thing: in CI it fails on Linux. On my machine it fails on Windows and passes in WSL.

This reverts commit 2d9681b. Test was fixed in main.

Molkree · 2022-12-31T00:46:21Z

Well, another CI run and both of those tests passed:

Log

[gw1] [ 5%] PASSED test/test_extractor.py::TestExtractFileIpk::test_extract_file_ipk
Log

[gw2] [ 60%] PASSED test/test_scanner.py::TestScanner::test_version_in_package[http://archive.ubuntu.com/ubuntu/pool/main/libv/libvncserver/-libvncserver1_0.9.12+dfsg-9ubuntu0.3_amd64.deb-libvncserver-0.9.12]

I guess just shows how unreliable those tests are...

Molkree · 2022-12-31T01:23:10Z

I think you'll want to

add 3.11 to setup.py

remove the code temporarily blocking the tests

@terriko, both are done.

What about bumping Windows tests?

Currently, Windows tests run on 3.10 but Windows long tests run on 3.9. Do we want to bump them by one version? So Win tests on 3.11 and long Win tests on 3.10? Or maybe the opposite with long tests on 3.11?

I think the latter makes more sense, as 3.11 is default now (when you try to use Python).

Molkree · 2022-12-31T01:29:44Z

Sigh, and of course flaky tests fail again.

This time it's a different extraction test, deb and not ipk.

FAILED test/test_extractor.py::TestExtractFileDeb::test_extract_file_deb

Not sure if just a coincidence or not but deb tests use ar tool, same as ipk test that was failing before.
Old friend test_version_in_package, on this particular file again

test_version_in_package[http://archive.ubuntu.com/ubuntu/pool/main/libv/libvncserver/-libvncserver1_0.9.12+dfsg-9ubuntu0.3_amd64.deb-libvncserver-0.9.12]

Edit: both passed on the same commit in my fork:

Log

[gw0] [ 5%] PASSED test/test_extractor.py::TestExtractFileDeb::test_extract_file_deb
Log

[gw1] [ 58%] PASSED test/test_scanner.py::TestScanner::test_version_in_package[http://archive.ubuntu.com/ubuntu/pool/main/libv/libvncserver/-libvncserver1_0.9.12+dfsg-9ubuntu0.3_amd64.deb-libvncserver-0.9.12]

terriko · 2023-01-03T23:42:50Z

Ugh. Okay, unstable tests are bad. But it would be nice to have 3.11 tests running regularly. How do you feel about disabling test_extract_file_deb and test_extract_file_ipk on 3.11 for now and then merging this? They're already disabled on 3.7 due to being unstable there, but they've been solid on 3.8-3.10 so I'm really not sure what's going on here. You could be right that it's ar based, but I feel like that's not super likely to be affected by only two disparate versions of python unless there were subprocess changes that is maybe stopping ar before it finishes only on those versions and only sometimes? But who knows. Let's disable them and get a partial CI setup working, then see if we can figure it out now that we've got a lot more motivation than we did when it was just a 3.7 issue.

(joking: it must be that it hates prime numbers, so 7 and 11 are out but 8, 9, 10 are fine.)

Molkree · 2023-01-04T00:24:20Z

How do you feel about disabling test_extract_file_deb and test_extract_file_ipk on 3.11 for now and then merging this?

Sure.

But what about test_version_in_package[http://archive.ubuntu.com/ubuntu/pool/main/libv/libvncserver/-libvncserver1_0.9.12+dfsg-9ubuntu0.3_amd64.deb-libvncserver-0.9.12]? Do you want to keep that one in? Skip the whole test_version_in_package with 3.11? Or this specific package (haven't seen any failures on other packages yet)?

terriko · 2023-01-04T00:49:47Z

If you can skip just that one, go ahead. If not, disable them all and then we'll file a bug to look into it further, I guess.

terriko · 2023-01-04T00:59:48Z

(Disable it all for python 3.11 only, that is)

Molkree · 2023-01-04T02:34:20Z

What about bumping Windows tests?

Currently, Windows tests run on 3.10 but Windows long tests run on 3.9. Do we want to bump them by one version? So Win tests on 3.11 and long Win tests on 3.10? Or maybe the opposite with long tests on 3.11?

I think the latter makes more sense, as 3.11 is default now (when you try to use Python).

@terriko, any thoughts on this?

terriko · 2023-01-04T19:08:59Z

I'm going to punt on updating the python used in windows tests until we can figure out what's wrong with the ones we currently have to skip.

But in theory: yes to long tests on 3.11 and windows short tests on 3.10 once we're sure they're not going to be flaky and that deb extraction is actually working as expected. (Or vice versa if there's a particular reason to go the other way, such as performance or availability of the things we need for long tests. But this isn't the bad old days of python 2.7 where half the libraries didn't work in 3 for years so I doubt it's going to make much difference in minor revs.)

terriko

Okay, let's get this merged. I'll file a separate bug to look into the flaky tests on 3.11. (The fail is in the HTML tests which are also flaky but I don't have a resolution for those yet either.)

Molkree added 4 commits December 4, 2022 02:10

Test Python 3.11 in CI

3e56fe3

docs: remove Python version mentions from readme

ee222d1

test: temporarily skip failing tests

2d9681b

fix: guard against removed asyncio.coroutine

f13d453

terriko requested changes Dec 7, 2022

View reviewed changes

test/README.md Show resolved Hide resolved

Molkree and others added 4 commits December 12, 2022 00:27

docs: point to testing workflow for Python versions

bd2a33c

Co-authored-by: Terri Oda <[email protected]>

Merge branch 'main' into py311

17063a4

Merge branch 'main' into py311

d5b50e4

test(test_exception): fix to work on Python 3.11

9d6a3d8

Revert "test: temporarily skip failing tests"

c573502

This reverts commit 2d9681b. Test was fixed in main.

chore: add Python 3.11 classifier for PyPI

6a21641

Molkree requested a review from terriko December 31, 2022 01:40

test: disable flaky tests on Python 3.11

1378aa5

test: skip another flaky test

0ee90cd

terriko approved these changes Jan 4, 2023

View reviewed changes

terriko merged commit c5bc53d into intel:main Jan 4, 2023

terriko mentioned this pull request Jan 4, 2023

fix flaky tests on python 3.11 #2486

Closed

Molkree deleted the py311 branch January 4, 2023 19:50

ci: test on Python 3.11 #2419

ci: test on Python 3.11 #2419

Uh oh!

Conversation

Molkree commented Dec 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Dec 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Molkree commented Dec 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terriko commented Dec 7, 2022

Uh oh!

terriko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Molkree commented Dec 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Molkree commented Dec 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Molkree commented Dec 31, 2022

Uh oh!

Molkree commented Dec 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terriko commented Jan 3, 2023

Uh oh!

Molkree commented Jan 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terriko commented Jan 4, 2023

Uh oh!

terriko commented Jan 4, 2023

Uh oh!

Molkree commented Jan 4, 2023

Uh oh!

terriko commented Jan 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terriko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Molkree commented Dec 4, 2022 •

edited

Loading

codecov-commenter commented Dec 4, 2022 •

edited

Loading

Molkree commented Dec 4, 2022 •

edited

Loading

Molkree commented Dec 30, 2022 •

edited

Loading

Molkree commented Dec 31, 2022 •

edited

Loading

Molkree commented Dec 31, 2022 •

edited

Loading

Molkree commented Jan 4, 2023 •

edited

Loading

terriko commented Jan 4, 2023 •

edited

Loading