Skip to content

bpo-45310: Fix parrallel shared memory tests #28661

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

serhiy-storchaka
Copy link
Member

@serhiy-storchaka serhiy-storchaka commented Sep 30, 2021

Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.

https://bugs.python.org/issue45310

Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
@serhiy-storchaka serhiy-storchaka added tests Tests in the Lib/test dir skip news 🔨 test-with-buildbots Test PR w/ buildbots; report in status section needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes labels Sep 30, 2021
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @serhiy-storchaka for commit 69ebc33 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 30, 2021
Comment on lines +3780 to +3781
name_tsmb = self._new_shm_name('test01_tsmb')
sms = shared_memory.SharedMemory(name_tsmb, create=True, size=512)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may add an helper method which create a name and calls SharedMemory() with it. Or leave the code as it is, it's up to you.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is used not only with SharedMemory, but with ShareableList.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't notice. It's just a minor coding style suggestion. The code is good as it is.

@serhiy-storchaka serhiy-storchaka merged commit eb4495e into python:main Oct 1, 2021
@miss-islington
Copy link
Contributor

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9, 3.10.
🐍🍒⛏🤖

@serhiy-storchaka serhiy-storchaka deleted the test-multiprocessing-shared-memory branch October 1, 2021 07:56
@miss-islington
Copy link
Contributor

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker eb4495e8e275c83d691add116c4f2b74e73e3cc8 3.9

@bedevere-bot
Copy link

GH-28669 is a backport of this pull request to the 3.10 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.10 only security fixes label Oct 1, 2021
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 1, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <[email protected]>
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Oct 1, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel..
(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <[email protected]>
@bedevere-bot bedevere-bot removed the needs backport to 3.9 only security fixes label Oct 1, 2021
@bedevere-bot
Copy link

GH-28670 is a backport of this pull request to the 3.9 branch.

@vstinner
Copy link
Member

vstinner commented Oct 1, 2021

Thanks for adding the comment, it's useful to me at least ;-)

serhiy-storchaka added a commit that referenced this pull request Oct 1, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
(cherry picked from commit eb4495e)
miss-islington added a commit that referenced this pull request Oct 1, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <[email protected]>
@bedevere-bot
Copy link

bedevere-bot commented Oct 1, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Shared 3.9 has failed when building commit 4d5d161.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/151/builds/271) and take a look at the build logs.
  4. Check if the failure is related to this commit (4d5d161) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/151/builds/271

Summary of the results of the build (if available):

Click to see traceback logs
Note: switching to '4d5d161d2aae41738d28e22bac5e1e08c01394bb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 4d5d161d2a [3.9] [bpo-45310](https://bugs.python.org/issue45310): Fix parrallel shared memory tests (GH-28661) (GH-28670)
Switched to and reset branch '3.9'

test_devpoll skipped -- test works only on Solaris OS family
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ok
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... skipped 'need select.epoll'
test_kqueue (__main__.SelectEINTRTest) ... ok
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok

----------------------------------------------------------------------
Ran 24 tests in 8.518s

OK (skipped=2)
test_turtle skipped -- No module named '_tkinter'
test_msilib skipped -- No module named '_msi'
test_startfile skipped -- object <module 'os' from '/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/os.py'> has no attribute 'startfile'
<string>:2: PendingDeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
test_idle skipped -- No module named '_tkinter'
test_winconsoleio skipped -- test only relevant on win32
test_ttk_guionly skipped -- No module named '_tkinter'
test_tk skipped -- No module named '_tkinter'
test_ioctl skipped -- Unable to open /dev/tty
Timeout (0:05:00)!
Thread 0x0000000801d51c00 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/selectors.py", line 416 in select
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1979 in _communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1134 in communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 227 in _run_process
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 265 in _runtest
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 299 in run
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 973 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 930 in _bootstrap

Thread 0x0000000801d51500 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/selectors.py", line 416 in select
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1979 in _communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1134 in communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 227 in _run_process
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 265 in _runtest
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 299 in run
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 973 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 930 in _bootstrap

Thread 0x0000000801d50e00 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/selectors.py", line 416 in select
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1979 in _communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1134 in communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 227 in _run_process
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 265 in _runtest
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 299 in run
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 973 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 930 in _bootstrap

Thread 0x0000000800fa3200 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/selectors.py", line 416 in select
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1979 in _communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1134 in communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 227 in _run_process
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 265 in _runtest
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 299 in run
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 973 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 930 in _bootstrap

Thread 0x0000000800e12000 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 157 in log
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 168 in display_progress
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 436 in display_result
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 449 in _process_result
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 471 in run_tests
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 488 in run_tests_multiprocess
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 533 in run_tests
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 708 in _main
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 655 in main
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 733 in main
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/__main__.py", line 2 in <module>
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/runpy.py", line 87 in _run_code
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/runpy.py", line 197 in _run_module_as_main

Cannot open file '/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/test-results.xml' for upload

pablogsal pushed a commit that referenced this pull request Oct 4, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <[email protected]>
@miss-islington
Copy link
Contributor

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

@miss-islington
Copy link
Contributor

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker eb4495e8e275c83d691add116c4f2b74e73e3cc8 3.8

@bedevere-bot
Copy link

GH-28979 is a backport of this pull request to the 3.8 branch.

ambv pushed a commit that referenced this pull request Oct 19, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.

(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <[email protected]>
@serhiy-storchaka serhiy-storchaka removed their assignment Jun 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip news tests Tests in the Lib/test dir
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants