Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-45310: Fix parrallel shared memory tests #28661

Conversation

serhiy-storchaka
Copy link
Member

@serhiy-storchaka serhiy-storchaka commented Sep 30, 2021

Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.

https://bugs.python.org/issue45310

Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
@serhiy-storchaka serhiy-storchaka added tests skip news 🔨 test-with-buildbots needs backport to 3.9 needs backport to 3.10 labels Sep 30, 2021
@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Sep 30, 2021

🤖 New build scheduled with the buildbot fleet by @serhiy-storchaka for commit 69ebc33 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots label Sep 30, 2021
Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved
name_tsmb = self._new_shm_name('test01_tsmb')
sms = shared_memory.SharedMemory(name_tsmb, create=True, size=512)
Copy link
Member

@vstinner vstinner Sep 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may add an helper method which create a name and calls SharedMemory() with it. Or leave the code as it is, it's up to you.

Copy link
Member Author

@serhiy-storchaka serhiy-storchaka Sep 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is used not only with SharedMemory, but with ShareableList.

Copy link
Member

@vstinner vstinner Sep 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't notice. It's just a minor coding style suggestion. The code is good as it is.

Lib/test/_test_multiprocessing.py Show resolved Hide resolved
@serhiy-storchaka serhiy-storchaka merged commit eb4495e into python:main Oct 1, 2021
11 checks passed
@miss-islington
Copy link
Contributor

@miss-islington miss-islington commented Oct 1, 2021

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9, 3.10.
🐍🍒🤖

@serhiy-storchaka serhiy-storchaka deleted the test-multiprocessing-shared-memory branch Oct 1, 2021
@miss-islington
Copy link
Contributor

@miss-islington miss-islington commented Oct 1, 2021

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker eb4495e8e275c83d691add116c4f2b74e73e3cc8 3.9

@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Oct 1, 2021

GH-28669 is a backport of this pull request to the 3.10 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 1, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Oct 1, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel..
(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Oct 1, 2021

GH-28670 is a backport of this pull request to the 3.9 branch.

@vstinner
Copy link
Member

@vstinner vstinner commented Oct 1, 2021

Thanks for adding the comment, it's useful to me at least ;-)

serhiy-storchaka added a commit that referenced this issue Oct 1, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
(cherry picked from commit eb4495e)
miss-islington added a commit that referenced this issue Oct 1, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Oct 1, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Shared 3.9 has failed when building commit 4d5d161.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/151/builds/271) and take a look at the build logs.
  4. Check if the failure is related to this commit (4d5d161) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/151/builds/271

Summary of the results of the build (if available):

Click to see traceback logs
Note: switching to '4d5d161d2aae41738d28e22bac5e1e08c01394bb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 4d5d161d2a [3.9] [bpo-45310](https://bugs.python.org/issue45310): Fix parrallel shared memory tests (GH-28661) (GH-28670)
Switched to and reset branch '3.9'

test_devpoll skipped -- test works only on Solaris OS family
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ok
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... skipped 'need select.epoll'
test_kqueue (__main__.SelectEINTRTest) ... ok
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok

----------------------------------------------------------------------
Ran 24 tests in 8.518s

OK (skipped=2)
test_turtle skipped -- No module named '_tkinter'
test_msilib skipped -- No module named '_msi'
test_startfile skipped -- object <module 'os' from '/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/os.py'> has no attribute 'startfile'
<string>:2: PendingDeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
test_idle skipped -- No module named '_tkinter'
test_winconsoleio skipped -- test only relevant on win32
test_ttk_guionly skipped -- No module named '_tkinter'
test_tk skipped -- No module named '_tkinter'
test_ioctl skipped -- Unable to open /dev/tty
Timeout (0:05:00)!
Thread 0x0000000801d51c00 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/selectors.py", line 416 in select
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1979 in _communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1134 in communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 227 in _run_process
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 265 in _runtest
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 299 in run
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 973 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 930 in _bootstrap

Thread 0x0000000801d51500 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/selectors.py", line 416 in select
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1979 in _communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1134 in communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 227 in _run_process
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 265 in _runtest
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 299 in run
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 973 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 930 in _bootstrap

Thread 0x0000000801d50e00 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/selectors.py", line 416 in select
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1979 in _communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1134 in communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 227 in _run_process
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 265 in _runtest
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 299 in run
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 973 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 930 in _bootstrap

Thread 0x0000000800fa3200 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/selectors.py", line 416 in select
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1979 in _communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/subprocess.py", line 1134 in communicate
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 227 in _run_process
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 265 in _runtest
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 299 in run
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 973 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/threading.py", line 930 in _bootstrap

Thread 0x0000000800e12000 (most recent call first):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 157 in log
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 168 in display_progress
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 436 in display_result
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 449 in _process_result
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 471 in run_tests
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest_mp.py", line 488 in run_tests_multiprocess
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 533 in run_tests
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 708 in _main
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 655 in main
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/main.py", line 733 in main
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/__main__.py", line 2 in <module>
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/runpy.py", line 87 in _run_code
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/runpy.py", line 197 in _run_module_as_main

Cannot open file '/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/test-results.xml' for upload

pablogsal pushed a commit that referenced this issue Oct 4, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@miss-islington
Copy link
Contributor

@miss-islington miss-islington commented Oct 15, 2021

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒🤖

@miss-islington
Copy link
Contributor

@miss-islington miss-islington commented Oct 15, 2021

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker eb4495e8e275c83d691add116c4f2b74e73e3cc8 3.8

@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Oct 15, 2021

GH-28979 is a backport of this pull request to the 3.8 branch.

ambv pushed a commit that referenced this issue Oct 19, 2021
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.

(cherry picked from commit eb4495e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA signed skip news tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants