Skip to content

[WIP/RFC] multiprocessing: proxy: keep _manager after forking#17333

Draft
blueyed wants to merge 1 commit intopython:mainfrom
blueyed:mp-manager
Draft

[WIP/RFC] multiprocessing: proxy: keep _manager after forking#17333
blueyed wants to merge 1 commit intopython:mainfrom
blueyed:mp-manager

Conversation

@blueyed
Copy link
Contributor

@blueyed blueyed commented Nov 22, 2019

Fixes iterating over dicts:

import multiprocessing as mp

def run(d):
    for k in d:
        print(k)

manager = mp.Manager()

d = manager.dict({1: 2, 3: 4})

process = mp.Process(target=run, args=(d,))
process.start()
process.join()

With this fix:

_callmethod __iter__ #PROXY (('__next__', 'send', 'throw', 'close'), Token(typeid='Iterator', address='/tmp/pymp-wfn3r3nd/listener-ze4ow_dl', id='7f12414ca130'))
_callmethod __next__ #RETURN 1
1
_callmethod __next__ #RETURN 3
3
_callmethod __next__ #ERROR

Without:

_callmethod __iter__ #PROXY (('__next__', 'send', 'throw', 'close'), Token(typeid='Iterator', address='/tmp/pymp-uppzhf9p/listener-mvmwwxma', id='7f5b7b42da90'))
Process Process-2:
Traceback (most recent call last):
  File "…/pyenv/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "…/pyenv/3.8.0/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "t-mp.py", line 5, in run
    for k in d:
  File "<string>", line 2, in __iter__
  File "…/pyenv/3.8.0/lib/python3.8/multiprocessing/managers.py", line 842, in _callmethod
    proxytype = self._manager._registry[token.typeid][-1]
AttributeError: 'NoneType' object has no attribute '_registry'

The line is from when multiprocessing was added in e711caf, so I am
probably missing something here? This PR is meant to get tests run with
it, and of course for any feedback already.

@blueyed
[WIP/RFC] multiprocessing: proxy: keep _manager after forking
825358a 
Fixes iterating over dicts:

    import multiprocessing as mp

    def run(d):
        for k in d:
            print(k)

    manager = mp.Manager()

    d = manager.dict({1: 2, 3: 4})

    process = mp.Process(target=run, args=(d,))
    process.start()
    process.join()

With this fix:

    _callmethod __iter__ #PROXY (('__next__', 'send', 'throw', 'close'), Token(typeid='Iterator', address='/tmp/pymp-wfn3r3nd/listener-ze4ow_dl', id='7f12414ca130'))
    _callmethod __next__ #RETURN 1
    1
    _callmethod __next__ #RETURN 3
    3
    _callmethod __next__ #ERROR

Without:

    _callmethod __iter__ #PROXY (('__next__', 'send', 'throw', 'close'), Token(typeid='Iterator', address='/tmp/pymp-uppzhf9p/listener-mvmwwxma', id='7f5b7b42da90'))
    Process Process-2:
    Traceback (most recent call last):
      File "…/pyenv/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
        self.run()
      File "…/pyenv/3.8.0/lib/python3.8/multiprocessing/process.py", line 108, in run
        self._target(*self._args, **self._kwargs)
      File "t-mp.py", line 5, in run
        for k in d:
      File "<string>", line 2, in __iter__
      File "…/pyenv/3.8.0/lib/python3.8/multiprocessing/managers.py", line 842, in _callmethod
        proxytype = self._manager._registry[token.typeid][-1]
    AttributeError: 'NoneType' object has no attribute '_registry'

The line is from when multiprocessing was added in e711caf, so I am
probably missing something here?  This PR is meant to get tests run with
it, and of course for any feedback already.
@blueyed blueyed mentioned this pull request Nov 22, 2019
Merged
blueyed added a commit to blueyed/pytest that referenced this pull request Nov 22, 2019
@blueyed
terminal: _get_main_color: help pytest-parallel
1f736a6 
Use `dict.keys()` to work around `__iter__` not working with a
multiprocessing DictProxy.

Ref: python/cpython#17333
Fixes pytest-dev#6254.
Ref: kevlened/pytest-parallel#36
blueyed added a commit to blueyed/pytest that referenced this pull request Nov 22, 2019
@blueyed
terminal: _get_main_color: help pytest-parallel
0c53e67 
Use `dict.keys()` to work around `__iter__` not working with a
multiprocessing DictProxy.

Ref: python/cpython#17333
Fixes pytest-dev#6254.
Ref: kevlened/pytest-parallel#36

(cherry picked from commit 1f736a6)
@blueyed
Copy link
Contributor Author

blueyed commented Nov 24, 2019

/cc @benjaminp

vinaycalastry pushed a commit to vinaycalastry/pytest that referenced this pull request Dec 11, 2019
@blueyed @vinaycalastry
terminal: _get_main_color: help pytest-parallel
5453bbb 
Use `dict.keys()` to work around `__iter__` not working with a
multiprocessing DictProxy.

Ref: python/cpython#17333
Fixes pytest-dev#6254.
Ref: kevlened/pytest-parallel#36
@invisibleroads
Copy link
Contributor

invisibleroads commented Jul 29, 2022

@benjaminp Do you know why self._manager is set to None in _after_fork?

The above-referenced issue prevents dictionary syntax like list(d), set(d), for x in d when using manager.dict()

However, list(d.keys()), set(d.keys()), for x in d.keys() work as noted in https://stackoverflow.com/a/70886363

@serhiy-storchaka serhiy-storchaka marked this pull request as draft January 17, 2024 20:08
@python-cla-bot
Copy link

python-cla-bot bot commented Apr 6, 2025

The following commit authors need to sign the Contributor License Agreement:

CLA signed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead Awaiting requested review from gpshead gpshead will be requested when the pull request is marked ready for review gpshead is a code owner

Assignees

No one assigned

Labels

topic-multiprocessing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@blueyed @invisibleroads @vstinner @the-knights-who-say-ni @ezio-melotti @bedevere-bot