Multiprocess does not work in a venv #1052
Description
Short description
The multiprocessing module does not work in a python3 venv on windows. This is related to the use of Popen to create a new python instance, and incorrect detection of PID
Code to reproduce
- Create a new venv:
$ mkdir tempvenv
$ cd tempvenv
$ python3 -m venv .- Try to spawn a multiprocess version of pyqtgraph
import pyqtgraph as pg
import pyqtgraph.multiprocess as mp
app = pg.mkQApp()
rp = mp.QtProcess()
ros = rp._import("os")Expected behavior
The last line should return a remote proxy to the os module on the remote python instance.
Real behavior
We get a key error:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
c:\users\x\documents\pyqtgraph\pyqtgraph\multiprocess\remoteproxy.py in handleRequest(self)
210 print(optStr)
211 print(os.getpid())
--> 212 opts = pickle.loads(optStr)
213 self.debugMsg(" handleRequest: id=%s opts=%s", reqId, opts)
214 #print os.getpid(), "received request:", cmd, reqId, opts
c:\users\x\documents\pyqtgraph\pyqtgraph\multiprocess\remoteproxy.py in unpickleObjectProxy(processId, proxyId, typeStr, attributes, opts)
735 return obj
736 else:
--> 737 proxy = ObjectProxy(processId, proxyId=proxyId, typeStr=typeStr)
738 if opts is not None:
739 proxy._setProxyOptions(**opts)
c:\users\x\documents\pyqtgraph\pyqtgraph\multiprocess\remoteproxy.py in __init__(self, processId, proxyId, typeStr, parent)
805 }
806
--> 807 self.__dict__['_handler'] = RemoteEventHandler.getHandler(processId)
808 self.__dict__['_handler'].registerProxy(self) ## handler will watch proxy; inform remote process when the proxy is deleted.
809
c:\users\x\documents\pyqtgraph\pyqtgraph\multiprocess\remoteproxy.py in getHandler(cls, pid)
93 def getHandler(cls, pid):
94 try:
---> 95 return cls.handlers[pid]
96 except:
97 print(pid, cls.handlers)
KeyError: 12608Tested environment(s)
- PyQtGraph version: 0.11.0.dev0 (ed6586c)
- Qt Python binding: PyQt5 5.12.3 Qt 5.12.5
- Python version: Python 3.7.4
- Operating system: Windows 10
Additional context
The reason this occurs is because in a venv on windows, python creates a stub executable in the venv, which then spawns a real instance of python.
This causes the detected PID of the child process to not match with the child process stored in RemoteEventHandler.handlers.
We can see this with a minimal example on windows. Running inside the venv:
import os, sys, time
from subprocess import Popen
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
if __name__ == '__main__':
if len(sys.argv) == 1:
info("Parent Process")
p = Popen((sys.executable, __file__, "proc"))
print("Detected Child PID:", p.pid)
while p.poll() is None:
time.sleep(1)
else:
info("Child Process")I get the output:
Parent Process
module name: __main__
parent process: 10220
process id: 17508
Detected Child PID: 10100
Child Process
module name: __main__
parent process: 10100
process id: 17480
Note that the detected child PID is 10100, while the child itself thinks it has the PID 17480. The child's parent process is 10100, which is the PID of the stub python.exe that venv creates.
Potential Solutions
- Using
multiprocessing.Processdoes not have this issue. - Send the child PID back once the child has spawned.