multiprocessing fails in MacOS due a pickle issue in a module in 10.5.0 [was: byexample fails to pickle an ExampleRunner] #220
Description
In our CI setup, we are using a custom C++ runner to check the output of C++ code snippets.
When we upgrade byexample from 10.4 to 10.5, running byexample on our snippets fails with:
[i:exec.cpp] Initializing C++ Interpreter
[w:exec.cpp] Initialization of C++ Interpreter failed.
[!] Something went wrong processing the file '/Users/runner/work/e-antic/e-antic/libeantic/test/byexample/../../../doc/manual/libeantic/overview_c.md':
Traceback (most recent call last):
File "/usr/local/miniconda/envs/test/lib/python3.10/site-packages/byexample/common.py", line 217, in human_exceptions
yield o
File "/usr/local/miniconda/envs/test/lib/python3.10/site-packages/byexample/byexample.py", line 22, in execute_examples
return executor.execute(examples, filename)
File "/usr/local/miniconda/envs/test/lib/python3.10/site-packages/byexample/log.py", line 267, in wrapped
return func(*args, **kargs)
File "/usr/local/miniconda/envs/test/lib/python3.10/site-packages/byexample/executor.py", line 271, in execute
self.initialize_runners(runners, options)
File "/usr/local/miniconda/envs/test/lib/python3.10/site-packages/byexample/executor.py", line 189, in initialize_runners
runner.initialize(options)
File "/Users/runner/work/e-antic/e-antic/libeantic/test/byexample/extensions/cxx.py", line 187, in initialize
self._child.start()
File "/usr/local/miniconda/envs/test/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/local/miniconda/envs/test/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/local/miniconda/envs/test/lib/python3.10/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/usr/local/miniconda/envs/test/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/usr/local/miniconda/envs/test/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/usr/local/miniconda/envs/test/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/usr/local/miniconda/envs/test/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'cxx.CxxInterpreter'>: it's not the same object as cxx.CxxInterpreter
The problem only appears in CI runs on macOS (works fine on Ubuntu in CI and also locally on ArchLinux.) It seems to be independent of the Python version, since downgrading to Python 3.9 did not fix this.
I have not tried to run this on a macOS machine myself yet, so I have no better reproducer to offer at the moment, than our full CI setup: it calls byexample through automake here https://github.com/flatsurf/e-antic/blob/master/.github/workflows/test.yml#L103
Obviously this is not a great help to debug it. I'll try to understand better myself what is going on here but maybe somebody has an idea already what might be the problem?
The implementation of our custom runner is here: https://github.com/flatsurf/e-antic/blob/master/libeantic/test/byexample/extensions/cxx.py
Expected behavior
We expected byexample to run our tests without an error instead of raising an error.
byexample version
We are seeing this problem with byexample 10.5.0 (build py310h2ec42d9_0) and python 3.10.2 (build hea1dfa3_3_cpython) as distributed by conda-forge.
We don't see the issue when downgrading byexample to 10.4.0 (build py310h2ec42d9_1.)