Summary
For Python 3.9.x through what's on main now,
When a worker process terminates abruptly, concurrent.futures.ProcessPoolExecutor creates a single BrokenProcessPool exception to share among all futures (code). This is unsafe because exceptions are mutable.
As a consequence, calling Future.result() on all failed futures will result in malformed tracebacks. Each traceback will be the concatenation of all prior tracebacks with the current one.
This issue can be resolved by creating a separate exception instance for each failed future.
Note that in the example below, the size of the malformed traceback is limited. In real-world applications, it can easily be thousands of lines.
Example Code
from concurrent.futures import ProcessPoolExecutor
import time
import traceback
def main():
with ProcessPoolExecutor() as e:
futures = [e.submit(time.sleep, 10) for _ in range(3)]
# Get one of the processes, and terminate (kill) it.
p = next(iter(e._processes.values()))
p.terminate()
for i, fut in enumerate(futures):
try:
fut.result()
except Exception as e:
print(f"\nException {i}:")
traceback.print_exception(e)
if __name__ == "__main__":
main()Current (Incorrect) Output
Notice that each consecutive traceback appends to the previous one.
Exception 0:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Exception 1:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Exception 2:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.New (Correct) Output
Exception 0:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Exception 1:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Exception 2:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.Linked PRs
Summary
For Python 3.9.x through what's on main now,
When a worker process terminates abruptly,
concurrent.futures.ProcessPoolExecutorcreates a singleBrokenProcessPoolexception to share among all futures (code). This is unsafe because exceptions are mutable.As a consequence, calling
Future.result()on all failed futures will result in malformed tracebacks. Each traceback will be the concatenation of all prior tracebacks with the current one.This issue can be resolved by creating a separate exception instance for each failed future.
Note that in the example below, the size of the malformed traceback is limited. In real-world applications, it can easily be thousands of lines.
Example Code
Current (Incorrect) Output
Notice that each consecutive traceback appends to the previous one.
New (Correct) Output
Linked PRs