Python Process which is joined will not call atexit

Question

I thought Python Processes call their atexit functions when they terminate. Note that I'm using Python 2.7. Here is a simple example:

from __future__ import print_function
import atexit
from multiprocessing import Process


def test():
    atexit.register(lambda: print("atexit function ran"))

process = Process(target=test)
process.start()
process.join()

I'd expect this to print "atexit function ran" but it does not.

Note that this question: Python process won't call atexit is similar, but it involves Processes that are terminated with a signal, and the answer involves intercepting that signal. The Processes in this question are exiting gracefully, so (as far as I can tell anyway) that question & answer do not apply (unless these Processes are exiting due to a signal somehow?).

@MichaelBrennan that question involves Processes which are terminated with a signal. My Processes are simply exiting gracefully, so this isn't a duplicate (the fix posted there doesn't apply here as far as I can tell, unless mine are getting killed by a signal too somehow?) — nonagon
– nonagon, Commented Oct 19, 2014 at 21:38
You're right, somehow I draw the conclusion that a signal is used even without terminate(). I've deleted the duplication comment. Perhaps you can set up a signal handler and try to find out if there are any signals sent? — user861537
– user861537, Commented Oct 20, 2014 at 6:38
The exit code (i.e. process.exitcode, evaluated after join executes) is zero so as far as I can tell. If I terminate the process I get a negative exit code as expected from the docs. So I'm pretty confused! I can't imagine something this fundamental is broken, but I can't find evidence anywhere that this shouldn't work as I expect. — nonagon
– nonagon, Commented Oct 20, 2014 at 15:48
A possible workaround is to make use of undocumented multiprocessing.util.Finalize class, see the blog post for details: Guaranteed Finalization Without Context Manager. — Delgan
– Delgan, Commented Apr 29, 2023 at 14:19
Another workaround is to force processes via 'spawn' instead of 'fork' by using set_start_method('spawn'), as this problem only happens when forking. These workarounds probably won't be necessary anymore though, as it seems that Python 3.13 will fix this problem with forked processes: github.com/python/cpython/pull/114279 — Rodrigo Bonadia
– Rodrigo Bonadia, Commented Aug 19, 2024 at 11:55

score 4 · Accepted Answer · 2014-10-20 23:44:03Z

4

I did some research by looking at how this is implemented in CPython. This is assumes you are running on Unix. If you are running on Windows the following might not be valid as the implementation of processes in multiprocessing differs.

It turns out that os._exit() is always called at the end of the process. That, together with the following note from the documentation for atexit, should explain why your lambda isn't running.

Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.

Here's an excerpt from the Popen class for CPython 2.7, used for forking processes. Note that the last statement of the forked process is a call to os._exit().

# Lib/multiprocessing/forking.py

class Popen(object):

    def __init__(self, process_obj):
        sys.stdout.flush()
        sys.stderr.flush()
        self.returncode = None

        self.pid = os.fork()
        if self.pid == 0:
            if 'random' in sys.modules:
                import random
                random.seed()
            code = process_obj._bootstrap()
            sys.stdout.flush()
            sys.stderr.flush()
            os._exit(code)

In Python 3.4, the os._exit() is still there if you are starting a forking process, which is the default. But it seems like you can change it, see Contexts and start methods for more information. I haven't tried it, but perhaps using a start method of spawn would work? Not available for Python 2.7 though.

edited Oct 20, 2014 at 23:44

answered Oct 20, 2014 at 23:28

user861537

Sign up to request clarification or add additional context in comments.

1 Comment

nonagon Over a year ago

That is great, thank you for figuring this out! It seems like the multiprocessing.Process docs should mention this limitation explicitly. Anyway I'll come up with another mechanism to solve this in my case.

Collectives™ on Stack Overflow

Python Process which is joined will not call atexit

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related