Skip to content

add asynchronous support for the current Tracer#161

Merged
palazzem merged 95 commits into
masterfrom
palazzem/async-python
Mar 1, 2017
Merged

add asynchronous support for the current Tracer#161
palazzem merged 95 commits into
masterfrom
palazzem/async-python

Conversation

@palazzem

@palazzem palazzem commented Feb 2, 2017

Copy link
Copy Markdown

What it does

This PR aims to support:

  • Change the trace() approach to support asynchronous programming
  • asyncio stdlib
  • aiohttp and aiohttp_jinja2
  • gevent

Usage with synchronous code

Nothing has changed and you can instrument, patch() and patch_all() the same way as before.

Current status

  • asyncio
  • aiohttp
  • aiohttp_jinja2
  • Simplified API
  • manual instrumentation made easy
  • Gevent

Note about Tornado

Because Tornado requires more attention, it will be moved in a separated PR so that the review will be easier. Currently, the overall approach is working for Tornado too so this PR is good to be merged. Reference PR: #204

Related PRs that compose this feature branch

#167, #171, #172, #202, #192, #191, #188

Comment thread ddtrace/context.py Outdated
execution flow.

TODO: asyncio is not thread-safe by default. The fact that this class is
thread-safe is an implementation detail. Avoid mutex usage when the Context

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think the mutex hurts much?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually they should be avoided in async stuff because it's supposed to be single threaded. Actually I'm keeping it to have one single Context so yes, probably we can just keep it.

@clutchski

Copy link
Copy Markdown
Contributor

@palazzem can you add summary usage? i.e. what's the new public API?

@clutchski

clutchski commented Feb 3, 2017

Copy link
Copy Markdown
Contributor

we can't have from ddtrace.contrib.asyncio import tracer appear anywhere in user or integration code or thigns won't play well together.

what i think we need is

# in the configuration code
if os.getenv("DATADOG_TRACE_ASYNC"):
    ddtrace.tracer = async.Tracer()

# in other code (being completely safe)
import ddtrace as dd

with dd.tracer.trace(...) as span:
    pass

# ... or making sure you're importing after configuration
from ddtrace import tracer
with tracer.trace(...) as span:
    pass

this would eliminate the need to pass along the tracer into all of the patch methods as well.

Comment thread ddtrace/contrib/asyncio/tracer.py Outdated
try:
# return the active Context for this task (if any)
return task.__datadog_context
except (KeyError, AttributeError):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getattr instead?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I will fix all these nitpicks because I didn't pay attention to a lot of stuff while achieving the async compatibility.

@clutchski

clutchski commented Feb 3, 2017

Copy link
Copy Markdown
Contributor

I'd like this PR to be a step towards opentracing[1], so it might be nice to think about it in terms of a few types of functions:

  • plain functions which do no magic at all:
tracer.new_span("foo")
tracer.new_child_span("foo", parent_id, ...)
  • functions which can pluck the data they need from an explictly passed thread-local, async, whatever context, which stores the parent span and any other needed info. context
tracer.new_child_from_context("foo", my_context)
  • one function which magically does everything based on a globally configured context (or context factory):
 tracer.context_factory = AsyncContextFactory
 tracer.trace("foo")

if we can get something good and share it with the OT folks, then we have a plan / path forward to moving our stuff to the OT api.

[1] - https://github.com/opentracing/opentracing-python

@palazzem palazzem force-pushed the palazzem/async-python branch from 2f8012f to 492c8e6 Compare February 5, 2017 16:34
Comment thread ddtrace/contrib/redis/patch.py Outdated


def patch():
def patch(tracer=None):

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we override the default tracer (ddtrace.tracer), we may avoid passing the tracer around our instrumentation code. As suggested, we can simply:

import ddtrace

# at some point
tracer = ddtrace.tracer

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part will be removed in favor of a global configurable tracer.

Comment thread ddtrace/contrib/tornado/__init__.py Outdated
@@ -0,0 +1,22 @@
"""
TODO: how to use Tornado instrumentation

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example that creates a root span for all handlers (and attaches a Context in the handler request):

from ddtrace.contrib.tornado.middlewares import TraceMiddleware

class MainHandler(tornado.web.RequestHandler):
    @tornado.gen.coroutine
    def get(self):
        pass

def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])

if __name__ == "__main__":
    app = make_app()
    http_server = app.listen(8000)
    TraceMiddleware(http_server, tracer, service='mytornado')
    tornado.ioloop.IOLoop.current().start()

Comment thread ddtrace/contrib/tornado/__init__.py Outdated
from .stack_context import ContextManager

# a global Tornado tracer instance
tracer = TornadoTracer()

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to a proposal, this code will be removed in favor of a way to set the default tracer.

@palazzem palazzem force-pushed the palazzem/async-python branch from d9a928f to 338eb06 Compare February 7, 2017 09:48
pin = Pin.get_from(aiohttp_jinja2)
if not pin or not pin.enabled():
return func(*args, **kwargs)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could remind the render_template signature here, making it simpler to understand/really showing what args[0] / args[1] are.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh sure!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will introduce a signature protection here; in case the render_template() will be changed in future aiohttp_jinja2 releases

Comment thread ddtrace/tracer.py Outdated

def trace(self, name, service=None, resource=None, span_type=None):
"""Return a span that will trace an operation called `name`.
def trace(self, name, service=None, resource=None, span_type=None, ctx=None, span_parent=None):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will have to be very rigorous about our documentation her, since it won't be clear when span_parent must be used vs. when ctx will be enough.
Also, we could technically have span_parent only (ctx can come from it). But I'm not sure that is the nicerst API.
What is OT doing here?

@palazzem palazzem Feb 7, 2017

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I think that a good idea is always infer the context from the span if it's present. Something like: https://github.com/opentracing/opentracing-python/blob/master/opentracing/tracer.py#L72-L74

Also it doesn't make sense to have a parent span in a different Context.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, happy to see that OT is doing it that way ; I'd vote for relying on the parent_span only too.

@LotharSee

Copy link
Copy Markdown
Contributor

Concerning the selection of the tracer based on an env var (such as DATADOG_TRACE_ASYNC), I'm kind of against it.

It is a pain because it tries to make things magic in a way you ignore, you have to set it every time you call your all or any script from the codebase, configuring it in many places, a config certainly maintained in another spot not in sync with the code, etc...

We had a similar pain in dogweb with protobuf, where we had to set special env var to make it work the proper way: it meant defining it at 5 places in Chef, creating bugs every time we missed one, then we overwrote the venv directly in the dogweb code...

So, I'm not against having it as an extra sweet helper.
But I'd still allow and document the way to get the right tracer from the code.

Comment thread ddtrace/contrib/asyncio/tracer.py Outdated
# create a new Context using the Task as a Context carrier
# TODO: we may not want to create Context everytime
ctx = Context()
task.__datadog_context = ctx

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use set_call_context.
Also, set_call_context could be a static method.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Until we agree in the public API (keeping or not different kind of Tracer, or relying in something else) I will wait to do that change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@palazzem palazzem force-pushed the palazzem/async-python branch from 9f8c0de to 7d24e67 Compare February 7, 2017 19:33
Comment thread ddtrace/contrib/asyncio/tracer.py Outdated

# create a new Context using the Task as a Context carrier
ctx = Context()
setattr(task, '__datadog_context', ctx)

@palazzem palazzem Feb 7, 2017

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LotharSee discussion moved from here: #161 (comment)

@palazzem palazzem force-pushed the palazzem/async-python branch 3 times, most recently from 8c102e4 to 6036f33 Compare February 10, 2017 09:58
@palazzem palazzem added this to the 0.6.0 milestone Feb 11, 2017
@palazzem palazzem self-assigned this Feb 11, 2017

@ross ross left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimm

Comment thread ddtrace/contrib/tornado/middlewares.py Outdated
self._tracer = tracer
self._service = service
# the default http_server callback must be preserved
self._request_callback = http_server.request_callback

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work if the caller modifies/sets the callback after wrapping the server with TraceMiddleware. That'd overwrite what's happening below in line 31 and the middleware wouldn't be invoked. That should probably at least be documented if it's not a supported situation.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Ross for your details. Actually the work on Tornado isn't finished because there are cases like you said that should be handled. Also I'm defining what are the supported situations, given the fact that the tracer has a new public API that makes easier to instrument the code in asynchronous environment.

Will keep you updated.

Comment thread ddtrace/contrib/tornado/middlewares.py Outdated
"""
Wraps the Application class handler with tracing methods.
"""
cls.on_finish = handlers.wrapper_on_finish(cls.on_finish)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other place where things would want a hook into is Handler._handle_request_exception. It's the only way I could see to get the actual exception information included in the trace. http://www.tornadoweb.org/en/stable/_modules/tornado/web.html & search for def _handle_request_exception. You could potentially use def log_exception if it gets called in all of the needed situations, haven't looked into that. At the very least it seems cleaner given that it's intended to be overridden.

Anyway, the call into on_finish doesn't have enough information to actually grab the details of the exception for inclusion in the span. I was using a dedicated exception tracking system so I didn't get to the point of integrating thais w/APM, but had I not being using the other system I would have been looking for APM to record that information.

@palazzem palazzem added the wip label Feb 14, 2017
Nicolas Martyanoff and others added 4 commits February 20, 2017 11:11
We introduce contexts. A contexts store a hierarchy of spans; this way,
it is possible to have multiple traces in parallel, as long as each
component keeps track of its context.

The tracer contains a default tracer which is used for all traces
created without the Context argument. This means that the existing code
continues to work the same way without any modification.
…ware when adding / finishing new spans for the current trace flow
@palazzem

palazzem commented Mar 1, 2017

Copy link
Copy Markdown
Author

Because Tornado requires more attention, it will be moved in a separated PR so that the review will be easier. Currently, the overall approach is working for Tornado too so this PR is good to be merged. Reference PR: #204

Emanuele Palazzetti added 8 commits March 1, 2017 11:33
Improving current documentation
[asyncio] make the tracer.wrap() works with coroutines
[asyncio] improving asynchronous support for the stdlib module
[aiohttp] convert stateful TraceMiddleware in a trace_middleware
[gevent] improving support for the new tracing API
@palazzem palazzem changed the title [WIP] asynchronous support for the current Tracer asynchronous support for the current Tracer Mar 1, 2017
@palazzem palazzem changed the title asynchronous support for the current Tracer add asynchronous support for the current Tracer Mar 1, 2017
@palazzem palazzem removed the wip label Mar 1, 2017
@palazzem palazzem merged commit e96fa48 into master Mar 1, 2017
@palazzem palazzem deleted the palazzem/async-python branch March 1, 2017 13:25
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Apr 23, 2026
…llocator (#17664)

## Description

This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced.  The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer.

````
Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0   0x00007ff3c303a8d4  
#1   0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68)
#2   0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68)
#3   0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80)
#4   0x00007ff3c3087e1b PyUnicode_New 
#5   0x00007ff3c30889f4  
#6   0x00007ff3c3170c84  
#7   0x00007ff3c316b931  
#8   0x00007ff3c31aaac8  
#9   0x00007ff3c31033ac  
#10  0x00007ff3c310e2a6 PyObject_CallMethodObjArgs 
#11  0x00007ff3c310e46d  
#12  0x00007ff3c31a96c2  
#13  0x00007ff3c3102fd7 PyObject_Vectorcall 
#14  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#15  0x00007ff3c323c094  
#16  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#17  0x00007ff3c323c094  
#18  0x00007ff3c30e997d PyObject_CallOneArg 
#19  0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict 
#20  0x00007ff3c30c620d PyObject_GetAttr 
#21  0x00007ff3c32309e7 _PyEval_EvalFrameDefault 
#22  0x00007ff3c323c094  
#23  0x00007ff3c312880e  
#24  0x00007ff3c30e917c _PyObject_MakeTpCall 
#25  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#26  0x00007ff3c323c094  
#27  0x00007ff3c312880e  
#28  0x00007ff3c30e917c _PyObject_MakeTpCall 
#29  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#30  0x00007ff3c323c094  
#31  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#32  0x00007ff3c323c094  
#33  0x00007ff3c317d0fd  
#34  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#35  0x00007ff3c323c094  
#36  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#37  0x00007ff3c323c094  
#38  0x00007ff3c317d1b5  
#39  0x00007ff3c3102fd7 PyObject_Vectorcall 
#40  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#41  0x00007ff3c3240da5  
#42  0x00007ff3c324112d  
#43  0x00007ff3c3233be1 _PyEval_EvalFrameDefault 
#44  0x00007ff3c323c094  
#45  0x00007ff3c317d1b5  
#46  0x00007ff3c3102fd7 PyObject_Vectorcall 
#47  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#48  0x00007ff3c323c094  
#49  0x00007ff3c31033ac  
#50  0x00007ff3c310358d PyObject_CallFunctionObjArgs 
#51  0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750)
#52  0x00007ff3c3104055 _PyObject_Call 
#53  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#54  0x00007ff3c323c094  
#55  0x00007ff3c317d23c  
#56  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#57  0x00007ff3c323c094  
#58  0x00007ff3c310416f _PyObject_Call 
#59  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#60  0x00007ff3c3240da5  
#61  0x00007ff3c324112d  
#62  0x00007ff3c3233be1 _PyEval_EvalFrameDefault 
#63  0x00007ff3c323c094  
#64  0x00007ff3c317d1b5  
#65  0x00007ff3c3102fd7 PyObject_Vectorcall 
#66  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#67  0x00007ff3c323c094  
#68  0x00007ff3c317d1b5  
#69  0x00007ff3c310416f _PyObject_Call 
#70  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#71  0x00007ff3c323c094  
#72  0x00007ff3c317d1b5  
#73  0x00007ff3c310416f _PyObject_Call 
#74  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#75  0x00007ff3c323c094  
#76  0x00007ff3c31033ac  
#77  0x00007ff3c310358d PyObject_CallFunctionObjArgs 
#78  0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750)
#79  0x00007ff3c30e917c _PyObject_MakeTpCall 
#80  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#81  0x00007ff3c323c094  
#82  0x00007ff3c317d518  
#83  0x00007ff3c3155963  
#84  0x00007ff3c315393d  
#85  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#86  0x00007ff3c323c094  
#87  0x00007ff3c317d0fd  
#88  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#89  0x00007ff3c323c094  
#90  0x00007ff3c317d0fd  
#91  0x00007ff3c317d518  
#92  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#93  0x00007ff3c323c094  
#94  0x00007ff3c30e9371 _PyObject_FastCallDictTstate 
#95  0x00007ff3c30e958d _PyObject_Call_Prepend 
#96  0x00007ff3c3109150  
#97  0x00007ff3c3104055 _PyObject_Call 
#98  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#99  0x00007ff3c323c094  
#100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#101 0x00007ff3c30e958d _PyObject_Call_Prepend 
#102 0x00007ff3c3109150  
#103 0x00007ff3c30e917c _PyObject_MakeTpCall 
#104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#105 0x00007ff3c323c094  
#106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#107 0x00007ff3c30e958d _PyObject_Call_Prepend 
#108 0x00007ff3c3109150  
#109 0x00007ff3c30e917c _PyObject_MakeTpCall 
#110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#111 0x00007ff3c323c094  
#112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#113 0x00007ff3c30e958d _PyObject_Call_Prepend 
#114 0x00007ff3c3109150  
#115 0x00007ff3c30e917c _PyObject_MakeTpCall 
#116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#117 0x00007ff3c323c094  
#118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#119 0x00007ff3c30e958d _PyObject_Call_Prepend 
#120 0x00007ff3c3109150  
#121 0x00007ff3c30e917c _PyObject_MakeTpCall 
#122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#123 0x00007ff3c323c094  
#124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#125 0x00007ff3c30e958d _PyObject_Call_Prepend 
#126 0x00007ff3c3109150  
#127 0x00007ff3c30e917c _PyObject_MakeTpCall 
#128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#129 0x00007ff3c323c094  
#130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#131 0x00007ff3c30e958d _PyObject_Call_Prepend 
#132 0x00007ff3c3109150  
#133 0x00007ff3c30e917c _PyObject_MakeTpCall 
#134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#135 0x00007ff3c323c094  
#136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#137 0x00007ff3c30e958d _PyObject_Call_Prepend 
#138 0x00007ff3c3109150  
#139 0x00007ff3c30e917c _PyObject_MakeTpCall 
#140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#141 0x00007ff3c323c094  
#142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#143 0x00007ff3c30e958d _PyObject_Call_Prepend 
#144 0x00007ff3c3109150  
#145 0x00007ff3c30e917c _PyObject_MakeTpCall 
#146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#147 0x00007ff3c323c094  
#148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#149 0x00007ff3c30e958d _PyObject_Call_Prepend 
#150 0x00007ff3c3109150  
#151 0x00007ff3c30e917c _PyObject_MakeTpCall 
#152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#153 0x00007ff3c323c094  
#154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#155 0x00007ff3c30e958d _PyObject_Call_Prepend 
#156 0x00007ff3c3109150  
#157 0x00007ff3c30e917c _PyObject_MakeTpCall 
#158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#159 0x00007ff3c323c094  
#160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#161 0x00007ff3c30e958d _PyObject_Call_Prepend 
#162 0x00007ff3c3109150  
#163 0x00007ff3c30e917c _PyObject_MakeTpCall 
#164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#165 0x00007ff3c323c094  
#166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#167 0x00007ff3c30e958d _PyObject_Call_Prepend 
#168 0x00007ff3c3109150  
#169 0x00007ff3c30e917c _PyObject_MakeTpCall 
#170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#171 0x00007ff3c323c094  
#172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#173 0x00007ff3c30e958d _PyObject_Call_Prepend 
#174 0x00007ff3c3109150  
#175 0x00007ff3c30e917c _PyObject_MakeTpCall 
#176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#177 0x00007ff3c323c094  
#178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#179 0x00007ff3c30e958d _PyObject_Call_Prepend 
#180 0x00007ff3c3109150  
#181 0x00007ff3c30e917c _PyObject_MakeTpCall 
#182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#183 0x00007ff3c323c094  
#184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#185 0x00007ff3c30e958d _PyObject_Call_Prepend 
#186 0x00007ff3c3109150  
#187 0x00007ff3c30e917c _PyObject_MakeTpCall 
#188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#189 0x00007ff3c323c094  
#190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#191 0x00007ff3c323c094  
#192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#193 0x00007ff3c30e958d _PyObject_Call_Prepend 
#194 0x00007ff3c3109150  
#195 0x00007ff3c30e917c _PyObject_MakeTpCall 
#196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#197 0x00007ff3c323c094  
#198 0x00007ff3c317d0fd  
#199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#200 0x00007ff3c323c094  
#201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#202 0x00007ff3c323c094  
#203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#204 0x00007ff3c323c094  
#205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#206 0x00007ff3c323c094  
#207 0x00007ff3c317d23c  
#208 0x00007ff3c31a7ec5  
#209 0x00007ff3c301ac77  
#210 0x00007ff3c357c573  
````

The fix implements two key changes.

1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race.

2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state.

The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently.  By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state.

Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>
emmettbutler pushed a commit that referenced this pull request Apr 24, 2026
…llocator (#17664)

## Description

This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced.  The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer.

````
Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0   0x00007ff3c303a8d4  
#1   0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68)
#2   0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68)
#3   0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80)
#4   0x00007ff3c3087e1b PyUnicode_New 
#5   0x00007ff3c30889f4  
#6   0x00007ff3c3170c84  
#7   0x00007ff3c316b931  
#8   0x00007ff3c31aaac8  
#9   0x00007ff3c31033ac  
#10  0x00007ff3c310e2a6 PyObject_CallMethodObjArgs 
#11  0x00007ff3c310e46d  
#12  0x00007ff3c31a96c2  
#13  0x00007ff3c3102fd7 PyObject_Vectorcall 
#14  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#15  0x00007ff3c323c094  
#16  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#17  0x00007ff3c323c094  
#18  0x00007ff3c30e997d PyObject_CallOneArg 
#19  0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict 
#20  0x00007ff3c30c620d PyObject_GetAttr 
#21  0x00007ff3c32309e7 _PyEval_EvalFrameDefault 
#22  0x00007ff3c323c094  
#23  0x00007ff3c312880e  
#24  0x00007ff3c30e917c _PyObject_MakeTpCall 
#25  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#26  0x00007ff3c323c094  
#27  0x00007ff3c312880e  
#28  0x00007ff3c30e917c _PyObject_MakeTpCall 
#29  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#30  0x00007ff3c323c094  
#31  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#32  0x00007ff3c323c094  
#33  0x00007ff3c317d0fd  
#34  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#35  0x00007ff3c323c094  
#36  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#37  0x00007ff3c323c094  
#38  0x00007ff3c317d1b5  
#39  0x00007ff3c3102fd7 PyObject_Vectorcall 
#40  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#41  0x00007ff3c3240da5  
#42  0x00007ff3c324112d  
#43  0x00007ff3c3233be1 _PyEval_EvalFrameDefault 
#44  0x00007ff3c323c094  
#45  0x00007ff3c317d1b5  
#46  0x00007ff3c3102fd7 PyObject_Vectorcall 
#47  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#48  0x00007ff3c323c094  
#49  0x00007ff3c31033ac  
#50  0x00007ff3c310358d PyObject_CallFunctionObjArgs 
#51  0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750)
#52  0x00007ff3c3104055 _PyObject_Call 
#53  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#54  0x00007ff3c323c094  
#55  0x00007ff3c317d23c  
#56  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#57  0x00007ff3c323c094  
#58  0x00007ff3c310416f _PyObject_Call 
#59  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#60  0x00007ff3c3240da5  
#61  0x00007ff3c324112d  
#62  0x00007ff3c3233be1 _PyEval_EvalFrameDefault 
#63  0x00007ff3c323c094  
#64  0x00007ff3c317d1b5  
#65  0x00007ff3c3102fd7 PyObject_Vectorcall 
#66  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#67  0x00007ff3c323c094  
#68  0x00007ff3c317d1b5  
#69  0x00007ff3c310416f _PyObject_Call 
#70  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#71  0x00007ff3c323c094  
#72  0x00007ff3c317d1b5  
#73  0x00007ff3c310416f _PyObject_Call 
#74  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#75  0x00007ff3c323c094  
#76  0x00007ff3c31033ac  
#77  0x00007ff3c310358d PyObject_CallFunctionObjArgs 
#78  0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750)
#79  0x00007ff3c30e917c _PyObject_MakeTpCall 
#80  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#81  0x00007ff3c323c094  
#82  0x00007ff3c317d518  
#83  0x00007ff3c3155963  
#84  0x00007ff3c315393d  
#85  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#86  0x00007ff3c323c094  
#87  0x00007ff3c317d0fd  
#88  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#89  0x00007ff3c323c094  
#90  0x00007ff3c317d0fd  
#91  0x00007ff3c317d518  
#92  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#93  0x00007ff3c323c094  
#94  0x00007ff3c30e9371 _PyObject_FastCallDictTstate 
#95  0x00007ff3c30e958d _PyObject_Call_Prepend 
#96  0x00007ff3c3109150  
#97  0x00007ff3c3104055 _PyObject_Call 
#98  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#99  0x00007ff3c323c094  
#100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#101 0x00007ff3c30e958d _PyObject_Call_Prepend 
#102 0x00007ff3c3109150  
#103 0x00007ff3c30e917c _PyObject_MakeTpCall 
#104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#105 0x00007ff3c323c094  
#106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#107 0x00007ff3c30e958d _PyObject_Call_Prepend 
#108 0x00007ff3c3109150  
#109 0x00007ff3c30e917c _PyObject_MakeTpCall 
#110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#111 0x00007ff3c323c094  
#112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#113 0x00007ff3c30e958d _PyObject_Call_Prepend 
#114 0x00007ff3c3109150  
#115 0x00007ff3c30e917c _PyObject_MakeTpCall 
#116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#117 0x00007ff3c323c094  
#118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#119 0x00007ff3c30e958d _PyObject_Call_Prepend 
#120 0x00007ff3c3109150  
#121 0x00007ff3c30e917c _PyObject_MakeTpCall 
#122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#123 0x00007ff3c323c094  
#124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#125 0x00007ff3c30e958d _PyObject_Call_Prepend 
#126 0x00007ff3c3109150  
#127 0x00007ff3c30e917c _PyObject_MakeTpCall 
#128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#129 0x00007ff3c323c094  
#130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#131 0x00007ff3c30e958d _PyObject_Call_Prepend 
#132 0x00007ff3c3109150  
#133 0x00007ff3c30e917c _PyObject_MakeTpCall 
#134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#135 0x00007ff3c323c094  
#136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#137 0x00007ff3c30e958d _PyObject_Call_Prepend 
#138 0x00007ff3c3109150  
#139 0x00007ff3c30e917c _PyObject_MakeTpCall 
#140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#141 0x00007ff3c323c094  
#142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#143 0x00007ff3c30e958d _PyObject_Call_Prepend 
#144 0x00007ff3c3109150  
#145 0x00007ff3c30e917c _PyObject_MakeTpCall 
#146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#147 0x00007ff3c323c094  
#148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#149 0x00007ff3c30e958d _PyObject_Call_Prepend 
#150 0x00007ff3c3109150  
#151 0x00007ff3c30e917c _PyObject_MakeTpCall 
#152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#153 0x00007ff3c323c094  
#154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#155 0x00007ff3c30e958d _PyObject_Call_Prepend 
#156 0x00007ff3c3109150  
#157 0x00007ff3c30e917c _PyObject_MakeTpCall 
#158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#159 0x00007ff3c323c094  
#160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#161 0x00007ff3c30e958d _PyObject_Call_Prepend 
#162 0x00007ff3c3109150  
#163 0x00007ff3c30e917c _PyObject_MakeTpCall 
#164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#165 0x00007ff3c323c094  
#166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#167 0x00007ff3c30e958d _PyObject_Call_Prepend 
#168 0x00007ff3c3109150  
#169 0x00007ff3c30e917c _PyObject_MakeTpCall 
#170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#171 0x00007ff3c323c094  
#172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#173 0x00007ff3c30e958d _PyObject_Call_Prepend 
#174 0x00007ff3c3109150  
#175 0x00007ff3c30e917c _PyObject_MakeTpCall 
#176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#177 0x00007ff3c323c094  
#178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#179 0x00007ff3c30e958d _PyObject_Call_Prepend 
#180 0x00007ff3c3109150  
#181 0x00007ff3c30e917c _PyObject_MakeTpCall 
#182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#183 0x00007ff3c323c094  
#184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#185 0x00007ff3c30e958d _PyObject_Call_Prepend 
#186 0x00007ff3c3109150  
#187 0x00007ff3c30e917c _PyObject_MakeTpCall 
#188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#189 0x00007ff3c323c094  
#190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#191 0x00007ff3c323c094  
#192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#193 0x00007ff3c30e958d _PyObject_Call_Prepend 
#194 0x00007ff3c3109150  
#195 0x00007ff3c30e917c _PyObject_MakeTpCall 
#196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#197 0x00007ff3c323c094  
#198 0x00007ff3c317d0fd  
#199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#200 0x00007ff3c323c094  
#201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#202 0x00007ff3c323c094  
#203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#204 0x00007ff3c323c094  
#205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#206 0x00007ff3c323c094  
#207 0x00007ff3c317d23c  
#208 0x00007ff3c31a7ec5  
#209 0x00007ff3c301ac77  
#210 0x00007ff3c357c573  
````

The fix implements two key changes.

1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race.

2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state.

The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently.  By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state.

Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>
emmettbutler pushed a commit that referenced this pull request May 6, 2026
…llocator (#17664)

## Description

This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced.  The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer.

````
Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0   0x00007ff3c303a8d4  
#1   0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68)
#2   0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68)
#3   0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80)
#4   0x00007ff3c3087e1b PyUnicode_New 
#5   0x00007ff3c30889f4  
#6   0x00007ff3c3170c84  
#7   0x00007ff3c316b931  
#8   0x00007ff3c31aaac8  
#9   0x00007ff3c31033ac  
#10  0x00007ff3c310e2a6 PyObject_CallMethodObjArgs 
#11  0x00007ff3c310e46d  
#12  0x00007ff3c31a96c2  
#13  0x00007ff3c3102fd7 PyObject_Vectorcall 
#14  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#15  0x00007ff3c323c094  
#16  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#17  0x00007ff3c323c094  
#18  0x00007ff3c30e997d PyObject_CallOneArg 
#19  0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict 
#20  0x00007ff3c30c620d PyObject_GetAttr 
#21  0x00007ff3c32309e7 _PyEval_EvalFrameDefault 
#22  0x00007ff3c323c094  
#23  0x00007ff3c312880e  
#24  0x00007ff3c30e917c _PyObject_MakeTpCall 
#25  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#26  0x00007ff3c323c094  
#27  0x00007ff3c312880e  
#28  0x00007ff3c30e917c _PyObject_MakeTpCall 
#29  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#30  0x00007ff3c323c094  
#31  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#32  0x00007ff3c323c094  
#33  0x00007ff3c317d0fd  
#34  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#35  0x00007ff3c323c094  
#36  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#37  0x00007ff3c323c094  
#38  0x00007ff3c317d1b5  
#39  0x00007ff3c3102fd7 PyObject_Vectorcall 
#40  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#41  0x00007ff3c3240da5  
#42  0x00007ff3c324112d  
#43  0x00007ff3c3233be1 _PyEval_EvalFrameDefault 
#44  0x00007ff3c323c094  
#45  0x00007ff3c317d1b5  
#46  0x00007ff3c3102fd7 PyObject_Vectorcall 
#47  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#48  0x00007ff3c323c094  
#49  0x00007ff3c31033ac  
#50  0x00007ff3c310358d PyObject_CallFunctionObjArgs 
#51  0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750)
#52  0x00007ff3c3104055 _PyObject_Call 
#53  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#54  0x00007ff3c323c094  
#55  0x00007ff3c317d23c  
#56  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#57  0x00007ff3c323c094  
#58  0x00007ff3c310416f _PyObject_Call 
#59  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#60  0x00007ff3c3240da5  
#61  0x00007ff3c324112d  
#62  0x00007ff3c3233be1 _PyEval_EvalFrameDefault 
#63  0x00007ff3c323c094  
#64  0x00007ff3c317d1b5  
#65  0x00007ff3c3102fd7 PyObject_Vectorcall 
#66  0x00007ff3c3232f4a _PyEval_EvalFrameDefault 
#67  0x00007ff3c323c094  
#68  0x00007ff3c317d1b5  
#69  0x00007ff3c310416f _PyObject_Call 
#70  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#71  0x00007ff3c323c094  
#72  0x00007ff3c317d1b5  
#73  0x00007ff3c310416f _PyObject_Call 
#74  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#75  0x00007ff3c323c094  
#76  0x00007ff3c31033ac  
#77  0x00007ff3c310358d PyObject_CallFunctionObjArgs 
#78  0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750)
#79  0x00007ff3c30e917c _PyObject_MakeTpCall 
#80  0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#81  0x00007ff3c323c094  
#82  0x00007ff3c317d518  
#83  0x00007ff3c3155963  
#84  0x00007ff3c315393d  
#85  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#86  0x00007ff3c323c094  
#87  0x00007ff3c317d0fd  
#88  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#89  0x00007ff3c323c094  
#90  0x00007ff3c317d0fd  
#91  0x00007ff3c317d518  
#92  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#93  0x00007ff3c323c094  
#94  0x00007ff3c30e9371 _PyObject_FastCallDictTstate 
#95  0x00007ff3c30e958d _PyObject_Call_Prepend 
#96  0x00007ff3c3109150  
#97  0x00007ff3c3104055 _PyObject_Call 
#98  0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#99  0x00007ff3c323c094  
#100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#101 0x00007ff3c30e958d _PyObject_Call_Prepend 
#102 0x00007ff3c3109150  
#103 0x00007ff3c30e917c _PyObject_MakeTpCall 
#104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#105 0x00007ff3c323c094  
#106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#107 0x00007ff3c30e958d _PyObject_Call_Prepend 
#108 0x00007ff3c3109150  
#109 0x00007ff3c30e917c _PyObject_MakeTpCall 
#110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#111 0x00007ff3c323c094  
#112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#113 0x00007ff3c30e958d _PyObject_Call_Prepend 
#114 0x00007ff3c3109150  
#115 0x00007ff3c30e917c _PyObject_MakeTpCall 
#116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#117 0x00007ff3c323c094  
#118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#119 0x00007ff3c30e958d _PyObject_Call_Prepend 
#120 0x00007ff3c3109150  
#121 0x00007ff3c30e917c _PyObject_MakeTpCall 
#122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#123 0x00007ff3c323c094  
#124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#125 0x00007ff3c30e958d _PyObject_Call_Prepend 
#126 0x00007ff3c3109150  
#127 0x00007ff3c30e917c _PyObject_MakeTpCall 
#128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#129 0x00007ff3c323c094  
#130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#131 0x00007ff3c30e958d _PyObject_Call_Prepend 
#132 0x00007ff3c3109150  
#133 0x00007ff3c30e917c _PyObject_MakeTpCall 
#134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#135 0x00007ff3c323c094  
#136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#137 0x00007ff3c30e958d _PyObject_Call_Prepend 
#138 0x00007ff3c3109150  
#139 0x00007ff3c30e917c _PyObject_MakeTpCall 
#140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#141 0x00007ff3c323c094  
#142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#143 0x00007ff3c30e958d _PyObject_Call_Prepend 
#144 0x00007ff3c3109150  
#145 0x00007ff3c30e917c _PyObject_MakeTpCall 
#146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#147 0x00007ff3c323c094  
#148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#149 0x00007ff3c30e958d _PyObject_Call_Prepend 
#150 0x00007ff3c3109150  
#151 0x00007ff3c30e917c _PyObject_MakeTpCall 
#152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#153 0x00007ff3c323c094  
#154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#155 0x00007ff3c30e958d _PyObject_Call_Prepend 
#156 0x00007ff3c3109150  
#157 0x00007ff3c30e917c _PyObject_MakeTpCall 
#158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#159 0x00007ff3c323c094  
#160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#161 0x00007ff3c30e958d _PyObject_Call_Prepend 
#162 0x00007ff3c3109150  
#163 0x00007ff3c30e917c _PyObject_MakeTpCall 
#164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#165 0x00007ff3c323c094  
#166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#167 0x00007ff3c30e958d _PyObject_Call_Prepend 
#168 0x00007ff3c3109150  
#169 0x00007ff3c30e917c _PyObject_MakeTpCall 
#170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#171 0x00007ff3c323c094  
#172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#173 0x00007ff3c30e958d _PyObject_Call_Prepend 
#174 0x00007ff3c3109150  
#175 0x00007ff3c30e917c _PyObject_MakeTpCall 
#176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#177 0x00007ff3c323c094  
#178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#179 0x00007ff3c30e958d _PyObject_Call_Prepend 
#180 0x00007ff3c3109150  
#181 0x00007ff3c30e917c _PyObject_MakeTpCall 
#182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#183 0x00007ff3c323c094  
#184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#185 0x00007ff3c30e958d _PyObject_Call_Prepend 
#186 0x00007ff3c3109150  
#187 0x00007ff3c30e917c _PyObject_MakeTpCall 
#188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#189 0x00007ff3c323c094  
#190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#191 0x00007ff3c323c094  
#192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate 
#193 0x00007ff3c30e958d _PyObject_Call_Prepend 
#194 0x00007ff3c3109150  
#195 0x00007ff3c30e917c _PyObject_MakeTpCall 
#196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault 
#197 0x00007ff3c323c094  
#198 0x00007ff3c317d0fd  
#199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#200 0x00007ff3c323c094  
#201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#202 0x00007ff3c323c094  
#203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#204 0x00007ff3c323c094  
#205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault 
#206 0x00007ff3c323c094  
#207 0x00007ff3c317d23c  
#208 0x00007ff3c31a7ec5  
#209 0x00007ff3c301ac77  
#210 0x00007ff3c357c573  
````

The fix implements two key changes.

1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race.

2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state.

The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently.  By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state.

Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants