Skip to content

fix(iast): fix crashes at teardown#18534

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
kowalski/fix-iast-fix-crashes-at-teardown
Jun 9, 2026
Merged

fix(iast): fix crashes at teardown#18534
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
kowalski/fix-iast-fix-crashes-at-teardown

Conversation

@KowalskiThomas

Copy link
Copy Markdown
Contributor

Description

This PR fixes crashes (caught by Crash Tracking) caused by IAST trying to access freed memories during interpreter shutdown.

is_text supposedly safely access objects but can only check whether the addresses make sense, it can't know whether objects have been freed.

inline bool
is_text(const PyObject* pyptr)
{
if (pyptr == nullptr) {
return false;
};
// Check that it's aligned correctly
if (reinterpret_cast<uintptr_t>(pyptr) % alignof(PyObject) != 0)
return false;
;
// Try to safely access ob_type
if (const PyObject* temp = pyptr; !temp->ob_type)
return false;
return PyUnicode_Check(pyptr) or PyBytes_Check(pyptr) or PyByteArray_Check(pyptr);
}

When uvloop's run_until_complete unwinds (Py_XDECREF in frames 57 to 61), custom type objects can be freed by the cyclic GC before all of their instances finish cleanup. An instance can still have a positive refcount, its pointer is non-null and aligned, ob_type is non-null, but the type it points to has been deallocated.

There's already a shutting_down guard in get_tainted_object_map_from_pyobject, but get_tainted_object_map calls is_text(obj) before reaching that.

// 1) Direct text objects
if (is_text(obj)) {
auto map = get_tainted_object_map_from_pyobject(obj);
if (map && !map->empty()) {
return map;
}
return nullptr;
}

Example crashing stack

Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0   0x000061dcfbfffe73 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2137)
#1   0x0000000000000000 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381)
#2   0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66)
#3   0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51)
#4   0x000074d4052a7de1 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381)
#5   0x000074d4052a7de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66)
#6   0x000074d4052a7de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51)
#7   0x000074d4052a7de1 TaintEngineContext::get_tainted_object_map(_object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:141)
#8   0x0000000000000000 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732)
#9   0x0000000000000000 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183)
#10  0x0000000000000000 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121)
#11  0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357)
#12  0x0000000000000000 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137)
#13  0x0000000000000000 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105)
#14  0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430)
#15  0x000074d4052a8a70 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732)
#16  0x000074d4052a8a70 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183)
#17  0x000074d4052a8a70 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121)
#18  0x000074d4052a8a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357)
#19  0x000074d4052a8a70 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137)
#20  0x000074d4052a8a70 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105)
#21  0x000074d4052a8a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430)
#22  0x000074d4052a8a70 pybind11::cpp_function::initialize<pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object)#5}, bool, pybind11::object, pybind11::name, pybind11::scope, pybind11::sibling>(pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object&&)#5}, bool (*)(pybind11::object), pybind11::name const, pybind11::scope&, pybind11::sibling)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .cold] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:400)
#23  0x000074d40525fd00 pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:1063)
#24  0x000061dcfbff87cf cfunction_call (/usr/src/python/Objects/methodobject.c:551)
#25  0x000061dcfbfe95de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245)
#26  0x000061dcfc010357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719)
#27  0x000061dcfc0d8a91 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:93)
#28  0x000061dcfbfcd894 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:269)
#29  0x000061dcfbfe9ad4 object_vacall (/usr/src/python/Objects/call.c:850)
#30  0x000061dcfc046d3e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:961)
#31  0x000074d40e7d4d0a WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024)
#32  0x000061dcfbfe95de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245)
#33  0x000061dcfc010357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719)
#34  0x000061dcfbfeca86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238)
#35  0x000074d40fdeadc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869)
#36  0x000074d40fdeb5a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188)
#37  0x0000000000000000 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431)
#38  0x000074d3f24978eb __Pyx_PyObject_Call (/project/uvloop/loop.c:191431)
#39  0x000074d3f24978eb __Pyx_PyObject_FastCallDict (/project/uvloop/loop.c:191552)
#40  0x000074d3f2573a69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66873)
#41  0x000074d3f257796b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975)
#42  0x000074d3f2571e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927)
#43  0x000074d3f2573c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335)
#44  0x0000000000000000 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33)
#45  0x000074d3f258f311 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33)
#46  0x000074d3f258f311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68)
#47  0x000074d3f258c647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:440)
#48  0x000074d3f24addb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18471)
#49  0x000074d3f2515e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876)
#50  0x0000000000000000 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528)
#51  0x000074d3f2526cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528)
#52  0x000074d3f2526cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331)
#53  0x000061dcfbfea47c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887)
#54  0x0000000000000000 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700)
#55  0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798)
#56  0x000074d3f252ad60 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700)
#57  0x000074d3f252ad60 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798)
#58  0x000074d3f252ad60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33769)
#59  0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797)
#60  0x000074d3f252c591 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797)
#61  0x000074d3f252c591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33322)
#62  0x000061dcfbfe98f8 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325)
#63  0x000061dcfc010357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719)
#64  0x000061dcfbfeca86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238)
#65  0x000074d40fdeadc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869)
#66  0x000074d40fdeb5a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188)
#67  0x000061dcfbfe95de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245)
#68  0x000061dcfbf6c77a context_run (/usr/src/python/Python/context.c:668)
#69  0x000061dcfc05bbeb cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:439)
#70  0x000061dcfbf62275 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3227)
#71  0x000061dcfc093679 PyEval_EvalCode (/usr/src/python/Python/ceval.c:579)
#72  0x000061dcfc0b12bc run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1723)
#73  0x000061dcfc0b1234 run_mod (/usr/src/python/Python/pythonrun.c:1744)
#74  0x000061dcfc0b0df1 pyrun_file (/usr/src/python/Python/pythonrun.c:1643)
#75  0x000061dcfc0b0c37 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433)
#76  0x000061dcfc0b0a57 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78)
#77  0x000061dcfc0bafb0 Py_RunMain (/usr/src/python/Modules/main.c:713)
#78  0x000061dcfc0bab3d Py_BytesMain (/usr/src/python/Modules/main.c:768)
#79  0x000074d411000d90 __libc_start_call_main (sysdeps/nptl/libc_start_call_main.h:58)
#80  0x0000000000000000 call_init (csu/libc-start.c:128)
#81  0x000074d411000e40 call_init (csu/libc-start.c:128)
#82  0x000074d411000e40 __libc_start_main_alias_2 (csu/libc-start.c:379)
#83  0x000061dcfc033075 _start 

@KowalskiThomas KowalskiThomas added the ASM Application Security Monitoring label Jun 9, 2026
@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 8 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741491-d2b8243-musllinux_1_2_x86_64, 1]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]   View in Datadog   GitLab

View all 8 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6786517 | Docs | Datadog PR Page | Give us feedback!

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jun 9, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp   @DataDog/asm-python
ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.h     @DataDog/asm-python
releasenotes/notes/iast-fix-crash-at-interpreter-teardown-7baa1217ec0d6595.yaml  @DataDog/apm-python

@KowalskiThomas KowalskiThomas force-pushed the kowalski/fix-iast-fix-crashes-at-teardown branch from db9500e to fb15bbd Compare June 9, 2026 10:31
@KowalskiThomas KowalskiThomas marked this pull request as ready for review June 9, 2026 10:40
@KowalskiThomas KowalskiThomas requested review from a team as code owners June 9, 2026 10:40

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fb15bbd5a0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@pr-commenter

pr-commenter Bot commented Jun 9, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-09 13:53:29

Comparing candidate commit 6786517 in PR branch kowalski/fix-iast-fix-crashes-at-teardown with baseline commit c641709 in branch main.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 82 metrics, 0 unstable metrics.

scenario:iast_aspects-re_expand_aspect

  • 🟥 execution_time [+251.142µs; +297.444µs] or [+7.027%; +8.323%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+89.483µs; +98.417µs] or [+20.900%; +22.987%]

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This change is marked for backport to 4.8, but it conflicts with that branch.
Attempting to cherrypick this change to 4.8 yielded the following error:

Auto-merging ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp
CONFLICT (content): Merge conflict in ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp
error: could not apply 65a2be06eb... Merge 7f3241c1ef70d1d24ff9ae2be7d12c3259bf68d0 into 8d5eed65f9536365f6fc469ad444eacea40c356f

The command used to test backporting was

git checkout 4.8 && git cherry-pick -x --mainline 1 65a2be06eb13c92c4b48753c13e6a8ab3fddbb32

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This change is marked for backport to 4.9, but it conflicts with that branch.
Attempting to cherrypick this change to 4.9 yielded the following error:

Auto-merging ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp
CONFLICT (content): Merge conflict in ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp
error: could not apply 65a2be06eb... Merge 7f3241c1ef70d1d24ff9ae2be7d12c3259bf68d0 into 8d5eed65f9536365f6fc469ad444eacea40c356f

The command used to test backporting was

git checkout 4.9 && git cherry-pick -x --mainline 1 65a2be06eb13c92c4b48753c13e6a8ab3fddbb32

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This change is marked for backport to 4.10, but it conflicts with that branch.
Attempting to cherrypick this change to 4.10 yielded the following error:

Auto-merging ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp
CONFLICT (content): Merge conflict in ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp
error: could not apply 65a2be06eb... Merge 7f3241c1ef70d1d24ff9ae2be7d12c3259bf68d0 into 8d5eed65f9536365f6fc469ad444eacea40c356f

The command used to test backporting was

git checkout 4.10 && git cherry-pick -x --mainline 1 65a2be06eb13c92c4b48753c13e6a8ab3fddbb32

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This change is marked for backport to 4.11 and it does not conflict with that branch.
The command used to test backporting was

git checkout 4.11 && git cherry-pick -x --mainline 1 65a2be06eb13c92c4b48753c13e6a8ab3fddbb32

Comment thread releasenotes/notes/iast-fix-crash-at-interpreter-teardown-7baa1217ec0d6595.yaml Outdated
Co-authored-by: Alberto Vara <alberto.vara@datadoghq.com>
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot merged commit 856135e into main Jun 9, 2026
850 of 854 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot deleted the kowalski/fix-iast-fix-crashes-at-teardown branch June 9, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants