fix(iast): avoid crash due to missing INCREF [backport 4.7]#17642
Conversation
## Description This PR fixes a crash coming from IAST due to an inconsistent reference count contract between `new_pyobject_id` and its callers, where the callers would expect a new owned reference like it [already does today](https://github.com/DataDog/dd-trace-py/blob/c02775f9db03c05f90356181323d000b86aba7da/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.cpp#L169-L171) but some code paths were missing the `Py_INCREF`, causing segmentation faults (see [example usage](https://github.com/DataDog/dd-trace-py/blob/c02775f9db03c05f90356181323d000b86aba7da/ddtrace/appsec/_iast/_taint_tracking/aspects/aspect_operator_add.cpp#L30-L31)). This error has been around at least since 3.11.0 and is currently causing approximately [50k errors per week](https://app.datadoghq.com/error-tracking/issue/01522162-6bf3-11f0-b96b-da7ad0900002?query=%28%40tags.severity%3Acrash%20OR%20severity%3Acrash%20OR%20signum%3A%2A%20OR%20%40error.is_crash%3Atrue%29%20%40lib_language%3Apython&index=&tb=%40org_id&from_ts=1775841064700&to_ts=1776445864700&live=true). ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x000061335a8b72d4 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2126:1) #1 0x000061335a89e11c PyObject_TypeCheck (/usr/src/python/./Include/object.h:381:36) #2 0x000061335a89e11c object_isinstance (/usr/src/python/Objects/abstract.c:2571:18) #3 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2606:16) #4 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2628:17) #5 0x000061335a89cbeb object_recursive_isinstance (/usr/src/python/Objects/abstract.c:2602:1) #6 0x000061335a89cbeb PyObject_IsInstance (/usr/src/python/Objects/abstract.c:2670:12) #7 0x000061335a8c89ed _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3036:26) #8 0x000061335a98fd11 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #9 0x000061335a884945 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:267:11) #10 0x000061335a8a0bf4 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #11 0x000061335a8a0bf4 object_vacall (/usr/src/python/Objects/call.c:850:14) #12 0x000061335a8fdf8e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:957:14) #13 0x000074f48affeb28 WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024:18) #14 0x000061335a8a12e2 PyObject_Call #15 0x000061335a8cb1a9 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #16 0x000061335a8a3ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16) #17 0x000061335a8a3ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14) #18 0x000074f48d48bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22) #19 0x000074f48d48c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11) #20 0x000061335a8af877 cfunction_vectorcall_O (/usr/src/python/Objects/methodobject.c:509:24) #21 0x000074f48a434f69 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431:15) #22 0x000074f48a434f69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66901:25) #23 0x000074f48a43a96b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975:25) #24 0x000074f48a434e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927:24) #25 0x000074f48a436c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335:19) #26 0x000074f48a452311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68:1) #27 0x000074f48a44f647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:439:5) #28 0x000074f48a370db5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18458:23) #29 0x000074f48a3d8e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876:18) #30 0x000074f48a3e9cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528:18) #31 0x000074f48a3e9cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331:13) #32 0x000061335a8a159c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #33 0x000061335a8a159c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887:24) #34 0x000074f48a3edd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33768:23) #35 0x000074f48a3ef591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33318:13) #36 0x000061335a8a0a18 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #37 0x000061335a8a0a18 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325:12) #38 0x000061335a8c7807 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19) #39 0x000061335a8a3ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16) #40 0x000061335a8a3ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14) #41 0x000074f48d48bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22) #42 0x000074f48d48c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11) #43 0x000061335a8a06fe _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:240:18) #44 0x000061335a82380c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:90:16) #45 0x000061335a82380c context_run (/usr/src/python/Python/context.c:668:29) #46 0x000061335a912d7b cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:438:24) #47 0x000061335a8cb1a9 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #48 0x000061335a94a4b9 PyEval_EvalCode (/usr/src/python/Python/ceval.c:578:21) #49 0x000061335a96852c run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1722:9) #50 0x000061335a9684a4 run_mod (/usr/src/python/Python/pythonrun.c:1743:19) #51 0x000061335a968061 pyrun_file (/usr/src/python/Python/pythonrun.c:1643:15) #52 0x000061335a967ea7 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433:13) #53 0x000061335a967cc7 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78:15) #54 0x000061335a972230 pymain_run_file_obj (/usr/src/python/Modules/main.c:360:15) #55 0x000061335a972230 pymain_run_file (/usr/src/python/Modules/main.c:379:15) #56 0x000061335a972230 pymain_run_python (/usr/src/python/Modules/main.c:633:21) #57 0x000061335a972230 Py_RunMain (/usr/src/python/Modules/main.c:713:5) #58 0x000061335a971dbd Py_BytesMain (/usr/src/python/Modules/main.c:767:12) #59 0x000074f48e000e40 __libc_start_main #60 0x000061335a8ea2d5 _start ``` Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com> (cherry picked from commit 36bf68b) Co-authored-by: Thomas Kowalski <thomas.kowalski@datadoghq.com>
Codeowners resolved as |
Performance SLOsComparing candidate backport-17605-to-4.7 (91282ad) with baseline 4.7 (bc0b0f4) 📈 Performance Regressions (1 suite)📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 516.126µs (SLO: <700.000µs 📉 -26.3%) vs baseline: 📈 +20.8% Memory: ✅ 43.531MB (SLO: <46.000MB -5.4%) vs baseline: +4.6% ✅ ospathbasename_noaspectTime: ✅ 430.705µs (SLO: <700.000µs 📉 -38.5%) vs baseline: -0.6% Memory: ✅ 43.572MB (SLO: <46.000MB -5.3%) vs baseline: +4.6% ✅ ospathjoin_aspectTime: ✅ 632.815µs (SLO: <700.000µs -9.6%) vs baseline: +0.5% Memory: ✅ 43.552MB (SLO: <46.000MB -5.3%) vs baseline: +4.5% ✅ ospathjoin_noaspectTime: ✅ 631.101µs (SLO: <700.000µs -9.8%) vs baseline: -0.1% Memory: ✅ 43.448MB (SLO: <46.000MB -5.5%) vs baseline: +4.1% ✅ ospathnormcase_aspectTime: ✅ 353.622µs (SLO: <700.000µs 📉 -49.5%) vs baseline: +0.6% Memory: ✅ 43.672MB (SLO: <46.000MB -5.1%) vs baseline: +5.0% ✅ ospathnormcase_noaspectTime: ✅ 362.177µs (SLO: <700.000µs 📉 -48.3%) vs baseline: -0.5% Memory: ✅ 43.823MB (SLO: <46.000MB -4.7%) vs baseline: +5.7% ✅ ospathsplit_aspectTime: ✅ 495.480µs (SLO: <700.000µs 📉 -29.2%) vs baseline: +0.4% Memory: ✅ 43.665MB (SLO: <46.000MB -5.1%) vs baseline: +5.0% ✅ ospathsplit_noaspectTime: ✅ 502.463µs (SLO: <700.000µs 📉 -28.2%) vs baseline: +0.2% Memory: ✅ 43.700MB (SLO: <46.000MB -5.0%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 371.338µs (SLO: <700.000µs 📉 -47.0%) vs baseline: -0.6% Memory: ✅ 43.742MB (SLO: <46.000MB -4.9%) vs baseline: +5.0% ✅ ospathsplitdrive_noaspectTime: ✅ 73.586µs (SLO: <700.000µs 📉 -89.5%) vs baseline: +1.9% Memory: ✅ 43.631MB (SLO: <46.000MB -5.1%) vs baseline: +4.9% ✅ ospathsplitext_aspectTime: ✅ 460.894µs (SLO: <700.000µs 📉 -34.2%) vs baseline: -0.6% Memory: ✅ 43.687MB (SLO: <46.000MB -5.0%) vs baseline: +4.9% ✅ ospathsplitext_noaspectTime: ✅ 469.158µs (SLO: <700.000µs 📉 -33.0%) vs baseline: +1.0% Memory: ✅ 43.786MB (SLO: <46.000MB -4.8%) vs baseline: +5.1% ✅ All Tests Passing (2 suites)✅ iastaspectssplit - 12/12✅ rsplit_aspectTime: ✅ 154.927µs (SLO: <250.000µs 📉 -38.0%) vs baseline: +2.2% Memory: ✅ 43.454MB (SLO: <46.000MB -5.5%) vs baseline: +4.4% ✅ rsplit_noaspectTime: ✅ 161.017µs (SLO: <250.000µs 📉 -35.6%) vs baseline: -0.9% Memory: ✅ 43.552MB (SLO: <46.000MB -5.3%) vs baseline: +4.8% ✅ split_aspectTime: ✅ 150.572µs (SLO: <250.000µs 📉 -39.8%) vs baseline: +0.1% Memory: ✅ 43.554MB (SLO: <46.000MB -5.3%) vs baseline: +4.8% ✅ split_noaspectTime: ✅ 159.234µs (SLO: <250.000µs 📉 -36.3%) vs baseline: -1.0% Memory: ✅ 43.605MB (SLO: <46.000MB -5.2%) vs baseline: +4.6% ✅ splitlines_aspectTime: ✅ 151.522µs (SLO: <250.000µs 📉 -39.4%) vs baseline: -0.2% Memory: ✅ 43.471MB (SLO: <46.000MB -5.5%) vs baseline: +4.6% ✅ splitlines_noaspectTime: ✅ 157.465µs (SLO: <250.000µs 📉 -37.0%) vs baseline: +0.4% Memory: ✅ 43.504MB (SLO: <46.000MB -5.4%) vs baseline: +4.6% ✅ iastpropagation - 8/8✅ no-propagationTime: ✅ 48.722µs (SLO: <60.000µs 📉 -18.8%) vs baseline: +0.5% Memory: ✅ 39.951MB (SLO: <42.000MB -4.9%) vs baseline: +4.5% ✅ propagation_enabledTime: ✅ 135.425µs (SLO: <190.000µs 📉 -28.7%) vs baseline: -0.1% Memory: ✅ 40.010MB (SLO: <42.000MB -4.7%) vs baseline: +4.9% ✅ propagation_enabled_100Time: ✅ 1.568ms (SLO: <2.300ms 📉 -31.8%) vs baseline: +0.6% Memory: ✅ 40.029MB (SLO: <42.000MB -4.7%) vs baseline: +4.5% ✅ propagation_enabled_1000Time: ✅ 28.933ms (SLO: <34.550ms 📉 -16.3%) vs baseline: ~same Memory: ✅ 40.069MB (SLO: <42.000MB -4.6%) vs baseline: +5.0% ℹ️ Scenarios Missing SLO Configuration (20 scenarios)The following scenarios exist in candidate data but have no SLO thresholds configured:
|
Backport #17605 to 4.7