fix(internal): detach threads if joinable before destroying [backport 4.5]#17336
Conversation
## Description We have a possible situation where we can end up with orphaned `ThreadRestartTimer` threads in fork-heavy applications. If a fork occurs while the current `ThreadRestartTimer` thread is in the stopping state (after `cls._instance = None` has already been called) then it will appear in `periodic_threads` and be added to `_threads_to_restart_after_fork` and then force restarted. ## Testing <!-- Describe your testing strategy or note what tests are included --> ## Risks <!-- Note any risks associated with this change, or "None" if no risks --> ## Additional Notes This bug is really really hard to reproduce and technically shouldn't be possible because the GIL should be held throughout the whole `os.fork()` call, but this is a code path which could cause some of the crashes we've seen in a fork-heavy application and is an easy defensive guard to add. Co-authored-by: juanjux <juanjo.alvarezmartinez@datadoghq.com> (cherry picked from commit 96352ec)
## Description This fixes a crash that can happen post-fork in fork-heavy workloads. ``` Error UnixSignal: Process terminated with SI_TKILL (SIGABRT) #0 0x00007ed5f6d869fc pthread_kill #1 0x00007ed5f6d32476 gsignal #2 0x00007ed5f6d187f3 abort #3 0x00007ed5f4fdf277 std::terminate #4 0x00007ed5c34737e6 PeriodicThread__after_fork #5 0x0000627d348a79b8 method_vectorcall_VARARGS_KEYWORDS (/usr/src/python/Objects/descrobject.c:365:14) #6 0x0000627d34847ea8 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #7 0x0000627d34847ea8 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325:12) #8 0x0000627d3486ec97 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19) #9 0x0000627d34848e86 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #10 0x0000627d34848e86 method_vectorcall (/usr/src/python/Objects/classobject.c:69:20) #11 0x0000627d34847a94 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #12 0x00007ed5c3474631 std::thread::_State_impl<std::thread::_Invoker<std::tuple<_PeriodicThread_do_start(periodic_thread*, bool)::{lambda()#1}> > >::_M_run #13 0x00007ed5c3474e10 execute_native_thread_routine ``` _See IR-50207 for more details._ Co-authored-by: brettlangdon <brett.langdon@datadoghq.com> (cherry picked from commit 56acdf4)
Codeowners resolved as |
Performance SLOsComparing candidate juanjux/kowalski/fix-internal-detach-threads-if-joinable-before-destroying-backport-4.5 (96394dd) with baseline 4.5 (8ecee12) 📈 Performance Regressions (2 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 103.664µs (SLO: <130.000µs 📉 -20.3%) vs baseline: +2.9% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +4.9% ✅ add_inplace_aspectTime: ✅ 102.078µs (SLO: <130.000µs 📉 -21.5%) vs baseline: +1.0% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +6.6% ✅ add_inplace_noaspectTime: ✅ 28.325µs (SLO: <40.000µs 📉 -29.2%) vs baseline: ~same Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.8% ✅ add_noaspectTime: ✅ 48.980µs (SLO: <70.000µs 📉 -30.0%) vs baseline: +0.5% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ bytearray_aspectTime: ✅ 250.106µs (SLO: <400.000µs 📉 -37.5%) vs baseline: -0.8% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.1% ✅ bytearray_extend_aspectTime: ✅ 642.787µs (SLO: <800.000µs 📉 -19.7%) vs baseline: -1.4% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.1% ✅ bytearray_extend_noaspectTime: ✅ 263.198µs (SLO: <400.000µs 📉 -34.2%) vs baseline: -0.4% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.8% ✅ bytearray_noaspectTime: ✅ 139.613µs (SLO: <300.000µs 📉 -53.5%) vs baseline: +0.9% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.0% ✅ bytes_aspectTime: ✅ 217.839µs (SLO: <300.000µs 📉 -27.4%) vs baseline: ~same Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.1% ✅ bytes_noaspectTime: ✅ 133.537µs (SLO: <200.000µs 📉 -33.2%) vs baseline: +0.1% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.7% ✅ bytesio_aspectTime: ✅ 3.766ms (SLO: <5.000ms 📉 -24.7%) vs baseline: -0.2% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ bytesio_noaspectTime: ✅ 315.874µs (SLO: <420.000µs 📉 -24.8%) vs baseline: +0.6% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.4% ✅ capitalize_aspectTime: ✅ 88.200µs (SLO: <300.000µs 📉 -70.6%) vs baseline: +0.1% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +6.0% ✅ capitalize_noaspectTime: ✅ 249.892µs (SLO: <300.000µs 📉 -16.7%) vs baseline: +1.7% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +5.0% ✅ casefold_aspectTime: ✅ 88.600µs (SLO: <500.000µs 📉 -82.3%) vs baseline: +0.2% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.5% ✅ casefold_noaspectTime: ✅ 302.724µs (SLO: <500.000µs 📉 -39.5%) vs baseline: -1.2% Memory: ✅ 42.880MB (SLO: <46.000MB -6.8%) vs baseline: +5.1% ✅ decode_aspectTime: ✅ 86.975µs (SLO: <100.000µs 📉 -13.0%) vs baseline: +0.1% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +5.0% ✅ decode_noaspectTime: ✅ 152.649µs (SLO: <210.000µs 📉 -27.3%) vs baseline: ~same Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.5% ✅ encode_aspectTime: ✅ 84.218µs (SLO: <200.000µs 📉 -57.9%) vs baseline: +0.2% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.2% ✅ encode_noaspectTime: ✅ 142.173µs (SLO: <200.000µs 📉 -28.9%) vs baseline: +0.9% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.1% ✅ format_aspectTime: ✅ 14.607ms (SLO: <19.200ms 📉 -23.9%) vs baseline: ~same Memory: ✅ 43.045MB (SLO: <46.000MB -6.4%) vs baseline: +4.9% ✅ format_map_aspectTime: ✅ 16.438ms (SLO: <21.500ms 📉 -23.5%) vs baseline: +0.7% Memory: ✅ 43.058MB (SLO: <46.000MB -6.4%) vs baseline: +5.8% ✅ format_map_noaspectTime: ✅ 369.941µs (SLO: <500.000µs 📉 -26.0%) vs baseline: -0.4% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.8% ✅ format_noaspectTime: ✅ 298.571µs (SLO: <500.000µs 📉 -40.3%) vs baseline: -1.6% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.0% ✅ index_aspectTime: ✅ 126.077µs (SLO: <300.000µs 📉 -58.0%) vs baseline: +2.9% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.1% ✅ index_noaspectTime: ✅ 40.353µs (SLO: <300.000µs 📉 -86.5%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.5% ✅ join_aspectTime: ✅ 206.700µs (SLO: <300.000µs 📉 -31.1%) vs baseline: -2.2% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ join_noaspectTime: ✅ 144.804µs (SLO: <300.000µs 📉 -51.7%) vs baseline: +1.6% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.0% ✅ ljust_aspectTime: ✅ 568.842µs (SLO: <700.000µs 📉 -18.7%) vs baseline: 📈 +14.5% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.7% ✅ ljust_noaspectTime: ✅ 254.566µs (SLO: <300.000µs 📉 -15.1%) vs baseline: -0.4% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.4% ✅ lower_aspectTime: ✅ 289.725µs (SLO: <500.000µs 📉 -42.1%) vs baseline: -2.0% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ lower_noaspectTime: ✅ 228.717µs (SLO: <300.000µs 📉 -23.8%) vs baseline: -1.7% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.0% ✅ lstrip_aspectTime: ✅ 0.268ms (SLO: <3.000ms 📉 -91.1%) vs baseline: -2.1% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.0% ✅ lstrip_noaspectTime: ✅ 0.172ms (SLO: <3.000ms 📉 -94.3%) vs baseline: -3.2% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.1% ✅ modulo_aspectTime: ✅ 14.264ms (SLO: <18.750ms 📉 -23.9%) vs baseline: +0.2% Memory: ✅ 43.070MB (SLO: <46.000MB -6.4%) vs baseline: +4.9% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 14.809ms (SLO: <19.350ms 📉 -23.5%) vs baseline: ~same Memory: ✅ 43.110MB (SLO: <46.000MB -6.3%) vs baseline: +5.2% ✅ modulo_aspect_for_bytesTime: ✅ 14.478ms (SLO: <18.900ms 📉 -23.4%) vs baseline: +0.8% Memory: ✅ 42.995MB (SLO: <46.000MB -6.5%) vs baseline: +4.8% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 14.587ms (SLO: <19.150ms 📉 -23.8%) vs baseline: -0.4% Memory: ✅ 43.100MB (SLO: <46.000MB -6.3%) vs baseline: +5.1% ✅ modulo_noaspectTime: ✅ 0.356ms (SLO: <3.000ms 📉 -88.1%) vs baseline: -0.4% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.2% ✅ replace_aspectTime: ✅ 18.411ms (SLO: <24.000ms 📉 -23.3%) vs baseline: +0.1% Memory: ✅ 43.059MB (SLO: <46.000MB -6.4%) vs baseline: +4.9% ✅ replace_noaspectTime: ✅ 275.749µs (SLO: <300.000µs -8.1%) vs baseline: -1.8% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.0% ✅ repr_aspectTime: ✅ 313.938µs (SLO: <420.000µs 📉 -25.3%) vs baseline: -1.7% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.2% ✅ repr_noaspectTime: ✅ 46.587µs (SLO: <90.000µs 📉 -48.2%) vs baseline: -0.5% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ rstrip_aspectTime: ✅ 380.984µs (SLO: <500.000µs 📉 -23.8%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ rstrip_noaspectTime: ✅ 179.717µs (SLO: <300.000µs 📉 -40.1%) vs baseline: +0.6% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.0% ✅ slice_aspectTime: ✅ 179.975µs (SLO: <300.000µs 📉 -40.0%) vs baseline: -2.3% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ slice_noaspectTime: ✅ 54.030µs (SLO: <90.000µs 📉 -40.0%) vs baseline: -0.2% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +5.2% ✅ stringio_aspectTime: ✅ 4.388ms (SLO: <5.000ms 📉 -12.2%) vs baseline: 📈 +16.0% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.1% ✅ stringio_noaspectTime: ✅ 345.162µs (SLO: <500.000µs 📉 -31.0%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.1% ✅ strip_aspectTime: ✅ 265.926µs (SLO: <350.000µs 📉 -24.0%) vs baseline: -2.4% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 172.805µs (SLO: <240.000µs 📉 -28.0%) vs baseline: -3.4% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +5.2% ✅ swapcase_aspectTime: ✅ 328.365µs (SLO: <500.000µs 📉 -34.3%) vs baseline: -1.8% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.2% ✅ swapcase_noaspectTime: ✅ 268.112µs (SLO: <400.000µs 📉 -33.0%) vs baseline: ~same Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.9% ✅ title_aspectTime: ✅ 313.311µs (SLO: <500.000µs 📉 -37.3%) vs baseline: -1.5% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +5.2% ✅ title_noaspectTime: ✅ 252.529µs (SLO: <400.000µs 📉 -36.9%) vs baseline: -0.3% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.1% ✅ translate_aspectTime: ✅ 490.820µs (SLO: <700.000µs 📉 -29.9%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.2% ✅ translate_noaspectTime: ✅ 422.669µs (SLO: <500.000µs 📉 -15.5%) vs baseline: -0.5% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +5.0% ✅ upper_aspectTime: ✅ 294.084µs (SLO: <500.000µs 📉 -41.2%) vs baseline: -0.3% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.2% ✅ upper_noaspectTime: ✅ 230.561µs (SLO: <400.000µs 📉 -42.4%) vs baseline: -0.3% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.1% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 510.173µs (SLO: <700.000µs 📉 -27.1%) vs baseline: 📈 +19.8% Memory: ✅ 42.546MB (SLO: <46.000MB -7.5%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 424.486µs (SLO: <700.000µs 📉 -39.4%) vs baseline: -1.6% Memory: ✅ 42.566MB (SLO: <46.000MB -7.5%) vs baseline: +5.0% ✅ ospathjoin_aspectTime: ✅ 616.158µs (SLO: <700.000µs 📉 -12.0%) vs baseline: -0.8% Memory: ✅ 42.566MB (SLO: <46.000MB -7.5%) vs baseline: +5.1% ✅ ospathjoin_noaspectTime: ✅ 627.134µs (SLO: <700.000µs 📉 -10.4%) vs baseline: -0.4% Memory: ✅ 42.625MB (SLO: <46.000MB -7.3%) vs baseline: +5.1% ✅ ospathnormcase_aspectTime: ✅ 348.622µs (SLO: <700.000µs 📉 -50.2%) vs baseline: -0.8% Memory: ✅ 42.585MB (SLO: <46.000MB -7.4%) vs baseline: +5.1% ✅ ospathnormcase_noaspectTime: ✅ 355.667µs (SLO: <700.000µs 📉 -49.2%) vs baseline: +0.1% Memory: ✅ 42.644MB (SLO: <46.000MB -7.3%) vs baseline: +5.3% ✅ ospathsplit_aspectTime: ✅ 483.643µs (SLO: <700.000µs 📉 -30.9%) vs baseline: -0.5% Memory: ✅ 42.546MB (SLO: <46.000MB -7.5%) vs baseline: +5.1% ✅ ospathsplit_noaspectTime: ✅ 494.293µs (SLO: <700.000µs 📉 -29.4%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <46.000MB -7.4%) vs baseline: +5.1% ✅ ospathsplitdrive_aspectTime: ✅ 373.789µs (SLO: <700.000µs 📉 -46.6%) vs baseline: -0.8% Memory: ✅ 42.566MB (SLO: <46.000MB -7.5%) vs baseline: +5.0% ✅ ospathsplitdrive_noaspectTime: ✅ 72.502µs (SLO: <700.000µs 📉 -89.6%) vs baseline: -2.1% Memory: ✅ 42.625MB (SLO: <46.000MB -7.3%) vs baseline: +5.2% ✅ ospathsplitext_aspectTime: ✅ 456.639µs (SLO: <700.000µs 📉 -34.8%) vs baseline: -0.3% Memory: ✅ 42.605MB (SLO: <46.000MB -7.4%) vs baseline: +5.1% ✅ ospathsplitext_noaspectTime: ✅ 460.218µs (SLO: <700.000µs 📉 -34.3%) vs baseline: -0.8% Memory: ✅ 42.585MB (SLO: <46.000MB -7.4%) vs baseline: +5.0% 🟡 Near SLO Breach (2 suites)🟡 packagespackageforrootmodulemapping - 4/4✅ cache_offTime: ✅ 349.261ms (SLO: <354.300ms 🟡 -1.4%) vs baseline: -1.1% Memory: ✅ 42.505MB (SLO: <46.000MB -7.6%) vs baseline: +6.1% ✅ cache_onTime: ✅ 0.388µs (SLO: <10.000µs 📉 -96.1%) vs baseline: -0.9% Memory: ✅ 40.441MB (SLO: <46.000MB 📉 -12.1%) vs baseline: +4.9% 🟡 tracer - 6/6✅ largeTime: ✅ 31.901ms (SLO: <32.950ms -3.2%) vs baseline: +0.7% Memory: ✅ 36.726MB (SLO: <39.250MB -6.4%) vs baseline: +5.0% ✅ mediumTime: ✅ 3.135ms (SLO: <3.200ms -2.0%) vs baseline: +0.8% Memory: ✅ 35.252MB (SLO: <38.750MB -9.0%) vs baseline: +5.3% ✅ smallTime: ✅ 366.586µs (SLO: <370.000µs 🟡 -0.9%) vs baseline: +4.0% Memory: ✅ 35.232MB (SLO: <38.750MB -9.1%) vs baseline: +4.9%
|
…if-joinable-before-destroying-backport-4.5
…if-joinable-before-destroying-backport-4.5
…if-joinable-before-destroying-backport-4.5
…if-joinable-before-destroying-backport-4.5
…if-joinable-before-destroying-backport-4.5
…if-joinable-before-destroying-backport-4.5
Backport of #17297