fix(profiling): fix crash due to Frame cache eviction [backport 4.6]#17599
Conversation
https://datadoghq.atlassian.net/browse/PROF-13112 This fixes a crash in the Profiler codebase. The crash was caused by `ThreadInfo::unwind_task` trying to use `Frame` objects that it had a reference to (`Frame::Ref`) while those same Frames were being evicted from the cache, which can happen under high load. ``` Error UnixSignal: Process terminated with SI_KERNEL (SIGSEGV) ``` The proposed fix is to change `FrameStack` to extend `std::deque<Frame>` instead of `std::deque<Frame::Ref>`, which means pushing frames to `FrameStack` actually copies them and eliminates that race condition. This should be OK performance-wise because `Frame` is tiny and only has trivial fields to copy (no memory allocations, only integral types). Another possible fix would be to "lock" certain `Frame` objects in the cache (to prevent them from being evicted while any Task using them is being unwound) but this would be much more complicated to get right and although it would lead to less copies, the performance picture is blurry for more subtle reasons. So I suggest we don't go ahead with the alternative unless we absolutely need to (which we don't seem to). Running DoE with this change on the Enterprise archetype with asyncio/FastAPI (n=10) shows no significant difference (either in memory or CPU usage / latency), so I think this is safe to merge performance-wise. <details> | Commit | Run | p50 (ms) | p99 (ms) | CPU% | Mem (MB) | |--------|-----|----------|----------|------|----------| | `90de7913` (after) | 1 | 150.43 | 152.88 | 172.68 | 274.9 | | `90de7913` (after) | 2 | 150.45 | 152.38 | 172.17 | 266.2 | | `90de7913` (after) | 3 | 150.43 | 153.91 | 172.92 | 266.2 | | `90de7913` (after) | 4 | 150.46 | 154.16 | 172.83 | 265.8 | | `90de7913` (after) | 5 | 150.44 | 153.01 | 172.80 | 267.4 | | `90de7913` (after) | 6 | 150.43 | 153.28 | 172.70 | 265.8 | | `90de7913` (after) | 7 | 150.54 | 157.48 | 173.32 | 265.8 | | `90de7913` (after) | 8 | 150.51 | 156.61 | 172.18 | 265.5 | | `90de7913` (after) | 9 | 150.53 | 157.05 | 172.18 | 265.8 | | `90de7913` (after) | 10 | 150.45 | 152.84 | 172.08 | 265.5 | | `c26e6181` (before) | 1 | 150.51 | 152.80 | 173.63 | 265.5 | | `c26e6181` (before) | 2 | 150.48 | 152.41 | 173.49 | 265.4 | | `c26e6181` (before) | 3 | 150.47 | 156.31 | 172.53 | 265.9 | | `c26e6181` (before) | 4 | 150.49 | 152.60 | 172.86 | 265.5 | | `c26e6181` (before) | 5 | 150.40 | 152.86 | 172.99 | 266.0 | | `c26e6181` (before) | 6 | 150.46 | 152.66 | 172.53 | 265.5 | | `c26e6181` (before) | 7 | 150.49 | 154.91 | 172.74 | 265.3 | | `c26e6181` (before) | 8 | 150.46 | 152.46 | 171.93 | 265.7 | | `c26e6181` (before) | 9 | 150.52 | 156.70 | 172.96 | 265.7 | | `c26e6181` (before) | 10 | 150.43 | 153.02 | 172.52 | 265.8 | | Version | p50 (ms) | p99 (ms) | CPU% | Mem (MB) | |---------|----------|----------|------|----------| | `c26e6181` (before) | 150.470 ± 0.034 | 153.67 ± 1.66 | 172.82 ± 0.50 | 265.6 ± 0.2 | | `90de7913` (after) | 150.465 ± 0.043 | 154.36 ± 1.94 | 172.59 ± 0.41 | 266.9 ± 2.9 | | **Δ (after − before)** | **−0.005ms (0.00%)** | **+0.69ms (+0.45%)** | **−0.23pp (−0.13%)** | **+1.2MB (+0.47%)** | </details> Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>
Performance SLOsComparing candidate juanjux/backport-17456-4.6 (adeedd0) with baseline 4.6 (ad4587e) 📈 Performance Regressions (2 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 108.317µs (SLO: <130.000µs 📉 -16.7%) vs baseline: +4.5% Memory: ✅ 43.624MB (SLO: <46.000MB -5.2%) vs baseline: +5.3% ✅ add_inplace_aspectTime: ✅ 105.181µs (SLO: <130.000µs 📉 -19.1%) vs baseline: +0.2% Memory: ✅ 43.709MB (SLO: <46.000MB -5.0%) vs baseline: +5.0% ✅ add_inplace_noaspectTime: ✅ 28.196µs (SLO: <40.000µs 📉 -29.5%) vs baseline: ~same Memory: ✅ 43.668MB (SLO: <46.000MB -5.1%) vs baseline: +5.1% ✅ add_noaspectTime: ✅ 48.908µs (SLO: <70.000µs 📉 -30.1%) vs baseline: ~same Memory: ✅ 43.591MB (SLO: <46.000MB -5.2%) vs baseline: +4.8% ✅ bytearray_aspectTime: ✅ 249.960µs (SLO: <400.000µs 📉 -37.5%) vs baseline: -0.4% Memory: ✅ 43.722MB (SLO: <46.000MB -5.0%) vs baseline: +5.1% ✅ bytearray_extend_aspectTime: ✅ 642.599µs (SLO: <800.000µs 📉 -19.7%) vs baseline: ~same Memory: ✅ 43.619MB (SLO: <46.000MB -5.2%) vs baseline: +4.9% ✅ bytearray_extend_noaspectTime: ✅ 264.029µs (SLO: <400.000µs 📉 -34.0%) vs baseline: +0.9% Memory: ✅ 43.697MB (SLO: <46.000MB -5.0%) vs baseline: +5.0% ✅ bytearray_noaspectTime: ✅ 136.081µs (SLO: <300.000µs 📉 -54.6%) vs baseline: -0.1% Memory: ✅ 43.620MB (SLO: <46.000MB -5.2%) vs baseline: +4.6% ✅ bytes_aspectTime: ✅ 218.870µs (SLO: <300.000µs 📉 -27.0%) vs baseline: +1.4% Memory: ✅ 43.629MB (SLO: <46.000MB -5.2%) vs baseline: +5.3% ✅ bytes_noaspectTime: ✅ 132.953µs (SLO: <200.000µs 📉 -33.5%) vs baseline: -0.2% Memory: ✅ 43.646MB (SLO: <46.000MB -5.1%) vs baseline: +4.9% ✅ bytesio_aspectTime: ✅ 3.735ms (SLO: <5.000ms 📉 -25.3%) vs baseline: ~same Memory: ✅ 43.730MB (SLO: <46.000MB -4.9%) vs baseline: +5.0% ✅ bytesio_noaspectTime: ✅ 314.317µs (SLO: <420.000µs 📉 -25.2%) vs baseline: +0.3% Memory: ✅ 43.560MB (SLO: <46.000MB -5.3%) vs baseline: +4.6% ✅ capitalize_aspectTime: ✅ 88.590µs (SLO: <300.000µs 📉 -70.5%) vs baseline: +0.8% Memory: ✅ 43.558MB (SLO: <46.000MB -5.3%) vs baseline: +4.8% ✅ capitalize_noaspectTime: ✅ 249.267µs (SLO: <300.000µs 📉 -16.9%) vs baseline: -0.8% Memory: ✅ 43.636MB (SLO: <46.000MB -5.1%) vs baseline: +4.8% ✅ casefold_aspectTime: ✅ 91.611µs (SLO: <500.000µs 📉 -81.7%) vs baseline: +4.0% Memory: ✅ 43.533MB (SLO: <46.000MB -5.4%) vs baseline: +4.5% ✅ casefold_noaspectTime: ✅ 301.149µs (SLO: <500.000µs 📉 -39.8%) vs baseline: -0.9% Memory: ✅ 43.654MB (SLO: <46.000MB -5.1%) vs baseline: +5.1% ✅ decode_aspectTime: ✅ 86.787µs (SLO: <100.000µs 📉 -13.2%) vs baseline: +0.3% Memory: ✅ 43.535MB (SLO: <46.000MB -5.4%) vs baseline: +4.5% ✅ decode_noaspectTime: ✅ 152.060µs (SLO: <210.000µs 📉 -27.6%) vs baseline: ~same Memory: ✅ 43.623MB (SLO: <46.000MB -5.2%) vs baseline: +4.7% ✅ encode_aspectTime: ✅ 84.276µs (SLO: <200.000µs 📉 -57.9%) vs baseline: +0.5% Memory: ✅ 43.716MB (SLO: <46.000MB -5.0%) vs baseline: +5.3% ✅ encode_noaspectTime: ✅ 136.847µs (SLO: <200.000µs 📉 -31.6%) vs baseline: +0.1% Memory: ✅ 43.664MB (SLO: <46.000MB -5.1%) vs baseline: +5.0% ✅ format_aspectTime: ✅ 14.622ms (SLO: <19.200ms 📉 -23.8%) vs baseline: +0.1% Memory: ✅ 43.745MB (SLO: <46.000MB -4.9%) vs baseline: +4.9% ✅ format_map_aspectTime: ✅ 16.434ms (SLO: <21.500ms 📉 -23.6%) vs baseline: -0.1% Memory: ✅ 43.769MB (SLO: <46.000MB -4.8%) vs baseline: +5.0% ✅ format_map_noaspectTime: ✅ 369.158µs (SLO: <500.000µs 📉 -26.2%) vs baseline: +0.3% Memory: ✅ 43.650MB (SLO: <46.000MB -5.1%) vs baseline: +4.7% ✅ format_noaspectTime: ✅ 302.255µs (SLO: <500.000µs 📉 -39.5%) vs baseline: ~same Memory: ✅ 43.619MB (SLO: <46.000MB -5.2%) vs baseline: +4.9% ✅ index_aspectTime: ✅ 124.177µs (SLO: <300.000µs 📉 -58.6%) vs baseline: +1.8% Memory: ✅ 43.704MB (SLO: <46.000MB -5.0%) vs baseline: +5.4% ✅ index_noaspectTime: ✅ 40.426µs (SLO: <300.000µs 📉 -86.5%) vs baseline: -0.4% Memory: ✅ 43.696MB (SLO: <46.000MB -5.0%) vs baseline: +5.0% ✅ join_aspectTime: ✅ 212.533µs (SLO: <300.000µs 📉 -29.2%) vs baseline: +0.1% Memory: ✅ 43.635MB (SLO: <46.000MB -5.1%) vs baseline: +5.3% ✅ join_noaspectTime: ✅ 140.052µs (SLO: <300.000µs 📉 -53.3%) vs baseline: -0.2% Memory: ✅ 43.669MB (SLO: <46.000MB -5.1%) vs baseline: +4.8% ✅ ljust_aspectTime: ✅ 498.294µs (SLO: <700.000µs 📉 -28.8%) vs baseline: ~same Memory: ✅ 43.702MB (SLO: <46.000MB -5.0%) vs baseline: +5.2% ✅ ljust_noaspectTime: ✅ 256.507µs (SLO: <300.000µs 📉 -14.5%) vs baseline: +1.1% Memory: ✅ 43.617MB (SLO: <46.000MB -5.2%) vs baseline: +4.9% ✅ lower_aspectTime: ✅ 291.879µs (SLO: <500.000µs 📉 -41.6%) vs baseline: -0.4% Memory: ✅ 43.523MB (SLO: <46.000MB -5.4%) vs baseline: +5.3% ✅ lower_noaspectTime: ✅ 233.349µs (SLO: <300.000µs 📉 -22.2%) vs baseline: -1.6% Memory: ✅ 43.687MB (SLO: <46.000MB -5.0%) vs baseline: +4.8% ✅ lstrip_aspectTime: ✅ 0.312ms (SLO: <3.000ms 📉 -89.6%) vs baseline: 📈 +15.7% Memory: ✅ 43.681MB (SLO: <46.000MB -5.0%) vs baseline: +4.8% ✅ lstrip_noaspectTime: ✅ 0.175ms (SLO: <3.000ms 📉 -94.2%) vs baseline: -0.4% Memory: ✅ 43.636MB (SLO: <46.000MB -5.1%) vs baseline: +4.7% ✅ modulo_aspectTime: ✅ 14.284ms (SLO: <18.750ms 📉 -23.8%) vs baseline: ~same Memory: ✅ 43.741MB (SLO: <46.000MB -4.9%) vs baseline: +5.1% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 15.040ms (SLO: <19.350ms 📉 -22.3%) vs baseline: +2.0% Memory: ✅ 43.569MB (SLO: <46.000MB -5.3%) vs baseline: +4.9% ✅ modulo_aspect_for_bytesTime: ✅ 14.389ms (SLO: <18.900ms 📉 -23.9%) vs baseline: +0.1% Memory: ✅ 43.690MB (SLO: <46.000MB -5.0%) vs baseline: +4.4% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 14.655ms (SLO: <19.150ms 📉 -23.5%) vs baseline: ~same Memory: ✅ 43.751MB (SLO: <46.000MB -4.9%) vs baseline: +4.7% ✅ modulo_noaspectTime: ✅ 0.368ms (SLO: <3.000ms 📉 -87.7%) vs baseline: +1.9% Memory: ✅ 43.601MB (SLO: <46.000MB -5.2%) vs baseline: +5.0% ✅ replace_aspectTime: ✅ 18.419ms (SLO: <24.000ms 📉 -23.3%) vs baseline: +0.2% Memory: ✅ 43.667MB (SLO: <46.000MB -5.1%) vs baseline: +4.5% ✅ replace_noaspectTime: ✅ 281.438µs (SLO: <300.000µs -6.2%) vs baseline: +1.1% Memory: ✅ 43.655MB (SLO: <46.000MB -5.1%) vs baseline: +4.9% ✅ repr_aspectTime: ✅ 308.732µs (SLO: <420.000µs 📉 -26.5%) vs baseline: -1.4% Memory: ✅ 43.610MB (SLO: <46.000MB -5.2%) vs baseline: +5.1% ✅ repr_noaspectTime: ✅ 46.612µs (SLO: <90.000µs 📉 -48.2%) vs baseline: -0.6% Memory: ✅ 43.552MB (SLO: <46.000MB -5.3%) vs baseline: +4.9% ✅ rstrip_aspectTime: ✅ 388.763µs (SLO: <500.000µs 📉 -22.2%) vs baseline: +1.6% Memory: ✅ 43.696MB (SLO: <46.000MB -5.0%) vs baseline: +4.8% ✅ rstrip_noaspectTime: ✅ 180.601µs (SLO: <300.000µs 📉 -39.8%) vs baseline: +0.3% Memory: ✅ 43.588MB (SLO: <46.000MB -5.2%) vs baseline: +4.8% ✅ slice_aspectTime: ✅ 184.126µs (SLO: <300.000µs 📉 -38.6%) vs baseline: +0.5% Memory: ✅ 43.704MB (SLO: <46.000MB -5.0%) vs baseline: +5.4% ✅ slice_noaspectTime: ✅ 54.306µs (SLO: <90.000µs 📉 -39.7%) vs baseline: -0.1% Memory: ✅ 43.598MB (SLO: <46.000MB -5.2%) vs baseline: +4.8% ✅ stringio_aspectTime: ✅ 3.809ms (SLO: <5.000ms 📉 -23.8%) vs baseline: +0.6% Memory: ✅ 43.589MB (SLO: <46.000MB -5.2%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 344.239µs (SLO: <500.000µs 📉 -31.2%) vs baseline: +0.2% Memory: ✅ 43.708MB (SLO: <46.000MB -5.0%) vs baseline: +5.1% ✅ strip_aspectTime: ✅ 269.242µs (SLO: <350.000µs 📉 -23.1%) vs baseline: +0.6% Memory: ✅ 43.694MB (SLO: <46.000MB -5.0%) vs baseline: +5.4% ✅ strip_noaspectTime: ✅ 175.757µs (SLO: <240.000µs 📉 -26.8%) vs baseline: +0.6% Memory: ✅ 43.650MB (SLO: <46.000MB -5.1%) vs baseline: +4.6% ✅ swapcase_aspectTime: ✅ 334.008µs (SLO: <500.000µs 📉 -33.2%) vs baseline: +0.3% Memory: ✅ 43.646MB (SLO: <46.000MB -5.1%) vs baseline: +4.8% ✅ swapcase_noaspectTime: ✅ 269.899µs (SLO: <400.000µs 📉 -32.5%) vs baseline: -0.9% Memory: ✅ 43.673MB (SLO: <46.000MB -5.1%) vs baseline: +5.0% ✅ title_aspectTime: ✅ 344.879µs (SLO: <500.000µs 📉 -31.0%) vs baseline: +8.6% Memory: ✅ 43.598MB (SLO: <46.000MB -5.2%) vs baseline: +5.3% ✅ title_noaspectTime: ✅ 258.772µs (SLO: <400.000µs 📉 -35.3%) vs baseline: -0.1% Memory: ✅ 43.679MB (SLO: <46.000MB -5.0%) vs baseline: +5.0% ✅ translate_aspectTime: ✅ 493.169µs (SLO: <700.000µs 📉 -29.5%) vs baseline: +0.5% Memory: ✅ 43.632MB (SLO: <46.000MB -5.1%) vs baseline: +4.7% ✅ translate_noaspectTime: ✅ 422.202µs (SLO: <500.000µs 📉 -15.6%) vs baseline: -0.8% Memory: ✅ 43.557MB (SLO: <46.000MB -5.3%) vs baseline: +4.6% ✅ upper_aspectTime: ✅ 292.627µs (SLO: <500.000µs 📉 -41.5%) vs baseline: -0.1% Memory: ✅ 43.585MB (SLO: <46.000MB -5.3%) vs baseline: +5.3% ✅ upper_noaspectTime: ✅ 234.975µs (SLO: <400.000µs 📉 -41.3%) vs baseline: -0.7% Memory: ✅ 43.770MB (SLO: <46.000MB -4.8%) vs baseline: +5.2% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 491.580µs (SLO: <700.000µs 📉 -29.8%) vs baseline: 📈 +18.1% Memory: ✅ 43.536MB (SLO: <46.000MB -5.4%) vs baseline: +4.5% ✅ ospathbasename_noaspectTime: ✅ 424.871µs (SLO: <700.000µs 📉 -39.3%) vs baseline: ~same Memory: ✅ 43.535MB (SLO: <46.000MB -5.4%) vs baseline: +4.3% ✅ ospathjoin_aspectTime: ✅ 620.344µs (SLO: <700.000µs 📉 -11.4%) vs baseline: +0.3% Memory: ✅ 43.470MB (SLO: <46.000MB -5.5%) vs baseline: +5.1% ✅ ospathjoin_noaspectTime: ✅ 630.307µs (SLO: <700.000µs -10.0%) vs baseline: +1.1% Memory: ✅ 43.450MB (SLO: <46.000MB -5.5%) vs baseline: +4.9% ✅ ospathnormcase_aspectTime: ✅ 345.232µs (SLO: <700.000µs 📉 -50.7%) vs baseline: +0.4% Memory: ✅ 43.547MB (SLO: <46.000MB -5.3%) vs baseline: +4.7% ✅ ospathnormcase_noaspectTime: ✅ 353.185µs (SLO: <700.000µs 📉 -49.5%) vs baseline: +0.6% Memory: ✅ 43.561MB (SLO: <46.000MB -5.3%) vs baseline: +4.8% ✅ ospathsplit_aspectTime: ✅ 478.529µs (SLO: <700.000µs 📉 -31.6%) vs baseline: -0.4% Memory: ✅ 43.273MB (SLO: <46.000MB -5.9%) vs baseline: +3.9% ✅ ospathsplit_noaspectTime: ✅ 491.987µs (SLO: <700.000µs 📉 -29.7%) vs baseline: ~same Memory: ✅ 43.431MB (SLO: <46.000MB -5.6%) vs baseline: +4.6% ✅ ospathsplitdrive_aspectTime: ✅ 369.484µs (SLO: <700.000µs 📉 -47.2%) vs baseline: +0.7% Memory: ✅ 43.689MB (SLO: <46.000MB -5.0%) vs baseline: +5.2% ✅ ospathsplitdrive_noaspectTime: ✅ 72.738µs (SLO: <700.000µs 📉 -89.6%) vs baseline: -1.0% Memory: ✅ 43.667MB (SLO: <46.000MB -5.1%) vs baseline: +5.1% ✅ ospathsplitext_aspectTime: ✅ 449.421µs (SLO: <700.000µs 📉 -35.8%) vs baseline: -0.6% Memory: ✅ 43.681MB (SLO: <46.000MB -5.0%) vs baseline: +4.8% ✅ ospathsplitext_noaspectTime: ✅ 455.752µs (SLO: <700.000µs 📉 -34.9%) vs baseline: -1.0% Memory: ✅ 43.604MB (SLO: <46.000MB -5.2%) vs baseline: +4.8% 🟡 Near SLO Breach (2 suites)🟡 flasksimple - 18/18✅ appsec-getTime: ✅ 3.425ms (SLO: <4.750ms 📉 -27.9%) vs baseline: -0.4% Memory: ✅ 55.345MB (SLO: <66.500MB 📉 -16.8%) vs baseline: +4.9% ✅ appsec-postTime: ✅ 2.918ms (SLO: <6.750ms 📉 -56.8%) vs baseline: +0.2% Memory: ✅ 55.722MB (SLO: <66.500MB 📉 -16.2%) vs baseline: +4.8% ✅ appsec-telemetryTime: ✅ 3.450ms (SLO: <4.750ms 📉 -27.4%) vs baseline: +0.7% Memory: ✅ 55.393MB (SLO: <66.500MB 📉 -16.7%) vs baseline: +4.8% ✅ debuggerTime: ✅ 1.882ms (SLO: <2.000ms -5.9%) vs baseline: ~same Memory: ✅ 48.903MB (SLO: <51.500MB -5.0%) vs baseline: +4.7% ✅ iast-getTime: ✅ 1.873ms (SLO: <2.000ms -6.3%) vs baseline: ~same Memory: ✅ 45.996MB (SLO: <49.000MB -6.1%) vs baseline: +4.9% ✅ profilerTime: ✅ 1.915ms (SLO: <2.100ms -8.8%) vs baseline: ~same Memory: ✅ 51.954MB (SLO: <52.500MB 🟡 -1.0%) vs baseline: +4.9% ✅ resource-renamingTime: ✅ 3.418ms (SLO: <3.650ms -6.3%) vs baseline: ~same Memory: ✅ 55.398MB (SLO: <60.000MB -7.7%) vs baseline: +4.9% ✅ tracerTime: ✅ 3.436ms (SLO: <3.650ms -5.9%) vs baseline: +0.3% Memory: ✅ 55.352MB (SLO: <60.000MB -7.7%) vs baseline: +4.8% ✅ tracer-nativeTime: ✅ 3.431ms (SLO: <3.650ms -6.0%) vs baseline: +0.2% Memory: ✅ 55.365MB (SLO: <60.000MB -7.7%) vs baseline: +4.9% 🟡 packagespackageforrootmodulemapping - 4/4✅ cache_offTime: ✅ 347.425ms (SLO: <354.300ms 🟡 -1.9%) vs baseline: ~same Memory: ✅ 42.373MB (SLO: <46.000MB -7.9%) vs baseline: +5.1% ✅ cache_onTime: ✅ 0.390µs (SLO: <10.000µs 📉 -96.1%) vs baseline: +0.3% Memory: ✅ 41.454MB (SLO: <46.000MB -9.9%) vs baseline: +3.8%
|
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
mergequeue build completed successfully, but the github api returned an error while merging the pr DetailsError: PUT https://api.github.com/repos/DataDog/dd-trace-py/pulls/17599/merge: 405 Repository rule violations found 3 of 3 required status checks are expected. [] (Request ID: 88B0:198333:3D642D1:F76037B:69E5F6AE) FullStacktrace: 3 of 3 required status checks are expected. [] (Request ID: 88B0:198333:3D642D1:F76037B:69E5F6AE) (type: GitFailure, retryable: false): PUT https://api.github.com/repos/DataDog/dd-trace-py/pulls/17599/merge: 405 Repository rule violations found 3 of 3 required status checks are expected. [] (type: ErrorResponse, retryable: true) |
Codeowners resolved as |
https://datadoghq.atlassian.net/browse/PROF-13112
Backport of #17456.