Skip to content

fix(profiling): fix invalid lookups for asyncio frames#17057

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
mainfrom
dd/kowalski/fix/stack-profiler-string-table-crash
Mar 23, 2026
Merged

fix(profiling): fix invalid lookups for asyncio frames#17057
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
mainfrom
dd/kowalski/fix/stack-profiler-string-table-crash

Conversation

@KowalskiThomas

@KowalskiThomas KowalskiThomas commented Mar 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

This fixes a rare crash in the asyncio / uvloop sampling logic which could happen due to unwrapping a Result that was not ok.

I am planning to add a build option to the Profiler which will allow to add additional debug logging (something like PROFILING_DEBUG or something, like @taegyunkim added for memalloc with re-entry assertions). Under this condition, we could print or assert when that happens since I don't really think we expect those cases to ever happen... so it'd be helpful for debugging. (But not in this PR!)

Crash telemetry: ~58 crashes observed on v4.6.0 over the past 7 days, with the top crash signature being is_uvloop_wrapper_frame / ThreadInfo::unwind_tasks / ThreadInfo::sample / sampling_thread.

Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0   0x000073c78eeaf4b3 is_uvloop_wrapper_frame
#1   0x000073c78eeb69b3 ThreadInfo::unwind_tasks
#2   0x000073c78eeb6db9 ThreadInfo::sample
#3   0x000073c78eeb6eee std::_Function_handler<void (_ts*, ThreadInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}::operator()(InterpreterInfo&) const::{lambda(_ts*, ThreadInfo&)#1}>::_M_invoke
#4   0x000073c78eeb18b5 for_each_thread
#5   0x000073c78eeb194d std::_Function_handler<void (InterpreterInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}>::_M_invoke
#6   0x000073c78eeb0257 for_each_interp
#7   0x000073c78eeb1cc0 Datadog::Sampler::sampling_thread
#8   0x000073c78eeb1f27 call_sampling_thread

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Mar 21, 2026 •
edited
Loading

Copy link
Copy Markdown

View session in Datadog

Bits Dev status: ✅ Done

Comment @DataDog to request changes

@datadog-official

datadog-official Bot commented Mar 21, 2026

Copy link
Copy Markdown
Contributor

I can only run on private repositories.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Mar 21, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/internal/datadog/profiling/stack/src/echion/tasks.cc            @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/echion/threads.cc          @DataDog/profiling-python
releasenotes/notes/fix-stack-profiler-string-table-crash-cf4e7a9ec1af48bd.yaml  @DataDog/apm-python

@KowalskiThomas
fix(profiling): fix invalid lookups for asyncio frames
e0824b9 
Co-authored-by: KowalskiThomas <14239160+KowalskiThomas@users.noreply.github.com>
@KowalskiThomas KowalskiThomas force-pushed the dd/kowalski/fix/stack-profiler-string-table-crash branch from c9048a0 to e0824b9 Compare March 21, 2026 08:40
@KowalskiThomas KowalskiThomas added identified-by:crashtracking Identified by Crash Tracking Profiling Continous Profling labels Mar 21, 2026
@KowalskiThomas KowalskiThomas marked this pull request as ready for review March 21, 2026 09:00
@KowalskiThomas KowalskiThomas requested review from a team as code owners March 21, 2026 09:00
@taegyunkim

taegyunkim commented Mar 23, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Mar 23, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-03-23 13:25:13 UTC ℹ️ Start processing command /merge

2026-03-23 13:25:18 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 3h (p90).

2026-03-23 15:07:06 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot merged commit a9b467b into main Mar 23, 2026
436 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot deleted the dd/kowalski/fix/stack-profiler-string-table-crash branch March 23, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@taegyunkim taegyunkim taegyunkim approved these changes

@Kyle-Verhoog Kyle-Verhoog Awaiting requested review from Kyle-Verhoog Kyle-Verhoog is a code owner automatically assigned from DataDog/apm-python

Assignees

No one assigned

Labels

Bits AI identified-by:crashtracking Identified by Crash Tracking Profiling Continous Profling

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@KowalskiThomas @taegyunkim