fix(jitdump): support CPython jitdumps on MacOS with 64-bit thread-id #27
Closed
not-matthias wants to merge 3 commits into
Closed
Conversation
Add end-to-end parse tests over real cpython perf jitdump captures: a Linux x86_64 dump in the standard layout (u32 tid) and a macOS arm64 dump in the wider layout (u64 tid + alignment padding). Each must yield 195 CODE_LOAD + 195 CODE_UNWINDING_INFO records, exercising the per-record layout detection.
CPython on macOS declares the jitdump thread_id as uint64_t (filled via pthread_threadid_np) instead of the perf spec's uint32_t, adding 8 bytes (wider field + alignment padding) that shift the name and code bytes in every CODE_LOAD/CODE_MOVE record. The u32-only parser misread code_size as the code address and failed to parse these files at all. Widen tid to u64 and detect the layout per record from the body length: try the standard layout first, fall back to the wider macOS layout when the field offsets are not self-consistent with the record size.
try_parse now returns Result<Option<Self>, io::Error>: Ok(None) signals only a layout mismatch, while genuine read errors (e.g. UnexpectedEof on a truncated record) propagate. The narrower standard layout's read errors surface as truncation; the wider layout's are treated as "not this layout". Adds a regression test. Addresses PR #3 review feedback.
Owner
|
Oh nooooo Why didn't we catch this before it landed? Having to support two different struct layouts seems like a terrible outcome! |
Owner
|
This means that python is just straight up outputting invalid jitdump files. This needs to be fixed in python. On macOS the tid can be truncated, or set to zero. The tid is not useful information anyway - executable mappings are something that applies process-wide, the originating thread doesn't make a difference to how the information should be treated. |
Owner
|
Thank you for catching this! Before today I didn't even realize there were two different environment variables, I always thought |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CPython 3.15 introduces Jitdump support on MacOS: python/cpython#136461
However, since thread ids are 64-bit on MacOS they also changed the
tidfield in the struct which this crate currently fails to parse:https://github.com/python/cpython/blob/f31a89bb901067dd105b00cfa90523cf7ffdbbdd/Python/perf_jit_trampoline.c#L220-L224
I've added tests for both the Linux and MacOS jitdump format.