Skip to content

cpython: add separateDebugInfo, allow use of gdb libpython#93083

Merged
FRidh merged 2 commits intoNixOS:stagingfrom
risicle:ris-cpython-debug
Aug 15, 2020
Merged

cpython: add separateDebugInfo, allow use of gdb libpython#93083
FRidh merged 2 commits intoNixOS:stagingfrom
risicle:ris-cpython-debug

Conversation

@risicle
Copy link
Copy Markdown
Contributor

@risicle risicle commented Jul 13, 2020 •
edited
Loading

Motivation for this change

The first thing this does is enable separateDebugInfo for cpython. Bit surprised we didn't already have this.

It also exposes Tools/gdb/libpython.py in the package output. Used together with cpython's debugging symbols, this allows inspection of the python stack of cpython programs in gdb. This file is a little different from the rest of the python output by this package, in that it's not intended to be run by the current python being built, instead by whatever python being used by the gdb in question, which could be very different. Therefore I placed it placed in its own, but hopefully logical & predictable location: /usr/share/gdb/ /share/gdb/. Also skipped building bytecode for it for similar reasons.

This, of course, results in a massive rebuild and I'm not sure how I'd ever make a dent in it myself. I've tested python27 through python39. However I'm fairly sure we've got some custom python builds floating around, though I'm not sure how I'd find them.

Tested on linux x86_64 & macos 10.14, though on macos separateDebugInfo does nothing leaving this slightly pointless there.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@risicle risicle requested a review from FRidh as a code owner July 13, 2020 21:42
@ofborg ofborg bot added 6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch. 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Jul 13, 2020
doronbehar
doronbehar reviewed Jul 15, 2020
View reviewed changes
@risicle
cpython: expose gdb libpython.py
ede2e00 
used together with cpython's debugging symbols, this allows inspection of
the python stack of cpython programs in gdb. this file is a little
different from the rest of the python output by this package, in that it's
not intended to be run by the current python being built, instead by the
python being used by the gdb in question, which could be very different.
therefore placed in its own, but hopefully logical & predictable location.
@risicle risicle force-pushed the ris-cpython-debug branch from cd6abe7 to ede2e00 Compare July 15, 2020 18:39
@FRidh
Copy link
Copy Markdown
Member

FRidh commented Jul 26, 2020

@GrahamcOfBorg build python35 python36 python37 python38 python39

@FRidh FRidh merged commit 4b340cb into NixOS:staging Aug 15, 2020
@misuzu
Copy link
Copy Markdown
Contributor

misuzu commented Oct 28, 2020

Python size increased after this change by ~30 MB.

https://hydra.nixos.org/build/125194527#tabs-details
https://hydra.nixos.org/build/125824399#tabs-details

$ nix build /nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5 /nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5

$ nix path-info -sh /nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5 /nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5
/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5	  62.2M
/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5	  90.0M

$ du -hs /nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/* | sort -h
4,0K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/config.c
4,0K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/config.c.in
4,0K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/python-config.py
4,0K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/python.o
4,0K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/Setup.local
8,0K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/makesetup
16K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/install-sh
16K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/__pycache__
16K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/Setup
80K	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/Makefile
6,9M	/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a

$ du -hs  /nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/* | sort -h
4,0K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/config.c
4,0K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/config.c.in
4,0K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/python-config.py
4,0K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/Setup.local
8,0K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/makesetup
16K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/install-sh
16K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/__pycache__
16K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/Setup
36K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/python.o
80K	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/Makefile
35M	/nix/store/wbilyi119q8my0kz2pl4chgs35xhlnhm-python3-3.8.5/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a

Is lib/python3.8/config-3.8-x86_64-linux-gnu even needed? There is no such directory on Ubuntu 20.04: https://packages.ubuntu.com/focal/amd64/libpython3.8-stdlib/filelist

@risicle
Copy link
Copy Markdown
Contributor Author

risicle commented Oct 28, 2020

Oh dear. Looks like I had too much faith in the magic of separateDebugInfo.

@risicle
Copy link
Copy Markdown
Contributor Author

risicle commented Oct 28, 2020

I'm putting something on my TODO list to write some tests for cpython/gdb debugging so we can safely experiment with slimming it down.

@FRidh
Copy link
Copy Markdown
Member

FRidh commented Nov 3, 2020 •
edited
Loading

I'm going to revert this change, because its too big this way.

6ddd6b4

FRidh added a commit that referenced this pull request Nov 3, 2020
@FRidh
Revert "Merge pull request #93083 from risicle/ris-cpython-debug"
6ddd6b4 
This increase the closure size by 30 MB.

#93083 (comment)

This reverts commit 4b340cb, reversing
changes made to 47e499f.
This was referenced Jan 20, 2021
Merged
Merged
@risicle risicle mentioned this pull request Mar 17, 2022
Closed
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@doronbehar doronbehar doronbehar left review comments

@FRidh FRidh Awaiting requested review from FRidh

Assignees

No one assigned

Labels

6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@risicle @FRidh @misuzu @doronbehar