Improve support for cataloging nix package relationships#3837
Improve support for cataloging nix package relationships#3837
Conversation
| name: "Unit tests" | ||
| runs-on: ubuntu-24.04 | ||
| # we need more storage than what's on the default runner | ||
| runs-on: ubuntu-22.04-4core-16gb |
There was a problem hiding this comment.
Is this concerning? Is there something else we should do to limit/reduce the overall test resources used?
There was a problem hiding this comment.
yeah, I tried to limit these test fixtures, but it seems like when you're working with nix and building anything you'll need at least nixos/nix (or install with https://nixos.org/nix/install) which seems to be ~650MB right out of the door with no alterations. I'm adding jq and sqlite on top of it which seems to make the image swell to ~850MB... which is why I make these a multistage image so it's reduced to ~40MB.
But yeah, I'm a little concerned, we are getting up there in terms of disk usage for these fixtures (not including whats pulled down to build them):
$find . | grep stereoscope-fixture | xargs du -ch | tail -1
2.4G total
I don't think I have an answer within the scope of this PR though.
There was a problem hiding this comment.
It looks like tests didn't kick off the last commit on this PR, I hope this isn't due to some resource constraints on the larger runners.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
kzantow
left a comment
There was a problem hiding this comment.
LGTM -- nix is a little bit tricky!
| @@ -0,0 +1 @@ | |||
| Derive([("bin","/nix/store/h0cnbmfcn93xm5dg2x27ixhag1cwndga-glibc-2.34-210-bin","",""),("debug","/nix/store/hzp50ah3grczanw5fcgjsr390y3l8pb8-glibc-2.34-210-debug","",""),("dev","/nix/store/ap80aapy02950lnhkb3nsl58qqh55nbs-glibc-2.34-210-dev","",""),("out","/nix/store/jq8i4896sgmgbkk7nbjrnidxpxmgcid5-glibc-2.34-210","",""),("static","/nix/store/vqrka8z8k7spqsrvp41z0vvjfn0kcim6-glibc-2.34-210-static","","")],[("/nix/store/1zi0k7y01rhqr2gfqb42if0icswg65sj-locale-C.diff.drv",["out"]),("/nix/store/45j86ggi8mlpfslcrgvjf7m6phia21fp-raw.drv",["out"]),("/nix/store/4fnfsd9sc7bam6886hwyaprdsww66dg3-bison-3.8.2.drv",["out"]),("/nix/store/51azdrrvcqrk2hbky7ryphlwd99yz25d-linux-headers-5.18.drv",["out"]),("/nix/store/67s0qc21gyarmdwc181bqmjc3qzv8zkz-libidn2-2.3.2.drv",["out"]),("/nix/store/9rhliwskh3mrrs5nfzgz0x6wrccyfg7k-bootstrap-stage0-glibc-bootstrap.drv",["out"]),("/nix/store/cl1wcw2v1ifzjlkzi50h32a6lms9m25s-binutils-2.38.drv",["out"]),("/nix/store/ghjc8bkfk8lh53z14mk2nk7h059zh7vx-python3-minimal-3.10.5.drv",["out"]),("/nix/store/k3786wfzw637r7sylccdmm92saqp73d8-glibc-2.34.tar.xz.drv",["out"]),("/nix/store/l5zr5m1agvvnic49fg6qc44g5fgj3la1-glibc-reinstate-prlimit64-fallback.patch?id=eab07e78b691ae7866267fc04d31c7c3ad6b0eeb.drv",["out"]),("/nix/store/mf5kz6d01ab8h0rswzyr04mbcd6g5x9n-bootstrap-stage2-stdenv-linux.drv",["out"]),("/nix/store/nd1zy67vp028707pbh466qhrfqh4cpq6-bootstrap-stage2-gcc-wrapper-.drv",["out"]),("/nix/store/ra77ww7p2xx8jh8n4m9vmj6wc8wxijdb-bootstrap-tools.drv",["out"]),("/nix/store/wlldapf5bg58kivw520ll5bw0fmlaid7-raw.drv",["out"])],["/nix/store/001gp43bjqzx60cg345n2slzg7131za8-nix-nss-open-files.patch","/nix/store/7kw224hdyxd7115lrqh9a4dv2x8msq2s-fix-x64-abi.patch","/nix/store/8haph3ng4mgsqr6p4024vj8k6kg3mqc4-nix-locale-archive.patch","/nix/store/95hp6hs9g73h93safadb8x6vajyqkv6q-0001-Revert-Remove-all-usage-of-BASH-or-BASH-in-installed.patch","/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh","/nix/store/b1w7zbvm39ff1i52iyjggyvw2rdxz104-dont-use-system-ld-so-cache.patch","/nix/store/ikmqczy0y20n04a2b8qfflzwihv8139g-separate-debug-info.sh","/nix/store/mgx19wbmgrh3rblbxhs6vi47sha15n11-2.34-master.patch.gz","/nix/store/mnglr8rr7nl444h7p50ysyq8qd0fm1lm-dont-use-system-ld-so-preload.patch","/nix/store/xkd50xxii6k7l1kmw4l5x6xzbhamcs87-allow-kernel-2.6.32.patch","/nix/store/za0pg7fmysrcwrqcal26fnmzw6vycgdn-fix_path_attribute_in_getconf.patch"],"aarch64-linux","/nix/store/h0r631awi89fg5qpwn1nilpn36chqbkx-bootstrap-tools/bin/bash",["-e","/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"],[("BASH_SHELL","/bin/sh"),("NIX_CFLAGS_COMPILE",""),("NIX_HARDENING_ENABLE","pic strictoverflow format relro bindnow"),("NIX_NO_SELF_RPATH","1"),("bin","/nix/store/h0cnbmfcn93xm5dg2x27ixhag1cwndga-glibc-2.34-210-bin"),("buildInputs","/nix/store/b6hn4v59mi4wz7g1579ikbykd16qp4n1-linux-headers-5.18"),("builder","/nix/store/h0r631awi89fg5qpwn1nilpn36chqbkx-bootstrap-tools/bin/bash"),("configureFlags","-C --enable-add-ons --sysconfdir=/etc --enable-stackguard-randomization --enable-bind-now --with-headers=/nix/store/b6hn4v59mi4wz7g1579ikbykd16qp4n1-linux-headers-5.18/include --disable-profile --enable-static-pie --enable-kernel=3.2.0"),("debug","/nix/store/hzp50ah3grczanw5fcgjsr390y3l8pb8-glibc-2.34-210-debug"),("depsBuildBuild","/nix/store/rwwq8jmlc9dmasxa2ghjsj1p91mphvjx-bootstrap-stage2-gcc-wrapper-"),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("dev","/nix/store/ap80aapy02950lnhkb3nsl58qqh55nbs-glibc-2.34-210-dev"),("doCheck",""),("doInstallCheck",""),("enableParallelBuilding","1"),("enableParallelChecking","1"),("hardeningDisable","fortify pie stackprotector"),("installFlags","sysconfdir=$(out)/etc"),("is64bit","1"),("linuxHeaders","/nix/store/b6hn4v59mi4wz7g1579ikbykd16qp4n1-linux-headers-5.18"),("makeFlags","OBJCOPY=objcopy"),("name","glibc-2.34-210"),("nativeBuildInputs","/nix/store/hqls6k3pcic45spig9bir01i1f7biagk-bison-3.8.2 /nix/store/kb6n9zgssq80zkcm04admwm67gf480m4-python3-minimal-3.10.5 /nix/store/ikmqczy0y20n04a2b8qfflzwihv8139g-separate-debug-info.sh"),("out","/nix/store/jq8i4896sgmgbkk7nbjrnidxpxmgcid5-glibc-2.34-210"),("outputs","out bin dev static debug"),("patches","/nix/store/mgx19wbmgrh3rblbxhs6vi47sha15n11-2.34-master.patch.gz /nix/store/8haph3ng4mgsqr6p4024vj8k6kg3mqc4-nix-locale-archive.patch /nix/store/b1w7zbvm39ff1i52iyjggyvw2rdxz104-dont-use-system-ld-so-cache.patch /nix/store/mnglr8rr7nl444h7p50ysyq8qd0fm1lm-dont-use-system-ld-so-preload.patch /nix/store/za0pg7fmysrcwrqcal26fnmzw6vycgdn-fix_path_attribute_in_getconf.patch /nix/store/xkd50xxii6k7l1kmw4l5x6xzbhamcs87-allow-kernel-2.6.32.patch /nix/store/3l70d7kcfsh91w6792h4fqs4kjbq17py-glibc-reinstate-prlimit64-fallback.patch?id=eab07e78b691ae7866267fc04d31c7c3ad6b0eeb /nix/store/nqa9d4yxz1l2cgswzqr1pkm4jfrksm0q-locale-C.diff /nix/store/7kw224hdyxd7115lrqh9a4dv2x8msq2s-fix-x64-abi.patch /nix/store/001gp43bjqzx60cg345n2slzg7131za8-nix-nss-open-files.patch /nix/store/95hp6hs9g73h93safadb8x6vajyqkv6q-0001-Revert-Remove-all-usage-of-BASH-or-BASH-in-installed.patch /nix/store/qhlivc5m9wh8pq93v1smplibyxakbjby-raw /nix/store/955qy70ypq94m205iaaa3pm1pjrsdfya-raw"),("pname","glibc"),("postConfigure","# Hack: get rid of the `-static' flag set by the bootstrap stdenv.\n# This has to be done *after* `configure' because it builds some\n# test binaries.\nexport NIX_CFLAGS_LINK=\nexport NIX_LDFLAGS_BEFORE=\n\nexport NIX_DONT_SET_RPATH=1\nunset CFLAGS\n\n# Apparently --bindir is not respected.\nmakeFlagsArray+=(\"bindir=$bin/bin\" \"sbindir=$bin/sbin\" \"rootsbindir=$bin/sbin\")\n"),("postInstall","echo SUPPORTED-LOCALES=C.UTF-8/UTF-8 > ../glibc-2*/localedata/SUPPORTED\nmake -j${NIX_BUILD_CORES:-1} -l${NIX_BUILD_CORES:-1} localedata/install-locales\n\ntest -f $out/etc/ld.so.cache && rm $out/etc/ld.so.cache\n\nif test -n \"$linuxHeaders\"; then\n # Include the Linux kernel headers in Glibc, except the `scsi'\n # subdirectory, which Glibc provides itself.\n (cd $dev/include && \\\n ln -sv $(ls -d $linuxHeaders/include/* | grep -v scsi\\$) .)\nfi\n\n# Fix for NIXOS-54 (ldd not working on x86_64). Make a symlink\n# \"lib64\" to \"lib\".\nif test -n \"$is64bit\"; then\n ln -s lib $out/lib64\nfi\n\n# Get rid of more unnecessary stuff.\nrm -rf $out/var $bin/bin/sln\n\n# Backwards-compatibility to fix e.g.\n# \"configure: error: Pthreads are required to build libgomp\" during `gcc`-build\n# because it's not actually needed anymore to link against `pthreads` since\n# it's now part of `libc.so.6` itself, but the gcc build breaks if\n# this doesn't work.\nln -sf $out/lib/libpthread.so.0 $out/lib/libpthread.so\nln -sf $out/lib/librt.so.1 $out/lib/librt.so\nln -sf $out/lib/libdl.so.2 $out/lib/libdl.so\nln -sf $out/lib/libutil.so.1 $out/lib/libutil.so\ntouch $out/lib/libpthread.a\n\nfor i in \"$out\"/lib/*.a; do\n [ \"$i\" = \"$out/lib/libm.a\" ] || $STRIP -S \"$i\"\ndone\n\n# Put libraries for static linking in a separate output. Note\n# that libc_nonshared.a and libpthread_nonshared.a are required\n# for dynamically-linked applications.\nmkdir -p $static/lib\nmv $out/lib/*.a $static/lib\nmv $static/lib/lib*_nonshared.a $out/lib\n# Some of *.a files are linker scripts where moving broke the paths.\nsed \"/^GROUP/s|$out/lib/lib|$static/lib/lib|g\" \\\n -i \"$static\"/lib/*.a\n\n# Work around a Nix bug: hard links across outputs cause a build failure.\ncp $bin/bin/getconf $bin/bin/getconf_\nmv $bin/bin/getconf_ $bin/bin/getconf\n"),("postPatch","# Needed for glibc to build with the gnumake 3.82\n# http://comments.gmane.org/gmane.linux.lfs.support/31227\nsed -i 's/ot \\$/ot:\\n\\ttouch $@\\n$/' manual/Makefile\n\n# nscd needs libgcc, and we don't want it dynamically linked\n# because we don't want it to depend on bootstrap-tools libs.\necho \"LDFLAGS-nscd += -static-libgcc\" >> nscd/Makefile\n\n# Ensure that `__nss_files_fopen` can still be wrapped by `libredirect`.\nsed -i -e '/libc_hidden_def (__nss_files_fopen)/d' nss/nss_files_fopen.c\nsed -i -e '/libc_hidden_proto (__nss_files_fopen)/d' include/nss_files.h\n\n# Ensure that libidn2 is found.\npatch -p 1 <<EOF\n--- a/inet/idna.c\n+++ b/inet/idna.c\n@@ -25,1 +25,1 @@\n-#define LIBIDN2_SONAME \"libidn2.so.0\"\n+#define LIBIDN2_SONAME \"/nix/store/yhsfk2in77yalcy8dr7cwlixh8cnqp1l-libidn2-2.3.2/lib/libidn2.so.0\"\nEOF\n"),("preBuild",""),("preConfigure","export PWD_P=$(type -tP pwd)\nfor i in configure io/ftwtest-sh; do\n # Can't use substituteInPlace here because replace hasn't been\n # built yet in the bootstrap.\n sed -i \"$i\" -e \"s^/bin/pwd^$PWD_P^g\"\ndone\n\nmkdir ../build\ncd ../build\n\nconfigureScript=\"`pwd`/../$sourceRoot/configure\"\n\nmakeFlags=\"$makeFlags BUILD_LDFLAGS=-Wl,-rpath,/nix/store/46ily5fvz680l4sif6zp6pvs11ay936g-bootstrap-stage0-glibc-bootstrap/lib OBJDUMP=/nix/store/z2r64ripyh2nn23xdgicxkw8xbh5zs65-binutils-2.38/bin/objdump\"\n\n\n"),("preInstall","if [ -f /nix/store/h0r631awi89fg5qpwn1nilpn36chqbkx-bootstrap-tools/lib/libgcc_s.so.1 ]; then\n mkdir -p $out/lib\n cp /nix/store/h0r631awi89fg5qpwn1nilpn36chqbkx-bootstrap-tools/lib/libgcc_s.so.1 $out/lib/libgcc_s.so.1\n # the .so It used to be a symlink, but now it is a script\n cp -a /nix/store/h0r631awi89fg5qpwn1nilpn36chqbkx-bootstrap-tools/lib/libgcc_s.so $out/lib/libgcc_s.so\nfi\n"),("profilingLibraries",""),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("separateDebugInfo","1"),("src","/nix/store/wjbv1k6yigmb280wrvc1gkv8cnrsacij-glibc-2.34.tar.xz"),("static","/nix/store/vqrka8z8k7spqsrvp41z0vvjfn0kcim6-glibc-2.34-210-static"),("stdenv","/nix/store/jf31qhzp3d1zq3rz6b5mkfijw2xaib49-bootstrap-stage2-stdenv-linux"),("strictDeps","1"),("system","aarch64-linux"),("version","2.34-210")]) No newline at end of file | |||
There was a problem hiding this comment.
it looks like this format is sensitive to whitespace. I found splitting the lines significantly more readable but sadly made the tests fail, it would be great to somehow get a more readable version of this
There was a problem hiding this comment.
I thought the same thing too, but also wanted to have the real unaltered material for a test fixture
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The existing Nix cataloger looks for directories within
**/nix/storeand parses the directory names as packages in the formhash-name-version[-outputname]. This PR changes the behavior of the default Nix cataloger used:/nix/var/nix/db/db.sqlitefor package existencePackage + relationship results from the new DB approach should match that of the results of
nix-env -q --tree $(path to bin), so for example:The only difference being that cyclic and self dependencies are pruned.
When the internal nix DB cannot be found then:
Each package raised up by any nix cataloger (either the new one or the existing legacy one) has the following details:
{ "id": "b52daaf120fb1da9", "name": "zlib", "version": "1.3", "type": "nix", "foundBy": "nix-cataloger", "locations": [ { "path": "/nix/var/nix/db/db.sqlite", "layerID": "sha256:2f7c7b1832ba50ba3e3f06dd8815cb83985522ca2cf3d79d2d1d8f818b969435", "accessPath": "/nix/var/nix/db/db.sqlite", "annotations": { "evidence": "primary" } }, { "path": "/nix/store/4rx3vkkd91wkbhpflsplfga603cp1l1c-zlib-1.3", "accessPath": "/nix/store/4rx3vkkd91wkbhpflsplfga603cp1l1c-zlib-1.3", "annotations": { "evidence": "supporting" } }, { "path": "/nix/store/q9jf8xl1gsfmx4f1jhqpdc11yf0w9gv4-zlib-1.3.drv", "layerID": "sha256:2f7c7b1832ba50ba3e3f06dd8815cb83985522ca2cf3d79d2d1d8f818b969435", "accessPath": "/nix/store/q9jf8xl1gsfmx4f1jhqpdc11yf0w9gv4-zlib-1.3.drv", "annotations": { "evidence": "supporting" } } ], "licenses": [], "language": "", "cpes": [ { "cpe": "cpe:2.3:a:zlib:zlib:1.3:*:*:*:*:*:*:*", "source": "syft-generated" } ], "purl": "pkg:nix/zlib@1.3?drvpath=q9jf8xl1gsfmx4f1jhqpdc11yf0w9gv4-zlib-1.3.drv&output=out&outputhash=sha256%3Aec7b5d2fd77a9dd19ebe15f8b7f991ad16923e555634002c06f004ddb3dbc94c", "metadataType": "nix-store-entry", "metadata": { "path": "/nix/store/4rx3vkkd91wkbhpflsplfga603cp1l1c-zlib-1.3", "derivation": { "path": "/nix/store/q9jf8xl1gsfmx4f1jhqpdc11yf0w9gv4-zlib-1.3.drv", "system": "x86_64-linux", "inputDerivations": [ { "path": "/nix/store/6p6gk14y6b5mjpjz6al6ysyg6q9av59h-zlib-1.3.tar.gz.drv", "outputs": [ "out" ] }, { "path": "/nix/store/6zzlzgnsnsjygf82k83nz2vspbahibb3-bootstrap-stage3-stdenv-linux.drv", "outputs": [ "out" ] }, { "path": "/nix/store/ks6kir3vky8mb8zqpfhchwasn0rv1ix6-bootstrap-tools.drv", "outputs": [ "out" ] } ], "inputSources": [ "/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh" ] }, "output": "out", "outputHash": "sha256:ec7b5d2fd77a9dd19ebe15f8b7f991ad16923e555634002c06f004ddb3dbc94c" } }Where the specific changes are:
purlnow has an additionaldrvpathqualifier that is the basename of the derivation pathpath: full path to the derivation pathsystem: platform information from the standard system attributeinputDerivations: the set of other derivation outputs needed as inputs to perform the build for this packageinputSources: a set of paths required as input to perform the build for this package (essentially immutable blobs)The original
nix-store-catalogeris now deprecated and replaced with a newnix-catalogerwhich automatically determines the best source to reference when raising up packages. The legacynix-store-catalogerstill captures all files discovered, however, it will be raising up far more files than it had been in the past (a bug was fixed that resulted in fewer files found than what should have been raised).The new cataloger can be configured to include all files within store paths (off by default):
Type of change
Checklist: