nix flake check: Skip substitutable derivations#13574
nix flake check: Skip substitutable derivations#13574getchoo wants to merge 1 commit intoNixOS:masterfrom
Conversation
1d22e2a to
3484dfc
Compare
src/nix/flake.cc
Outdated
| [&](const DerivedPath::Built & bfd) { | ||
| auto drvPathP = std::get_if<DerivedPath::Opaque>(&*bfd.drvPath); | ||
| if (!drvPathP || missing.willBuild.contains(drvPathP->path)) | ||
| toBuild.push_back(path); | ||
| }, |
There was a problem hiding this comment.
What's the thought behind this code?
Surely we want to build dynamic derivations too?
If this code has a reasonable intent, I would expect it to be usable in more places, if not already used in more places, so why isn't this a helper function?
There was a problem hiding this comment.
If this code has a reasonable intent, I would expect it to be usable in more places, if not already used in more places, so why isn't this a helper function?
Where else would this be used? nix flake check seems to be the main use case I can think of.
In theory nix build could have a flag, but it doesn't seem to be a good default.
There was a problem hiding this comment.
I don't understand the comment about dynamic derivations. Would dynamic derivations not also be included in the missing set?
There was a problem hiding this comment.
I meant specifically the case matching on Built vs Opaque. An opaque drvPath is a normal one, whereas a Built drvPath is dynamic. In the case of dynamic we should not just realise the inner drvPath (the one that generates the derivation), but also the result. That seems to be missing.
reasonable intent
(my words)
That sounds rough? Of course your intent is good, and so is the general idea of this solution. I meant to refer to the highlighted lines, the handler for Built.
I apologize if that came across wrong.
There was a problem hiding this comment.
While porting this over to Lix in https://gerrit.lix.systems/c/lix/+/3841, I did a bit of digging and it seems to me that this snippet actually hits a lot of the same beats as queryMissing() itself. Based on this, I tried just using the willBuild member of the MissingPaths struct directly, and it seems to work just as fine?
I know next to nothing about libstore though, so sorry in advance 😆
Regarding dynamic derivations: running nix --extra-experimental-features 'ca-derivations dynamic-derivations recursive-nix' --store /tmp/dyn-drvs flake check -v with the following flake.nix
{
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
outputs =
{ nixpkgs, ... }:
let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
in
{
checks.${system}.dyn-drv =
pkgs.runCommand "inner.drv"
{
outputHashMode = "text";
requiredSystemFeatures = [ "recursive-nix" ];
}
''
echo "let pkgs = import \"${pkgs.path}\" {};
in
pkgs.runCommand \"inner\" {} '''
sleep 10;
echo \"Hello from inner!\" > \$out
'''
" > inner.nix
cp $(${pkgs.nix}/bin/nix-instantiate inner.nix) $out
'';
};
}does seem to only build the inner drv. However, running this with Nix 2.30.2 results in the same behavior, so...I think this might be an existing bug?
3484dfc to
c8f3b79
Compare
c8f3b79 to
2445367
Compare
2445367 to
77c97e7
Compare
Since `nix flake check` doesn't produce a `result` symlink, it doesn't actually need to build/substitute derivations that are already known to have succeeded, i.e. that are substitutable. This can speed up CI jobs in cases where the derivations have already been built by other jobs. For instance, a command like nix flake check github:NixOS/hydra/aa62c7f7db31753f0cde690f8654dd1907fc0ce2 should no longer build anything because the outputs are already in cache.nixos.org. Based-on: DeterminateSystems#134 Based-on: https://gerrit.lix.systems/c/lix/+/3841 Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
77c97e7 to
9108813
Compare
| }); | ||
| } | ||
|
|
||
| Activity act(*logger, lvlInfo, actUnknown, fmt("running %d flake checks", toBuild.size())); |
There was a problem hiding this comment.
This is another difference from the Determinate Nix implementation that I thought would be nice
It felt a bit confusing for Nix to report all checks were running when only a subset were actually being built. I think this makes the fact that this feature is even working much clearer, while also being more representative of what Nix is really doing
Alternatively, we could keep the total number of checks with a message like fmt("running %d/%d flake checks", toBuild.size(), drvPaths.size())
|
IMO this is a good idea, but a low quality implementation of the idea. The right way to do it is the general reworking the goals to make clear that being in the cache is fine. I am OK merging something like this (with tests!) as a stop-gap, though. |
|
The fundamental fix would be #5025 (i.e. the ability to specify a destination store, so we can skip builds if they are already in the destination store), but that will take more work. |
That's not quite the same, because the substituters can be multiple and read-only stores. It might be possible to shoehorn that into some sort of union store, but if they're all read-only, you'd have to add the local store in there as well, for writing. It gets weird. |
|
I did a rebase in #13998 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
|
Would it be possible to add support for checking individual attributes instead of only a whole flake? Then I could Right now this gives me: EDIT: I ended up trying to implement this with the help of my friendly stochastic parrot: https://github.com/NorfairKing/nix/tree/check-individual-attributes |
Motivation
Quoting from DeterminateSystems#134
I think this would be a good fit here too, as it's a small, but really nice QoL feature!
Context
DeterminateSystems@5c95921
Add 👍 to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.