Describe the bug

When enabling ca-derivations, if I build a bunch of drvs that use it or are downstream from one that does, I frequently get a crash like:

warning: output out of input /nix/store/096ihak5gdyh2x88r0ywjgh1hmjylfkg-Groq.Compiler.test.end_to_end.resize_linear_align_corners_float16_1x1x20x30_1x1x40x60_slt2x2.test.drv missing, aborting the resolving
error: unexpected end-of-file

This seems to preceded by this in the nix-daemon journal:

src/libstore/build/derivation-goal.cc:463: void nix::DerivationGoal::inputsRealised(): Assertion `attempt' failed.

From the source it looks like tryResolve warns about the failure, but it's actually an error not a warning, because next thing an assert trips over it.

However, it is incorrect about the drv failing, because it is in fact either generated. I can no longer check that non-invasively due to ca-derivations breaking drv->out and (apparently?) not expose its internal lookup mechanism, but nix-store -r instantly gives me the output so it must have worked. So if I just continually retry the build until the error stops happening, then it is able to get all the way through.

Steps To Reproduce

I only started getting this when I started doing remote builds, so it's probably related to either that or a certain amount of parallelism.

nix-env --version output

2.4.1. I'm not sure if newer versions have fixed this, but I couldn't find any references in closed issues. I'll try updating to 2.8, but it's tricky due to nix's lack of cross version support and tendency to introduce new bugs.