To be clear, this not about placeholders, but scratch paths, right?

Yes, thank you for the catch. RewritingSink substitutes scratch-path bytes (from makeFallbackPath-synthesised paths populated in outputRewrites when a sibling output is already in the store at build start), not builtins.placeholder bytes. The placeholder is the user-facing hook that causes a scratch path to end up embedded in the binary in the first place, but the bug is specifically in the scratch-path → final-path byte substitution after the builder exits.

Amending the PR description, commit message, and rl-next entry to use the precise terminology. The mechanism and the fix are unchanged — only the wording was wrong.

So I'll admit my current plan is to... just not have self references in the binary. How important is it to have paths like these?

(For other-output references, the idea is that you imperatively registered outputs, so you could do this sort of thing manually with Nix helping. E.g. install lib output, get store path back, use it in bin output.)

This is the root cause of NixOS/nixpkgs#507531

I doubt it: I expect this is an instance of the long-standing NixOS/nixpkgs#208951 bug, which is cursed and nondeterministic (fresh rebuilds tend to fix it and @zhaofengli found that it even depends on whether a machine has built the derivation before at all).

Since --check/--rebuild have the separate issue of clobbering code signatures, this patch will appear to fix the root cause under an obvious testing methodology without actually doing so.

(And TBH encoding Mach-O knowledge deep in Nix guts is a pretty awful layering violation, there are avenues to fix the rewriting issue that don't require that.)

So I'll admit my current plan is to... just not have self references in the binary.

Makes sense, and seems like the vastly superior architecture move. I wonder if something like this PR offers a short-term bridge for the existing darwin breakage (nixpkgs#507531, nixpkgs#208951) in the interim?

I don't see evidence that this PR fixes those issues.

Thank you, @emilazy. I'll try to address those three:

On whether this PR addresses the root cause of NixOS/nixpkgs#507531 / NixOS/nixpkgs#208951. You observed in nix-darwin#693 comment 38: "The bin/git in the broken derivation is identical to the correct one except for 32 bytes in the code signature section (a hash, maybe?). The share/man/man1/git.1.gz files, when uncompressed, differ only in the derivation hashes of the git-2.41.0-doc paths they link to." 32 bytes is the size of a SHA-256 hash entry in a CodeDirectory (CS_HASHSIZE_SHA256 = 256 bits), and the inline verification script in the PR description shows a single code-slot mismatch in each reproduction. The differently-embedded -doc paths match what RewritingSink rewrites when outputRewrites gets populated by a sibling output's presence in the store. My read is that this is the mechanism you were looking at. @winterqt later made an adjacent observation in #208951 comment 23: "It overrides the store path in the binaries, which breaks the code signatures of (at least) libraries (as they have their path embedded within). […]" I've added a source-level trace to the PR description for anyone who wants to follow the code path.

What this PR addresses is that specific mechanism — scratch-path → final-path byte substitution inside Mach-O code pages already covered by linker-signed page hashes. I shouldn't claim it covers every report in #208951; @tomberek's recent observation about "corrupted signature has been seen on different parts of the closure" may be a different root cause, of course.

This also explains the apparent nondeterminism that's made the bug hard to pin down. The trigger is hidden state: whether a sibling output happened to be in the store when the build started. On a fresh machine building an IA multi-output derivation cold, both outputs come out of the same builder run, neither is in the store yet, outputRewrites is empty for the sibling, and the binary is clean. On a machine that already has the sibling — from a prior build of this derivation, or substitution of one output but not the other — outputRewrites gets populated and RewritingSink runs over the binary. As I read it, this predicts that a cold-store build will produce a clean binary (it's the absence of the sibling at build start that matters, not the rebuild itself), and that whether a particular machine has previously built the derivation is what determines whether the bug surfaces. This would also explain why Hydra's cached artifacts tend to be valid: Hydra typically schedules a derivation's outputs together from a state where neither is in the store yet, so the rewrite path isn't exercised.

On the test methodology. I've added a non---check reproduction to the PR description: nix-store --delete <fish-out> followed by an ordinary nix build, with fish-doc still in the store, reproducing the same single page-hash mismatch. A positive counterpart against the patched daemon is included alongside it, showing the same derivation rebuilt cleanly (0/2526 mismatches). As I read the trigger condition, it isn't --check — it's "a sibling output is present in the store when the build starts". --check happens to force that state in a fresh store, which is convenient for the functional test, but an ordinary upgrade, a nix-store --delete of one output, or a failed build that left half the outputs behind would reach the same code path.

On the layering violation. I agree — Mach-O parsing logic in libstore is structurally wrong, and this PR makes it a little more wrong by adding page-hash recomputation alongside the existing darwin-specific code. My read is that the deeper cause is length-preserving byte substitution on already-signed outputs: ld -adhoc_codesign signs bytes that the daemon then rewrites via RewritingSink, and the Mach-O fixup only exists to repair what that substitution broke. The two avenues I can see that would actually move Mach-O knowledge out are @Ericson2314's imperative-output rework above (which would make the whole substitution machinery unnecessary) and a nixpkgs-side post-build hook (I'm not sure if the existing post-build-hook mechanism runs early enough to intercept the rewrite).

Short of those, both #14999 and this PR fix the corruption from inside the daemon, via different mechanisms. #14999 (@andrewgazelka) shells out to codesign -f -s - to re-sign the binary. That produces a working signature but not a bit-identical one — codesign's re-signed CodeDirectory differs from the original linker-signed signature in flags (the linker-signed bit is cleared), special-slot layout, and default page size. codesign -P 4096 would address the page size, but the other two remain, so rebuilds still fail --check. This PR recomputes the affected page hashes in place, which preserves bit-reproducibility.

a nix-store --delete of one output, or a failed build that left half the outputs behind would reach the same code path.

I don’t think these correspond to the circumstances in which we see Hydra produce these broken outputs.

It’s expected that local rebuilds of the derivations won’t have any issues, since it’s apparently nondeterministic and seemingly partially dependent on persistent system state of some kind. So building a broken‐in‐the‐cache derivation locally and seeing that it doesn’t exhibit the issue does not demonstrate that this PR fixes the issue; that’s already what we observe with no change.

I know @zhaofengli had a somewhat reproducible test setup, but it was difficult to arrange.

(For clarity: the sibling outputs thing is an interesting observation that I can imagine might have something to do with what we’re seeing here, but given the amount of times we see random stuff in staging-next crashing at startup from this bug, I’d be pretty surprised if they happened to all be builds that had failed before but still produced some outputs, and then got built again on the same machine with the output not having been cleaned up, or similar.)

Thank you, @emilazy. One correction first, then to your three points.

Correction: My statement that "a failed build that left half the outputs behind would reach the same code path" was wrong. Nix's registerOutputs is atomic per-derivation; a failed build doesn't leave one of a multi-output drv's outputs valid in the store. The other trigger scenarios are correct as written: nix-store --delete of one output, or substitution of one output but not the other.

It's expected that local rebuilds of the derivations won't have any issues

Local rebuilds with the specific trigger setup (sibling output already in the store, target output absent, substitution disabled) do reliably exhibit the issue on my machine. I just ran three consecutive iterations on aarch64-darwin (macOS 26.2, unpatched system daemon Nix 2.24.10), same nix-store --delete, same --option substitute false, same nixpkgs d96b37b, same store state (fish-doc present from a prior build):

iteration 1 NAR hash: sha256:1qplch87dy4242vxwi3s5h62m6gnywn0f8z9wf659vkrh6hm4a0g
iteration 2 NAR hash: sha256:1qplch87dy4242vxwi3s5h62m6gnywn0f8z9wf659vkrh6hm4a0g
iteration 3 NAR hash: sha256:1qplch87dy4242vxwi3s5h62m6gnywn0f8z9wf659vkrh6hm4a0g

Bit-identical between runs. Both produced codesign: invalid signature, exit-137 SIGKILL, and 1/2526 mismatches at page 1872 @ 0x00750000, the same single-slot mismatch as the cache.nixos.org artifact at the same store path.

since it's apparently nondeterministic

Seemingly nondeterministic in production, yes — but the trigger setup above is at least one state configuration under which it reproduces deterministically. As I read the bug I've verified, it's state-dependent rather than nondeterministic: it requires the sibling output to be in the store at the moment the build starts. Within that state, on this machine, it fires bit-identically. Outside that state (truly cold store, no sibling present), it doesn't trigger at all. Whether @zhaofengli's reproducible setup converges with this one or describes a distinct mechanism, I haven't traced yet; if there's a pointer to where it lives, I'd be glad to look and compare.

So building a broken‐in‐the‐cache derivation locally and seeing that it doesn't exhibit the issue does not demonstrate that this PR fixes the issue; that's already what we observe with no change.

This isn't the shape of the PR body's A/B. The unpatched local rebuild does exhibit the issue under the trigger setup — the same 1/2526 mismatch and SIGKILL shown above is what the unpatched block in the PR description shows. The patched-daemon rebuild from the same starting state and the same command sequence produces 0/2526 and runs. Same store path, same nixpkgs revision, same trigger condition; the only variable I changed is the daemon. So the comparison isn't "broken cache vs incidentally-clean local rebuild" — it's "broken cache, broken unpatched local rebuild under the trigger, clean patched local rebuild under the same trigger."

If you'd like to test on your machine, I've packaged the described trigger setup as a flake at https://github.com/ak2k/nix-507531-repro:

nix run github:ak2k/nix-507531-repro

On aarch64-darwin it runs the unpatched rebuild three times (to show the bug fires deterministically across iterations), runs the patched rebuild once via a private daemon built from this PR's commit, then prints a side-by-side comparison table and a final PASS/FAIL line. ~5–7 min wall time, one sudo prompt for the patched daemon spawn. A recorded transcript of a passing run is at examples/ab-test-output.txt if you'd rather read than run.

If on your darwin machine the corruption reproduces on the unpatched daemon under this trigger, the A/B in the PR body holds and the patched daemon's fix applies under the same trigger. If it doesn't reproduce, that would point to a second state factor I haven't isolated here, distinct from the trigger above. Either way, the bug is independently visible: cache.nixos.org is serving a fish that macOS refuses to execute, the unpatched local rebuilds above reproduce the same single-slot mismatch deterministically, and #507531 / #208951 seem to collect related reports from darwin users.

One epistemic caveat I should make: the observable here — a single SHA-256 page hash slot mismatch in a linker-signed CodeDirectory — is narrow enough that two distinct mechanisms could in principle produce indistinguishable outputs, and codesign --verify plus the kernel's page-in check can't tell them apart. The claim I can make is that this PR fixes the mechanism I've reproduced; the claim I can't make is that it fixes every report regardless of mechanism, since some of those could in principle have a distinct cause that converges on the same symptom. If there's something concrete pointing at a separate mechanism in some of those reports, I'd be glad to dig in.

Thanks @emilazy — your parenthetical caught a real narrowness in how I'd been framing this. Two refinements after looking more carefully at the mechanism and at Hydra's source.

1. The trigger isn't specifically "sibling present"; it's "any output being built has scratchPath != finalPath", which happens whenever that output was in the store at build start. In src/libstore/unix/build/derivation-builder.cc at this PR's base a37db9d24, scratch-path selection routes any already-present output through makeFallbackPath(status.known->path) at L798; the finish lambda at L1614 populates outputRewrites with a scratchHash → finalHash entry for every such output; and the rewriteOutput lambda at L1634 applies those rewrites via RewritingSink. Two variants:

• Sibling-reference: X is being built, sibling Y is present, X embeds Y's path. fish hits this: fish-4.2.1/bin/fish embeds its fish-4.2.1-doc sibling's share/doc/fish path as a literal string in __TEXT,__cstring.
• Self-reference: X is being built, X itself is already present at build start (forcing a fallback scratch path for X), X embeds $out-derived self-references. This fires under --rebuild / --repair-path specifically — a wanted=false already-valid output otherwise skips rewriteOutput via the AlreadyRegistered path at L1484, so the self-ref rewrite on X's own new bytes only runs when the rebuild explicitly re-enters rewriteOutput. zsh hits this under the PR's --rebuild functional test: zsh-5.9/bin/zsh has no sibling runtime references at all, but three self-references to /nix/store/s07v...zsh-5.9/{share/zsh/5.9/functions,lib/zsh/5.9,etc/zshenv}, all inside the flags=0x20002(adhoc,linker-signed) CodeDirectory.

The sibling/self split is the same distinction @Ericson2314 drew from the architectural direction above — "just not have self references in the binary" — reached here from the source side.

2. The trigger state is not rare on Hydra; it's a consequence of per-output substitution. My earlier examples (nix-store --delete of one output, substitution of one output but not the other) sounded narrow because I was thinking about end-user actions. Looking at Hydra's queue-runner source, the routes are routine:

• In subprojects/hydra-queue-runner/src/state/mod.rs at cd235f7, the master computes the set of missing outputs for a drv (L1788–L1815), then fans them out per output via substitute_output in a buffer_unordered(10) stream. A single failed substitute (transient network, partial cache state, S3 inconsistency) leaves the drv with some outputs substituted and others missing. The drv is then scheduled for a local build, with the partial state in place.
• The builder side does the same on its own store: substitute_paths loops ensure_path per path with the same partial-failure mode.
• Long-running workers: a worker that previously built package Y (where Y runtime-depends on D.out) ends up with D.out in its store. When Hydra later asks that worker to build D itself, D.out is present → fallback scratch path → rewrite fires in the newly-built bytes.
• GC asymmetry: workers GC on runtime-root reachability. For multi-output drvs where out is rooted but siblings aren't, the siblings get collected while out stays (or vice-versa for the rarer case).

None of these require "failed build that left some outputs"; they're routine consequences of Hydra's per-output substitution design plus long-running worker store state. For a given staging-next rebuild, the probability that a multi-output drv has at least one output present when its sibling needs rebuilding seems high enough that I don't think it would need to be a rare coincidence to explain the observed rate.

Quick pattern-check against the heavily-mentioned packages in the threads, verified in-situ on this machine via strings and otool -L. Sibling-reference: fish (bin/fish embeds its fish-4.2.1-doc sibling path as a cstring literal in __TEXT,__cstring), git (bin/git embeds its git-2.51.2-doc sibling path as a cstring literal), curl (-bin's bin/curl references its out sibling's libcurl.4.dylib via an LC_LOAD_DYLIB load command — same rewrite mechanism, different Mach-O section; the trigger isn't specific to cstring literals, it fires on any byte covered by a page hash). Self-reference: zsh (bin/zsh has three $out-derived self-references, no sibling refs), bash (bin/bash has one $out-derived self-reference, no sibling refs). gitFull isn't built locally so I haven't traced it, but it's the same derivation family as git.

If you or @zhaofengli have a reproduction that doesn't fit this trigger, I'd want to look at it.