Skip to content

Hash modulo meta#14686

Open
roberth wants to merge 8 commits intoNixOS:masterfrom
roberth:hash-modulo-meta
Open

Hash modulo meta#14686
roberth wants to merge 8 commits intoNixOS:masterfrom
roberth:hash-modulo-meta

Conversation

@roberth
Copy link
Copy Markdown
Member

@roberth roberth commented Dec 1, 2025

Motivation

meta is lost at the derivation level, but it doesn't need to be that way, thanks to quotient hashing aka hash modulo.

This change makes use of the existing "discrepancy" between derivation hashes and output hashes to allow more information to be stored in derivations without causing rebuilds.

The core of the change is fairly simple; see the hashDerivationModulo hunk.

Changes

  • Add system feature derivation-meta improving hash quotient logic to support storing meta
  • Adjust JSON formats to make meta a first class top level attr instead of the ATerm embedding

Notes

  • Must be opted in to by derivations/expressions, as usual, like ca derivations, structuredAttrs
  • Not supported in non-structuredAttrs. Didn't seem worth the added complexity. Users should move to structuredAttrs instead and afaik Nixpkgs is on track to do so.

Example

$ nix run github:roberth/nix/hash-modulo-meta -- \
  build --store ~/stores/tmp \
    --extra-experimental-features derivation-meta \
    --impure --expr \
    'with import (builtins.fetchTarball { url = "https://github.com/roberth/nixpkgs/archive/derivation-meta.tar.gz"; }) { config.derivationMeta = true; }; python3'
...
<progress bar, substituting from cache.nixos.org>
...
$ nix run github:roberth/nix/hash-modulo-meta -- derivation show --store ~/stores/tmp --extra-experimental-features derivation-meta --impure --expr 'with import (builtins.fetchTarball { url = "https://github.com/roberth/nixpkgs/archive/derivation-meta.tar.gz"; }) { config.derivationMeta = true; }; python3'
{
  "vcllzyc6d87wxlncjcwxlgy82lna4h4v-python3-3.13.9.drv": {
    "args": ...
    ...
    "meta": {
      "available": true,
      "broken": false,
      "changelog": "https://docs.python.org/release/3.13.9/whatsnew/changelog.html",
...
  }
}

Context


Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

@github-actions github-actions bot added documentation with-tests Issues related to testing. PRs with tests have some priority store Issues and pull requests concerning the Nix store labels Dec 1, 2025
@roberth roberth force-pushed the hash-modulo-meta branch 3 times, most recently from 846a9c9 to b2c772b Compare December 2, 2025 06:02
@edolstra
Copy link
Copy Markdown
Member

edolstra commented Dec 3, 2025

I didn't see this mentioned explicitly, but I assume __meta is not passed to the builder?

It's not entirely obvious how derivation meta data would be used in practice, especially since derivers are kind of useless (e.g. cache.nixos.org paths do have a deriver, but you have no way to get to them, so you wouldn't be able to get to the metadata). Perhaps in conjunction with #11749, the metadata could be propagated into the provenance records of store paths (which are already JSON).

@roberth
Copy link
Copy Markdown
Member Author

roberth commented Dec 3, 2025

I didn't see this mentioned explicitly, but I assume __meta is not passed to the builder?

Correct, and this is now also covered by a functional test.

not entirely obvious how derivation meta data would be used in practice

You can instantiate your stuff and inspect the metadata.
If you don't have the expression sources for what you've deployed, you probably should(!), but if you don't provenance could connect that back, either by eval reproduction, by reference or by value.

  • eval reproduction: provenance has a pinned flakeref, and you can instantiate that to acquire derivations
  • by reference: maybe you actually do have derivations in a store
  • by value: meta could be duplicated into the provenance structures, and it will benefit from the infrastructure that's set up around this:
    • having a reliable interface between Nix and the expression world (ie derivation { __meta })
      (meta attribute is not reliable because not the whole derivation closure occurs as packages that are seen by the CLI)
    • that interface getting used, pkgs: Add config.derivationMeta nixpkgs#466932

So this is a good step towards making #11749 more effective.

@roberth
Copy link
Copy Markdown
Member Author

roberth commented Mar 23, 2026

Marking as draft.
CA should operate on the stripped derivation too. We shouldn't need distinct traces for this, because meta is not part of the build process.

@roberth roberth marked this pull request as draft March 23, 2026 09:06
@roberth
Copy link
Copy Markdown
Member Author

roberth commented Mar 26, 2026

Still fits the model.

                derivation resolution                            usages
       ┌─────────────────────────────────────┐  ┌────────────────────────────────────────────────────────────────────────┐  



                  ╔════════════╗
                  ║  drvAttrs  ║
                  ╚══════╤═════╝
                         ║
                         ║ instantiate
                         ▼  
              ╔════════════════════╗             ╔════════════════╗
              ║ drvPath derivation ╟┄┄┄┄┄┄┄┄┄┄┄┄▷║  ✨ SBOMs ✨   ║
              ╚══════════╤═════════╝ ⋱          ╚════════════════╝
                         ║              ⋱──(status quo)
                         ║ resolve meta    ⋱
                         ▼                    ↘  ╔════════════════╗   ┌────────────────┐
       ╔═════════════════════════════════════╗┄┄▷║ IA output hash ╟┄┄▷│ .narinfo query │
       ║                                     ║   ╚════════════════╝   └────────────────┘
       ║  derivation for IA substitution     ║
       ║            (and more)               ║   ╔════════════════╗   ┌──────────────────────────────────────────────────┐
       ╚═════════════════╤═══════════════════╝┄┄▷║ hash           ╟┄┄▷│ 💀 CA deep build trace query f.k.a. realisation  │
                         ║                       ╚════════════════╝   └──────────────────────────────────────────────────┘
                         ║ resolve inputs
                         ▼
       ╔═════════════════════════════════════╗   ╔════════════════╗   ┌────────────────────────────┐
       ║  derivation, fully resolved         ╟┄┄▷║ hash           ╟┄┄▷│ Shallow realisation query  │
       ╚═════════════════════════════════════╝   ╚════════════════╝   └────────────────────────────┘




could probably improve some labels, but there ya go

@roberth roberth marked this pull request as ready for review March 27, 2026 00:09
@roberth roberth force-pushed the hash-modulo-meta branch from 1806b86 to 4845fd6 Compare April 1, 2026 21:09
@Ericson2314 Ericson2314 added the idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. label Apr 1, 2026
@Ericson2314
Copy link
Copy Markdown
Member

My view is that we can land it now (needs review but idea approved), but before it is stabilized, it should be rolled into a "v2 input addressing" feature, which will no longer involve hashDerivationModulo at all.

roberth and others added 4 commits April 1, 2026 23:39
Make the meta field optional in DerivationOptions JSON deserialization to maintain backward compatibility with JSON that predates the derivation-meta feature.

Changes:
- Use optionalValueAt instead of valueAt for meta field deserialization
- Mark meta as optional in derivation-options-v1 schema
- Add unit test for backward compatibility
When the derivation-meta experimental feature is enabled, extract the
__meta field from structuredAttrs to a top-level meta field in the
Derivation JSON format (used by `nix derivation show/add`). This makes
the format consistent with DerivationOptions JSON and provides better
visibility of metadata without it being buried in structuredAttrs.

The implementation:
- Creates derivationToJson() helper accepting ExperimentalFeatureSettings
- Extracts __meta to top-level meta field when feature is enabled
- Requires experimental feature when deserializing JSON with meta field
- Reconstructs __meta in structuredAttrs during deserialization
- Maintains backward compatibility with JSON lacking meta field

Tests verify both serialization and deserialization with mocked
experimental feature settings, and functional tests ensure the
experimental feature requirement is enforced.
Thank you fricklerhandwerk!

Co-authored-by: Valentin Gagarin <valentin@gagarin.work>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. store Issues and pull requests concerning the Nix store with-tests Issues related to testing. PRs with tests have some priority

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Have our cake and eat it too derivation metadata

4 participants