Nix Derivation Madness
44 comments
·October 31, 2025edolstra
Ericson2314
The biggest problem of all is that derivers are not unique! A separate "build trace" map will solve this.
tomberek
Presumably this would support a big improvement to both SBOM generation as well as various UX features and workflow improvements.
setheron
is that the 'build-trace' feature I saw John write about ? (I want to explore that more)
Ericson2314
I think Eelco has in mind a separate thing that would still be a store object field. But IMO we should not do that since derives are unique, and we should instead use the "build trace" instead, which properly handles that.
As Martin Schwaighofer has discussed, it is fine and in fact good for build traces entries to have arbitrary meta data, so the "claims" being cryptographically signed are more precise. (This is good for auditing, and if something looks suspicious, having full accountability.)
So on that grounds, if eelco would like to include some "this came from this flake" information as informal metadata. (formally the key must still the resolved derivation.) That is fine with me.
---
As I linked in my other reply, see my fast-growing https://github.com/NixOS/nix/pull/14408 docs PR where I try to formally nail all this stuff down for the first time.
mschwaig
I mentioned another alternative to adding flake-specific metadata to data structures that are transferred over the network, as part of the signed traces or otherwise, in a comment on that PR Eelco linked.
It's keeping flake-specific data locally, to guarantee that it matches how the user ended up with the data, not how the builder produced it. I think otherwise from the user POV such data could again look misleading.
YouAreWRONGtoo
[dead]
ronef
+1 to Farid, great write-up! What you’re seeing is the long-standing “deriver” mismatch: fixed-output derivations can change their .drv without changing the output path. Eelco is calling it out as well in the comment below. I believe the idea behind the path forward is there but happy to hear more!
Also. Check out Farid's other posts.
beardsciences
If I understand this correctly, upcoming Ca-derivations will fix this by making these situations expected, properly-handled cases rather than a weird bug? https://nixos.wiki/wiki/Ca-derivations
Ericson2314
Yes, a hope of mine is that we can stop using "hash derivation modulo" entirely.
I've recently started some fancy formal spec-level documentation here https://github.com/NixOS/nix/pull/14408 The "resolution" equivalence class is both simpler and better than the "hash derivation modulo ..." one.
(The fact that it is a mouthful to say what the derivations are modulo kinda gives the game away! I put "hash quotient derivation" in the docs to side-step the issue.)
edolstra
To be clear, there is no bug here: derivers are simply not uniquely determined in the presence of fixed-output derivations, which is by design. That's even more true with CA derivations.
CA derivations also introduce the opposite situation, namely that the same derivation can produce different output paths for different users (if the build is not bitwise reproducible).
setheron
pick your poison: 1:N or N:1 ;P
Ericson2314
It's both, multiple derivations can produce the same (content-addressed) store object, and the derivations may not be reproducible and produce different (content-addressed) store objects each time.
The reality of executing arbitrary programs on non-deterministic computers is, unfortunately, N:M!
(Cue deterministic WASM derivations or something.)
setheron
ca-derivations from what i understand, fixed-output derivations but more general.
The point of the article to me (author) was that i found it odd that Nix replaces the derivations when calculating the output path but not the derivation path. (talking about "paths" in Nix is so hard!)
beardsciences
That makes sense, thanks for clarifying. Great writeup.
amelius
> The road to Nix enlightenment is no joke and full of dragons.
Nix was a great research project. Now is the time to rewrite it from the ground up.
Ericson2314
The core store layer is quite small, and I am trying to thoroughly document it, with all 3 of:
- a more "academic" spec of what it does
- nuts-and-bolts JSON schema for many data types
- JSON golden tests instead of C++ literals in the unit tests as often as possible.
I hope this will make additional store layer easy to churn out.
(The "hash derivation modulo" that is so fiddly described in this blog post can be dropped in a world where we no longer have input addressing, and just have content-addressing. Or, in a world where we have a new, simpler type of input-addressing instead.)
jbstack
Well, there's Guix as an alternative if you want a similar concept but different implementation philosophy. For me the major disadvantage of Guix is lack of package availability compared to Nix.
amelius
Isn't there a way to transpile the scripts from Nix to Guix?
Y_Y
It's not to hard to translate manually, but since the dependency tree is massive it doesn't seem feasible to do wholesale.
YouAreWRONGtoo
[dead]
mystifyingpoi
I feel the same about HCL in Terraform. The tool is perfect, the language is bollocks.
otabdeveloper4
It has been rewritten a few times already. The "fixed output hash" is a dirty optimisation hack borne out of real-world needs and not a research idea.
Valodim
Eh. This can be applied to so many technologies that run the world..
eviks
> nix/store/24v9wpp393ib1gllip7ic13aycbi704g-ruby-3.3.9.drv
A different type of madness, but are ugly names so common, why not start with ruby-3.3.9 so any list of files is semantically sorted/readable?
rkomorn
The package name is "secondary" information in this context. The hash is the primary one because it's stable unless the input changes.
The semantic is "what did this configuration generate", not "what's this package's version".
eviks
it's primary for every human involved, also, the way you check whether it's changed is by automatically comparing that full hash, not its starting symbols, so you don't care where in the full string it's positioned
> The semantic is "what did this configuration generate", not "what's this package's version".
Then why have the name/version at all like in those nameless cache dirs?
rkomorn
It made sense to me when I looked at it, at mount points, at when it changed vs when it didn't, etc, so IDK what to tell you.
FWIW, I'm also pretty sure I'm human.
Edit: also, I'm pretty sure that I wouldn't find it any more or less complicated if the package name came first.
Kootle
In nix packages (derivations) are so lightweight that your store has tens of thousands of them, many with the same name, or with no meaningful name at all. On the rare occasions that you need to look in the store for a package you’re much more likely to be looking for a particular hash than a particular name. That, and having the hash as a prefix looks nicer in tabular output.
Ericson2314
If I had my way
1. store paths would have no names at all
2. listing the contents of the store directory would not be allowed
3. store paths have more bits of information
Then store paths are halfway decent (but non-revocable) capabilities.
eviks
> 2. listing the contents of the store directory would not be allowed
Wow, that's awful, that's what Windows AppStore does, so it's even hard to see how much of the preinstalled garbage there is or even whether you might have a huge game you forgot to uninstall but might want to to free up some space.
What's the cool benefit that could justify this limitation?
vatsachakrvthy
How could one debug if we couldn't view contents of the store directory?
singron
It really doesn't matter. As a normal user, you don't use `drv` files directly, and everything you configure yourself will use attribute paths in nixpkgs. E.g. `pkgs.ruby` or `pkgs.ruby_3_3`.
otabdeveloper4
It's done that way on purpose. Precisely so you don't try to use the paths semantically. The names literally mean nothing in this context.
catlover76
[dead]
The deriver field in Nix has always been a misfeature. It was intended to provide traceability back to the Nix expression used to create the derivation, but it doesn't actually do that (since that wasn't really possible in the pre-flakes world, without hermetic evaluation). So instead it just causes a lot of confusion when the deriver recorded in the binary cache doesn't match the local evaluation result, due to fixed-output derivations changing.
In the future, Nix will hopefully gain proper provenance tracking that will tell you exactly where a store path came from: https://github.com/NixOS/nix/pull/11749