Is NixOS truly reproducible?

> Our most important finding is that the reproducibility rate in nixpkgs has increased steadily from 69% in 2017 to about 91% in April 2023. The high reproducibility rate in our most recent revision is quite impressive, given both the size of the package set and the absence of systematic monitoring in nixpkgs. We knew that it was possible to achieve very good reproducibility rate in smaller package sets like Debian, but this shows that achieving very high bitwise reproducibility is possible at scale, something that was believed impossible by practitioners4

I think people in this thread are focusing on the wrong thing. Sure, not all packages are reproducible, but the project is systematically increasing the percentage of projects that are reproducible while ALSO adding new projects and demonstrating conclusively that what was considered infeasible is actually readily achievable.

> The interesting aspect of these causes is that they show that even if nixpkgs already achieves great reproducibility rates, there still exists some low hanging fruits towards improving reproducibility that could be tackled by the Nix community and the whole FOSS ecosystem.

This work is helpful I think for the community to tackle the sources of unreproducible builds to push the percentage up even further. I think it also highlights the need for automation to validate that there aren't systematic regressions or regressions in particularly popular packages (doing individual regressions for all packages is a futile effort unless a lot of people volunteer to be part of a distributed check effort).

I think this debate comes down to exactly what "reproducible" means. Nix doesn't give bit-exact reproducibility, but it does give reproducible environments, by ensuring that the inputs are always bit-exact. It is closer to being fully reproducible than most other build systems (including Bazel) -- but because it can only reasonably ensure that the inputs are exact, it's still necessary for the build processes themselves to be fully deterministic to get end-to-end bit-exactness.

Nix on its own doesn't fully resolve supply chain concerns about binaries, but it can provide answers to a myriad of other problems. I think most people like Nix reproducibility, and it is marketed as such, for the sake of development: life is much easier when you know for sure you have the exact same version of each dependency, in the exact same configuration. A build on one machine may not be bit-exact to a build on another machine, but it will be exactly the same source code all the way down.

The quest to get every build process to be deterministic is definitely a bigger problem and it will never be solved for all of Nixpkgs. NixOS does have a reproducibility project[1], and some non-trivial amount of NixOS actually is properly reproducible, but the observation that Nixpkgs is too vast is definitely spot-on, especially because in most cases the real issues lie upstream. (and carrying patches for reproducibility is possible, but it adds even more maintainer burden.)

[1]: https://reproducible.nixos.org/

How this article discusses reproducibility in NixOS and declines to even mention the intensional model or efforts to implement it are surprising to me, since it appears they have done a lot of research into the matter.

If you don’t know, the intensional model is an alternative way to structure the NixOS store so that components are content-addressable (store hash is based on the targets) as opposed to being addressed based on the build instructions and dependencies. IIUC, the entire purpose of the intensional model is to make Nix stores shareable so that you could just depend on Cachix and such without the worry of a supply-chain attack. This approach was an entire chapter in the Nix thesis paper (chapter 6) and has been worked on recently (see https://github.com/NixOS/rfcs/pull/62 and https://github.com/NixOS/rfcs/pull/17 for current progress).

> Our most important finding is that the reproducibility rate in nixpkgs has increased steadily from 69% in 2017 to about 91% in April 2023. The high reproducibility rate in our most recent revision is quite impressive, given both the size of the package set and the absence of systematic monitoring in nixpkgs.

That's one way to read the statistic. Another way you could read the graph is that they still have about the same number (~5k) of non-reproducible builds, which has been pretty constant over the time period. Adding a bunch of easily reproducible additional builds maybe doesn't make me believe it's solving the original issues.

> We knew that it was possible to achieve very good reproducibility rate in smaller package sets like Debian, but this shows that achieving very high bitwise reproducibility is possible at scale, something that was believed impossible by practitioners.

Maybe I miss some nuance here, but why is Debian written off as being so much smaller scale? The top end of the graph here suggests a bit over 70k packages, Debian apparently also currently has 74k packages available (https://www.debian.org/doc/manuals/debian-reference/ch02.en....); I guess there's maybe a bit of time lag here but I'm not sure that is enough to claim Debian is somehow not "at scale".

I'll repeat my comment from last time this came up.[0]

I could be wrong (and I probably am) but I feel like the term "reproducible build" has shifted/solidified since 2006 when Dolstra's thesis was first written (which itself doesn't really use that term all that much). As evidence the first wikipedia page on "Reproducible builds" seems to have appeared in 2016, a decade after Dolstra's thesis, and even that stub from 2016 appears to prefer to use the term "Deterministic compilation".

Anyhow, when the Nix project originally spoke about "reproducible builds", what I understood was meant by that term was "being able to repeat the same build steps with the same inputs". Because of the lack of determinstic compilation, this doesn't always yield bit-by-bit identical outputs, but are simply presumed to be "functionally identical". There is, of course, no reason to believe that they will necessarily be functionally identical, but it is what developers take for granted every day, and if otherwise would be considered a bug somewhere in the package.

With Nix, when some software "doesn't work for me, but works for you", we can indeed recursively compare the nix derivation files locating and eliminating potential differences, a debugging process I have used on occasion.

I agree that "reproducible builds" now means something different, but that isn't exactly the fault of Nix advocates. I guess a new term for "being able to repeat the same build steps with the same inputs" is needed.

[0]https://news.ycombinator.com/item?id=41953155

I work on a matching decomp project that has tooling to recompile C into binaries matching a 28 year old game.

In the final binaries created by compiled with gcc 2.6.3 and assembled with a custom assembler there appear to be unused, uninitialized data that is whatever was in RAM when whoever compiled the game created the release build.

Since the goal is a matching (reproducible) binary, we have tools to restore that random data at specific offsets. Fortunately our targets are fixed

Although I'm aware many distros care somewhat about reproducible builds these days, I tend to associate it primarily with Guix System, I never really considered it a feature of NixOS, having used both (though spent much more time on Guix System now).

For the record, even in the land of Guix I semi-regularly see reports on the bug-guix mailing list that some package isn't reproducible. It seems to get treated as a bug and fixed then. With that in mind, and personally considering Guix kind of the flagship of these efforts, it doesn't surprise me if anyone else doesn't have perfectly reproducible builds yet either. Especially Nix with the huge number of things in nixpkgs. It's probably easier for stuff to fall through the cracks with that many packages to manage.

Is anyone actually implementing the concept of checking hashes with trusted builders? This is all wasted effort if that isn't needed.

I've seen it pointed out (by mjg59, perhaps?) that if you have a trusted builder, why don't you just use their build? That seems to be the actual model in practice.

Reproducibility seems only to be useful if you have a pool of mostly trustworthy builders and somehow want to build a consensus out of that. Which I suppose is useful for a distributed community but does seem like a stretch for the amount of work going in to reproducible builds.

I guess this is a tangent, but Nix to me feels like the right idea with the wrong abstraction. I can't explain / it would take a serious bit of genius to come up with an alternative worth switching to.

Has anyone done any better?

so looks like reproducibility rate of nixos is not that high, roughly similar with debian?

https://wiki.debian.org/ReproducibleBuilds

IIRC any package that uses Java isn't reproducible because system time and fixing it to epoch permamently causes issues in some application builds.

* there're maven and gradle plugins to make builds reproducible.

Aside from this being a great article with lots of interesting details, it's also a rare example of a headline that does NOT follow "Betteridge's law of headlines"