Demonstrably Secure Software Supply Chains with Nix
78 comments
·May 12, 2025beardedwizard
kuruczgy
Actually this is not quite true, in the xz hack part of the malicious code was in generated files only present in the release tarball.
When I personally package stuff using Nix, I go out of my way to build everything from source as much as possible. E.g. if some repo contains checked in generated files, I prefer to delete and regenerate them. It's nice that Nix makes adding extra build steps like this easy. I think most of the time the motivation for having generated files in repos (or release tarballs) is the limitations of various build systems.
XorNot
The xz attack did hit nix though. The problem is no one is inspecting the source code. Which is still true with nix, because everyone writes auto bump scripts for their projects).
If anyone was serious about this issue, we'd see way more focus on code signing and trust systems which are transferable: i.e. GitHub has no provision to let anyone sign specific lines of a diff or a file to say "I am staking my reputation that I inspected this with my own eyeballs".
zelphirkalt
Is it really stacking ones reputation? Think about it: If everyone is doing it all the time, an overlooked something is quickly dismissed as a mistake that is bound to happen sooner or later. Person X is reviewing so much code and does such a great job usually, but now they overlooked that one thing. And they even admitted their mistake. Surely they are not bad.
I think it would quickly fade out. What are we going to do, if even some organization for professional code reviews signs off the code but after 5y in the business they make 1 mistake? Are we no longer going to trust them from that day on?
I think besides signing code, there need to be multiple pairs of eyeballs looking at it independently. And even then nothing is really safe. People get lazy all the time. Someone else surely has already properly reviewed this code. Let's just sign it and move on! Management is breathing down our necks and we gotta hit those KPI improvements ... besides, I gotta pick up the kids a bit earlier today ...
Don't let perfect be the enemy of good. There is surely some benefit, but one can probably never be 100% sure, unless one goes into mathematical proofs and understands them oneself.
jrockway
Yeah, the more I read through actual package definitions in nixpkgs, the more questions I have about selling this as some security thing. nixpkgs is very convenient, I'll give it that. But a lot of packages have A LOT of unreviewed (by upstream) patches applied to them at nix build time. This burned Debian once, so I expect it to burn nixpkgs someday too. It's inevitable.
I do think reproducible builds is important. It lets people that DO review the source code trust upstream binaries, which is often convenient. I made this work at my last job... if you "bazel build //oci/whatever-image" you end up with a docker manifest that has the same sha256 as what we pushed to Docker Hub. You can then read all the code and know that at least that's the code you're running in production. It's neat, but it's only one piece of the security puzzle.
p1necone
(Effectively) nobody will ever be serious about this issue unless it were somehow mandated for everyone. Anyone who was serious about it would take 3x as long to develop anything compared to their competitors, which is not a viable option.
spookie
This is why distros with long release cycles are better. Usually more time for eyeballs to parse things.
Take Debian for example, the commit never made it to stable.
transpute
> provision to let anyone sign specific lines of a diff
Good idea that should be implemented by git itself, for use by any software forge like github, gitlab, codeberg, etc.
justinrubek
While the code itself did get to nix, the exploit was not functional specifically due to how nix works. That doesn't mean that a more sophisticated attack couldn't succeed though. It was mostly luck that kept it from affecting NixOS.
throwawayqqq11
Your preference to compile your backdoors does not really fix the problem of malicious code supply.
I have this vague idea to fingerprint the relevant AST down to all syscalls and store it in a lock file to have a better chance of detection. But this isnt a true fix either.
kuruczgy
Yes you are right, what I am proposing is not a solution by itself, it's just a way to be reasonably confident that _if you audit the code_, that's going to be the actual logic running on your computer.
(I don't get the value of your AST checksumming idea over just checksumming the source text, which is what almost all distro packages do. I think the number of changes that change the code but not the AST are negligible. If the code (and AST) is changed, you have to audit the diff no matter what.)
The more interesting question that does not have a single good answer is how to do the auditing. In almost all cases right now the only metric you have is "how much you trust upstream", in very few cases is actually reading through all the code and changes viable. I like to look at how upstream does their auditing of changes, e.g. how they do code review and how clean is their VCS history (so that _if_ you discover something fishy in the code, there is a clean audit trail of where that piece of code came from).
SOLAR_FIELDS
This makes perfect sense on a beefy super powered dev laptop with the disk space upgrade on an unsaturated symmetrical gig connection.
I’m only exaggerating a little bit here. Nix purism is for those who can afford the machines to utilize it. Doing the same on old hardware is so slow it’s basically nontenable
justinrubek
This shows only a surface level understanding of what nix provides here.
One of the biggest benefits is the binary cache mechanism, which allows you to skip building something but have the effective result of the build pulled from the cache. It's classical distributions that make building from source only possible for those who can afford the infrastructure, nix is what enables the rest of us to do so.
zeec123
The nix cache exists for a reason.
turboponyy
Glossing over some details, the build artifact and build definition are equivalent in Nix. If you know the build definition, you can pull the artifact from the cache and be assured that you have the same result.
0xDEAFBEAD
>When I personally package stuff using Nix, I go out of my way to build everything from source as much as possible. E.g. if some repo contains checked in generated files, I prefer to delete and regenerate them. It's nice that Nix makes adding extra build steps like this easy. I think most of the time the motivation for having generated files in repos (or release tarballs) is the limitations of various build systems.
You know what would be really sweet?
Imagine if every time a user opted to build themselves from source, a build report was by default generated and sent to a server alongside the resulting hashes etc. And a diff report gets printed to your console.
So not only are builds reproducible, they're continuously being reproduced and monitored around the world, in the background.
Even absent reproducibility, this could be a useful way to collect distribution data on various hashes, esp. in combination w/ system config info, to make targeted attacks more difficult.
yencabulator
I think a big part of the push is just being able to easily & conclusively answer "are we vulnerable or not" when a new attack is discovered. Exhaustive inventory already is huge.
tough
i read somewhere go has a great package for this that checks statically typed usage of the vuln specific functions not whole package deps
XiZhao
I run a sw supply chain company (fossa.com) -- agree that there's a lot of low hanging gains like inventory still around. There is a shocking amount of very basic but invisible surface area that leads to downstream attack vectors.
From a company's PoV -- I think you'd have to just assume all 3rd party code is popped and install some kind of control step given that assumption. I like the idea of reviewing all 3rd party code as if its your own which is now possible with some scalable code review tools.
nyrikki
Those projects seem to devolve into a boil the ocean style projects and tend to be viewed as intractable and thus ignorable.
In the days everything was http I use to set a proxy variable and have the proxy save all downloaded assets to compair later, today I would probably blacklist the public CAs and do an intercept, just for the data of what is grabbing what.
Fedramp was defunded and is moving forward with a GOA style agile model. If you have the resources I would highly encourage you to participate in conversations.
The timelines are tight and they are trying to move fast, so look into their GitHub discussions and see if you can move it forward.
There is a chance to make real changes but they need feedback now.
beardedwizard
+1, I think you have to assume owned as well and start defending from there. Companies like edera are betting on that, but sandbox isn't panacea, you really need some way to know expected behavior.
TZubiri
When you have so many dependencies that you need to create complex systems to manage and "secure" the dependencies. The problem is that you have too many dependencies, you are relying on too many volunteer work, and you are demanding too many features while paying for too little.
The professional solution is to PAY for your Operating System, and rely on them to secure it. Whether it be to Microsoft or to Red Hat. You KNOW it's the right thing to do, and this is overintellectualizing your need to have a gratis operating system and charge non-gratis prices to your clients in turn.
lmm
How does that solve the problem? Both Microsoft and IBM/Red Hat have shipped backdoored code in the past and will no doubt do so again. At most you might be able to sue them for a refund of what you paid them, at which point you're no better off than if you'd used a free system from the start.
null
justinrubek
I couldn't disagree more with everything you've said.
ngangaga
I wish we would use terms like "verifiable" or "reproducible" rather than "secure", which is quite difficult to evaluate out of context of usage.
sollewitt
Valuably you also get demonstrable _insecure_ status - half the pain for our org of log4js was figuring out where it was in the stacks, and at which versions. This kind of accounting is really valuable when you're trying to figure out if and where you are affected.
niam
> it offers integrity and reproducibility like no other tool (btw. guix also exists)
This rubs me the wrong way. They acknowledge that alternative tools exist, but willfully use the wrong-er statement in pursuit of a vacuous marketing idiom.
Zambyte
To be fair, Guix was originally a Nix fork, so dismissing it was "just" a Nix fork doesn't seem that out there. I believe at this point all original Nix code has been replaced and Guix has completely diverged and is also awesome software, but I can see why someone would be inclined to say something like that if they were missing the full picture.
SkyMarshal
It wasn't really a "fork" though, was it? Rather a from-scratch re-implementation of the Nix/NixOS concepts using Guile Scheme in place of the Nix config language.
Zambyte
From the link you shared:
> It uses low-level mechanisms from the Nix package manager, but packages are defined as native Guile modules, using extensions to the Scheme language—which makes it nicely hackable.
Also check out: https://git.savannah.gnu.org/cgit/guix.git/tree/?h=v0.1
transpute
What's the best place to read about the history of Guix forking from Nix?
abhisek
This solves the problem of provenance and possibly build integrity. Given an artifact, it will allow identifying exact source from which each components are built.
But it still implicitly assumes that the source is secure and trusted. This is where a lot of problem happens when the source is compromised and malicious code is added.
SkyMarshal
Source code signing goes a long way to solving that problem. Assuming the source author reviews their own code and signs it immediately after.
huimang
Is the header image ai generated? For shame. No point in reading any further.
civodul
Shameless plug but probably relevant in the context of supply chain security: https://programming-journal.org/2023/7/1/
XorNot
This still doesn't fix the "trusting trust" attack: which Guix actually can, and which can bootstrap sideways to build other distros.
It also doesn't do anything which regular packaging systems don't (nix does have some interesting qualities, security ain't one of them): I.e. that big list of dependencies isn't automatic in any way, someone had to write them, which in turn makes it exactly the same as any other packaging systems build-deps.
christophilus
How does guix fix the trusting trust attack?
Aside: I wonder if AI code inspection and review could be put in place to detect xz-like malicious changes to the supply chain for major distros.
CBLT
https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-...
Guix bootstraps (in 2023, no clue about now) from a 357-byte program. You audit the bytecode.
cyrnel
This seems to only address a few of the nine threats to the software supply chain, mainly "(D) External build parameters" and maybe the content-addressable storage addresses some of the distribution phase threats: https://slsa.dev/spec/v1.1/threats
There are still many other ways that a dependency can be exploited before or after the build phase.
jchw
Nix doesn't, can't, and will obviously never be able to audit your dependencies, but what it can do is give you a way in which you can audit everything byte-for-byte and end-to-end from input to output. In most architectures it is pretty hard to even get to this point because there is no rigorous tracking of dependencies and side-effects; e.g. if your builds are not sandboxed from the network, how can you be sure that the inputs you audited really account for all of the inputs to a build? Nix has a (complete) answer for that, among other things.
transpute
Debian reproducible builds, Guix, StageX and Yocto/OpenEmbedded have also worked in this area.
jchw
Reproducible builds are adjacent, but ultimately orthogonal, to what Nix does.
Reproducible builds provide strong evidence that a given set of inputs were used to produce a specific output. A lot of the work to be done here by each of these individual projects is beneficial to the entire ecosystem, including Nix. A lot of it is just fixing bugs and removing accidental non-determinism from builds. The main value this provides is that it allows a third-party to verify with relatively good certainty that binaries provided by some entity match the expected source code.
Nix provides a hermetic environment to build software with every single input from the bootstrap seed to the final build fully accounted for. Builds can't access the Internet or the host filesystem, they can only access files from their inputs. Impurities must go through the FODs, and the outputs from FODs have to match a prerecorded cryptographic hash, so they must be fully bit-reproducible or the build fails. When you have a reproducible Nix derivation, it is a strict superset of a reproducible package virtually everywhere else, because you can have a very high assurance that you can know and can audit individually every input to the derivation. This is useful for both auditing and reproducible builds.
Reproducible builds are important, but reproducible builds alone are not a panacea. They obviously don't tell you if your source code is free of defects, accidental or otherwise, and neither does Nix. Still, Nix does something basically nothing else does, by making the entire build process fully hermetic.
(Guix, being inspired by Nix, is, as far as I know, roughly the same, although Guix has put more effort into the bootstrap and package reproducibility. Still, Guix and Nix stand in a class of their own as far as usefulness to supply chain security go, even if they probably won't fit neatly into the compliance theater version of supply chain security.)
carlhjerpe
Yocto is the worst thing over ever worked with. Its just a pile of dirt on top of a pile of dirt
seeknotfind
Nothing is demonstrably secure, only not demonstrably insecure. This is - hey our builds come with a bunch of resources you can use to try to prove they're insecure, but you probably can't - but it's an advertisement.
Tractor8626
Classical Nih
"It's easier to do ThingA with Nix because you don't have to do ThingB!" (proceed to explain how to do ThingB but with Nix)
> You don't need to maintain your own forks and patchsets
... but you need to maintain your own nix packages and build scripts which is basically same amount of work
miloignis
Presumably, most to all of your dependencies will be in nixpkgs already, so you don't have to maintain them.
tucnak
The laborious extents to which people would go simply to not use Guix.
Zambyte
I also use Guix. Quickly skimming the article, I don't see anything that jumps out that Guix does all that different. What are you suggesting?
mtndew4brkfst
The laborious extents to which the Guix project would go simply to alienate people who own certain classes of computer hardware. I have a list of Nix criticisms as long as my arm, but at least its documentation and user community never chastised me for my work laptop's OS or brand/model.
The bummer about lots of supply chain work is that it does not address the attacks we see in the wild like xz where malicious code was added at the source, and attested all the way through.
There are gains to be had through these approaches, like inventory, but nobody has a good approach to stopping malicious code entering the ecosystem through the front door and attackers find this much easier than tampering with artifacts after the fact.