NixOS and reproducible builds could have detected the xz backdoor
182 comments
·March 22, 2025lolinder
nextos
> NixOS and reproducible builds did not detect the xz backdoor
Nix declarativeness is quite useful to increase protection against exploits in a number of ways. Unfortunately, there is still a lot of untapped potential. My number one priority would be to implement fine-grained ephemeral containers. Guix has these already.
This would make it convenient to run every single process with restricted privileges, including no access to ~/, except those directories that are needed by the task. That would prevent e.g. a rogue pip package from stealing SSH keys.
Still, I think the xz backdoor did not work on NixOS because its unusual non FHS-compliant filesystem structure.
lolinder
> I think the xz backdoor did not work on NixOS because its unusual non FHS-compliant filesystem structure.
Right, but this is not part of the security model, it's an incidental attribute of the OS that's there for other reasons and easily solved for if the attacker had prioritized it. The only reason why it didn't work is because the attacker didn't bother making it work on NixOS, not because he couldn't have if he'd wanted to.
sirspamalot101
[flagged]
sirspamalot112
[dead]
dralley
>Still, I think the xz backdoor did not work on NixOS because its unusual non FHS-compliant filesystem structure.
It didn't work on NixOS because the build-time hooks that inserted the backdoor only activated itself when it recognized that it was being built for an RPM or Debian package.
johnisgood
Which could have been adjusted, of course.
jstanley
> including no access to ~/
This is such a headache with snap and flatpak though.
If you're trying to do something that the package maintainer thought of in 5 minutes of testing then it's usually fine, but still inside any non-trivial application you'll often find parts that try to use extra privileges that aren't documented because they're not normally considered "privileges".
Some examples of sandboxing issues:
* FreeCAD doesn't have access to /usr which means when you try to make a Draft ShapeString you can't pick any fonts
* FreeCAD stores the path to the shape of a milling cutter inside your project, but that path is inside /mnt/.FreeCAhjhffg or whatever so it doesn't work after you restart the program
* Inkscape gcodetools has its own "idiomatic" way of saving out results which doesn't use a file dialog, and therefore can't save any gcode because it can't write to your home directory
* Matrix is only allowed to access ~/Downloads/ which means you can't share any files with anyone unless you copy them into Downloads first
* "Recent" in the Gimp file picker doesn't show any of my recent files, presumably because it is using its own sandboxed file dialog which doesn't have access to the actual "Recent" files
* Docker can't access /tmp/.X11-unix which means you can't give docker containers access to X
In all of these cases you can work around it of course (mainly by having an accurate mental model of the problem and guessing a path that the program is allowed to access), but the user experience is just made worse for no benefit.
The general theme is that the user wants the sandboxed program to be able to do something that the person who assigned privileges didn't think of.
So maybe if we must do sandboxing, let's make it easy for users to break programs out of the sandbox when it suits them?
vladvasiliu
This looks like a chicken and egg problem to me. You can't sandbox things properly because things don't specify their required privileges properly. Things don't specify their privileges properly because things aren't sandboxed so there's no need to think about that.
As a user, I quite like the iOS approach, of apps "sharing" their ressources with other apps or asking for permission to access this or that collection of ressources. This can probably be improved and adapted to a non-touch model, but I think the concept is nice.
But, of course, apps have to be built for this kind of environment, so I think there will unfortunately be some janky transition period, with the customary competing, incompatible implementations.
fc417fc802
You raise many good points. One of the primary reasons I don't use nix or guix for everything is because it seems like there's too much magic and it's often too difficult for me to figure out how to modify a detail if a toggle wasn't explicitly provided for it. Even just getting insight into the chain of events to debug things was incredibly obtuse the last time I played with nix. Like cmake on steroids. At least cmake will spit out a multi-megabyte trace file for me to pick through. (I'm convinced cmake is an elaborate conspiracy to waste developer time.)
> let's make it easy for users to break programs out of the sandbox when it suits them?
At least for Flatpak given the things you described this is quite straightforward via bind mounts. Although it did seem a bit goofy having an entire list of per-application bind mounts in my fstab. Maybe things have improved since I last tried going that route?
> "idiomatic" ... doesn't use a file dialog,
That's an Inkscape (plugin?) bug plain and simple. GUI apps should be using the appropriate xdg portals for all supported operations at this point. The only excuse (IMHO) is for missing or broken functionality.
It would be like a system tray widget not working and blaming the DE instead of the program that fails to implement the decently old and widely adopted standard.
> Matrix ... you can't share any files with anyone
Which client is this? Anyway the xdg portal should work. Did the dev try it? Programs should never need blanket access to save or open specific user supplied paths as a one off. That's a large part of the point of sandboxing stuff in the first place.
johnisgood
I wonder how much it applies to Guix.
ratorx
You don’t even need to run in a container for this. It’s possible to do this entirely in systemd service configuration. The easiest way is just to have separate user for every service and reduce stuff running as root. You can also restrict filesystem access, network access and even syscall access (although some of this may be implemented as a container under the hood).
Unfortunately, this wouldn’t help with the xz vulnerability because the SSH server is the one loading the compromised library in that case (indirectly). Since SSH itself needs to have access to the private keys, it’s not really easy to secure it against vulnerabilities in the library it loads itself.
On the flip side, unless the vulnerability is in one of the important binaries/shared libraries, the amount of damage it can cause it probably quite contained with simply having good user isolation. Nix can make this analysis really simple (because of explicitly specified dependencies), so you can crack down on critical dependencies a lot more easily.
nextos
> It’s possible to do this entirely in systemd service configuration
Sure, but I think that leaves out many use cases. What if I want to e.g. start a Python shell that has access to certain directories, and nothing else, including no network access?
Nix provides a good way of doing that for common use cases, as it has decent support for Firejail. But I would like something like Guix containers, which is convenient for any ad hoc use case. This greatly reduces any security threat. It's a poor-man's QubesOS.
markhahn
sshd was compromised, and it doesn't access private keys. (well, probably the host key.)
sirspamalot101
[flagged]
conradev
The maintainer would just change the sandboxing constraints to weaken the software. Just like they did in the first place. You can try to make obfuscation difficult, but it’s always possible.
jorvi
> This would make it convenient to run every single process with restricted privileges, including no access to ~/
Please no. I understand why Flatpaks do it, but this is one of the most ridiculously annoying things about the Flatpak sandbox. You can often only drag 'n drop from ~/Downloads/, and from any other location either causes the receiving application to glitch out, fail silently, or fail with a general error. Hell, you sometimes can't even copy-paste an image from one application to another via the copy-paste buffer! Meanwhile on macOS and Windows it works perfectly.
nextos
Why? I am just asking for a simple UI, which Guix already has. Mainly for CLI applications. The idea is to be able to launch an ephemeral shell with any combination of packages, filesystem R/W privileges, and network access in a convenient way.
I think launching e.g. a Python shell with some packages that are potentially compromised and letting those read ~/.ssh and whatever else they want is fundamentally insecure. Rogue PyPI packages that steal SSH keys is not a theoretical security breach, it already happened several times [1].
The current security model in Unix is untenable. But I agree well-implemented sandboxing should be frictionless. What you are experiencing is probably a X or Wayland sandboxing glitch. I also dislike Flatpak, for other reasons, but that doesn't make sandboxing a bad abstraction. It's just that we don't happen to like this particular implementation.
[1] https://www.packtpub.com/en-tw/learning/tech-news/python-lib...
easton
I’ll note that macOS doesn’t necessarily always let you do this the first time, it’ll pop a dialogue saying “hey, you’re cool with this app seeing (your files/other apps files), right?” I wonder if such a thing could be implemented in flatpak.
charcircuit
>You can often only drag 'n drop from ~/Downloads/
Drag and drop could be made to always work since it's being done by a user. Request this feature from your operating system's developer.
genewitch
This is by design do you can't exfiltrate data. Nix is so advanced I feel like a caveman with my openrc Gentoo.
mid-kid
> Still, I think the xz backdoor did not work on NixOS because its unusual non FHS-compliant filesystem structure.
It didn't work on nixos because the build-time check included checking whether the build was being executed in a debian or fedora build environment. This was to avoid suspicious build failures on distros with weird toolchains or incompatible architectures/ABIs/library versions. (The backdoor was a precompiled .o file so rather ABI sensitive)
xp84
I get the potential need for such draconian measures in perhaps, some top secret government installations or something, but gosh that sounds tiring -- a lot like MacOS lately asking me "(AppName) wants to access your Downloads folder, cancel or allow?" when I have just directed it to open a file.
danieldk
Even if you only use trusted applications and they have stringent security policies avoiding supply chain compromises, RCEs are a fact of life. E.g. iMessage vulnerabilities are found all the time and there are probably a lot of vulnerabilities that are not reported because state actors hold on to them. This is the reason why iOS uses application sandboxing and on top of that Blastdoor for iMessage.
Maybe Linux isn't as effected now because it is not very popular as a desktop system. But this issue will have to be addressed as/when Linux becomes more popular. Having networked clients that do image parsing, etc. (usually in C code) without any sandboxing will just lead to mass exploitation, data exfiltration, etc.
The Linux desktop has to move away from the 90ies security model where the internet was relatively safe and attackers would only be after UID 0.
a lot like MacOS lately asking me "(AppName) wants to access your Downloads folder, cancel or allow?" when I have just directed it to open a file.
I don't think it asks that when it goes through a portal (e.g. file dialog)?
nebulous1
Right. The title seemed to be suggesting that the Nix way of doing things might have detected the backdoor. It's actually intending to suggest ways that Nix could be changed in order to detect the backdoor.
jjmarr
I still like the blogpost, because NixOS bills itself as a technical solution to prevent build artifacts that are decoupled from the source code (i.e. not reproducible), and the xz backdoor was hidden in build artifacts.
lolinder
Yeah, it's a good blog post in part because it gets into the details of how it was possible that this vulnerability made it into NixOS, which purports to solve the problem.
Also, I'm not a NixOS critic either: I'm writing this from NixOS! I just don't think there's such a thing as a security cure-all as long as humans are in the loop anywhere.
crote
Sure, but you could achieve the same thing by requiring that the build artifacts are generated by a Github Actions runner.
tomjakubowski
That would only make the xz attack harder, not make it impossible. Just add some step to the action run that fetches blobs from an internet resource you control, then swap out the blobs for malicious ones.
null
dist-epoch
The reason is a bit funny - the NixOS bootstrap downloads it's source code which is a xz compressed tarball.
johnea
> As always theory and reality are different
Thank you very much for citing that! along with highlighting the fact that the exploit was in fact, not detected by reproducible builds prior to other means of discovery.
In recent times, actual reality is often maligned when compared to how people feel about objective reality and how it meshes with their individual value systems.
I have personal values too, but I don't hold the opinion that actual reality is less significant than how I feel about it. It's not a popular perspective 8-/
I've always liked to say: The difference between theory and reality is that, in theory they're the same, and in reality they're not.
I hope the realization that the reproducible builds of NixOS _could_ have detected the xz exploit, but didn't, will lead to new advances in the analysis of those reproducible builds to detect other exploits sooner in the future.
sirspamalot103
[dead]
sirspamalot105
[dead]
sirspamalot112
[dead]
nialv7
I feel the author is a bit tunnel visioned by what happens to happen this time. The Jiatan incident has a sample size of one, it'd be a bit short sighted to think that's the only way it could happen. You can imagine various scenarios where the defenses suggested here will not have worked.
Also I (as a nix user myself) think it's unlikely NixOS would have caught it. As evidenced by the fact that it didn't. (Yeah I realize I just said next time it might happen differently but it'd be foolish to put faith in nix without evidence).
sirspamalot104
[flagged]
sirspamalot106
[flagged]
sirspamalot111
[flagged]
sirspamalot113
[flagged]
sirspamalot110
[dead]
donnachangstein
NixOS is really irrelevant here because the xz backdoor specifically targeted RedHat and Debian. It's equally relevant to say the xz backdoor didn't affect Windows (ironically the backdoor was ultimately found by a Microsoft employee, an oft-overlooked detail).
attila-lendvai
a slightly improved version of NixOS (or Guix) would have automatically caught this backdoor once it reached their repos.
they are relevant.
brabel
A slightly improved version of any OS would've automatically caught it.
attila-lendvai
mechanism, or it didn't happen.
and if any OS, then for windoze please!
sirspamalot110
[dead]
sirspamalot103
[dead]
sirspamalot106
[dead]
sirspamalot111
[flagged]
sirspamalot104
[flagged]
massysett
Article says that distributions should get source code directly from the VCS (for instance Github) rather than the traditional installation tarball.
I don’t see what this solves though. Couldn’t a malicious maintainer simply add binary blobs directly to the source code repository?
The author suggests Github is trusted, as though Github validates code in some way. Which of course it does not.
dwheeler
What this solves is the problem that often "what is reviewed" is different from "the source code used to build the software".
Verified reproducible builds could have countered the xz utils break, SolarWinds Orion subversion, and many others. It's worth doing.
massysett
But would that have solved anything here? The main maintainer was overwhelmed. The back door was obfuscated inside a binary blob there for so-called testing purposes. I doubt anyone was reviewing the binary blobs or the autoconf code used to load it in, and for that matter it’s not clear anything was getting reviewed. Fetching and building straight from GitHub doesn’t solve that if the malicious actor simply puts the binary blob into the repo.
dzaima
Might not be a big chance depending on the project in question, but it's still tons more likely for someone randomly clicking through commits to find a backdoor committed to a git repo than within autogenerated text in a tarball. I click around random commits of random projects I'm interested in every now and then at least. At the very least it changes the attack from "yeah noone's discovering this code change" to "let's hope no random weirdo happens to click into this commit".
A binary blob by itself is harmless, you need something to copy it over from the build env to the final binary. So it's "safe" to ignore binary blobs that you're sure that the build system (which should all be human-written human-readable, and a small portion of the total code in sane projects) never touches.
That said, of course, there's still many options for problematicness - some projects have commit autogenerated code; bootstrapping can bring in a much larger surface area of things that might copy the binary blob; and more.
sirspamalot110
[dead]
sirspamalot106
[flagged]
yarekt
Your comment doesn’t quite make sense: Building from source lets you (and everyone else) inspect the source, while building from provided tarballs means if you compare it to source it’ll be inherently different, as the autoconf process makes changes to the files.
If you’re downloading and executing a binary from github releases, then you’re completely at the mercy of the maintainer (nix only does that with closed source packages)
sirspamalot104
[flagged]
__MatrixMan__
If we want to focus on a thing that NixOS could have prevented, we should focus on the CrowdStrike incident. Being able to boot to yesterday's config because today's config isn't working would've mitigated most of the problems.
XorNot
Except that's a Windows thing where you don't have boot flexibility.
Ubuntu on ZFS can do this as well.
__MatrixMan__
My point is that the lack of boot flexibility caused a lot of problems. If we want to be able to rely on people to get the job done even on days when something is wonky in the bits, then we should give them boot flexibility. NixOS just happens to do it especially well.
As for ZFS... Dealing in filesystem snapshots is comparatively a bit awkward. If you want to recreate that config elsewhere you have to move the whole snapshot rather than just the recipe for building it, and even then it'll break if the system architecture is different on the target machine. If you've got two of them (perhaps labeled "good" and "bad"), you're not going to get anything friendly when you try to diff them, nor is there an obvious way to use things like `git bisect` to reason about where the problem occurred.
None of these things are show stoppers, but working with code that defines some state is just so much easier than anything you get out of a filesystem which happens to remember that state, but can't tell you why it should be the way it is.
chungy
> As for ZFS... Dealing in filesystem snapshots is comparatively a bit awkward. If you want to recreate that config elsewhere you have to move the whole snapshot rather than just the recipe for building it
That wasn't the highlight of the point. It was that you can restore to a known good version of the operating system, effortlessly, regardless of what the operating system is. It could be Ubuntu, Nix, or FreeBSD. Broken OS, select an older snapshot in the boot loader, and you're golden again.
null
sirspamalot113
[flagged]
gsliepen
While NixOS goes a bit further with it, most other distrubtions also compile everything from source, cryptographically verify that the sources they use are not tampered with, and have versioned dependencies between packages. Debian also has reproducible builds.
The problem is just that the build systems did not strip pre-compiled object files before building from source. Even with that fixed, if nobody checks the source code then you can add all the backdoors you want, and there is nothing in NixOS or any other distro that would protect against that.
mcint
Excellent descriptive analysis. Wrong, misleading title, perhaps "technically correct," but at best with a "backdoored" meaning.
It points out the need and use for build-manager tools that go a step beyond union file system layers, but track then enforce that e.g. tests cannot pollute build artifacts. Take a causal trace graph of files affecting files, in the build process, make that trace graph explicit, and then build a way to enforce that graph, or report on deviations from previous trace graphs.
drpossum
If the title is bad I'm not going to waste my time reading the article to see if it has merit. It destroys the credibility of the author before anything they have to say is out of the gate.
brabel
In defense of the author: nobody reads your article if the name is boring (that is my experience at least), which it would've been if they titled it more accurately. That gives incentive to authors to use click-bait titles.
sirspamalot110
[dead]
gbraad
"Could have" means unproven here, and actually... They shipped it
attila-lendvai
?
the article demonstrates how to imorove NixOS (or Guix) to automatically catch any such discrepancy in the future.
IshKebab
Yeah it certainly would have made hiding the backdoor more difficult. But far from impossible. You can always hide backdoors in source code if you want, it just takes more effort to make a plausible bug, and probably has a higher chance of detection.
thdhhghgbhy
If Jia Tan's PR was approved, malicious artifacts could go to github releases just as easily as in a tarball. Struggling to understand the point made about github releases being a security mitigation.
dsp_person
When I did my first github "release" I was shocked that it involved picking files to upload...
mikrotikker
And how many of these files have I downloaded over the years? :sob:
lima
The real answer here is downloading a tarball generated from a tag - those are generated by the GitHub backend.
johnisgood
> the release tarball being different than the source is
> the maintainer provided tarball was honestly generated from the original source code.
How, then? What about differing versions, etc. or has it been mentioned and I just missed it?
Just make sure the generated tarball can be generated from the source code itself, do not exclude anything, git add & commit everything. Can't we do that? We would still have to look at commit history in this case, I believe, and again, he said it himself, it was harmless to the naked eye, so even then, how could we verify? Maybe I don't understand what he meant by verification, but if maintained tarballs are generated from the owner's source code and is not on GitHub (or anywhere else, just a git repo), that is a problem in itself.
Of course there was more to it than just pushing poisoned test files, but still. I do not see how Nix would have prevented it, if the git repo has those test files and with seemingly harmless code (and is reproducible).
Perhaps what we can do is: if an (in)famous project has changed its main lead, then pay closer attention to the commits and check who it is? I don't know, TBH.
Did I misunderstand the article, or am I missing something?
ggggrah
I think it's a vuln to think of OSS in terms of 'a community'. It's an abstract thought construct that does not represent reality (though it helps to make sense of it in a rather specific manner) xz happened because of the absence of community. It could happen inside this abstract thought of a community as well but here it did not.
xz targeted deb and rpm. The vast majority of what is facing the world.
Nix did not stop it.
I believe this article feeds the possible vuln rather than prevent it.
Note that NixOS and reproducible builds did not detect the xz backdoor, and in fact NixOS shipped the malicious builds of xz (though they didn't do anything because the malware didn't target NixOS):
> I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users.
As always theory and reality are different, and the thing that made xz possible was never a technical vulnerability with a technical solution—xz was possible because of a meatspace exploit. We as a community are very very bad at recognizing that you can't always just patch meatspace with better software.