Nix Derivations, Without Guessing
65 comments
·April 7, 2025tikhonj
setheron
Professionally for 10 years ? I would like to hear more.
I remember Target funded some earlier Nix work like lorri.
Interestingly, I found operating at this level to have been a necessity if I wanted to fix or write a derivation. Otherwise I would face random errors and be left clueless as to what the problem was.
Writing a derivation builder (nix pill) was illuminating.
tikhonj
Yep, I was one of the people who originally started using Nix on that particular team at Target back in 2016. I wasn't the only person interested in Nix but, if I'm recalling correctly, I was the person who first set it up and supported our Nix environment on Linux and macOS for the first year or two. After we scaled up a bit we hired some more Nix-focused people and some consultants (for stuff like lorri), so I stopped having to do much Nix stuff day-to-day.
That was a really special team. Nobody "let" me use Nix, we just did it because it made sense to us, and I went out of my way to support it internally not because I felt pressured to do it, but because I was enthusiastic about making it work. The culture there reminded me of this story[1] about getting Python into production at Goldman—and it turns out that the VP who built the supply chain team at Target started his career in strats at Goldman and was heavily inspired by the same Armen from that story :)
[1]: https://news.ycombinator.com/item?id=29104401
After leaving Target, I also used Nix for my own work setup at CX Score. That was a seed-stage startup and I didn't want to push Nix on anybody else, but it made my own work setup much nicer.
Most recently I moved to Mercury where we use Nix at a pretty big scale (several hundred developers). We have a team that handles most of it so I haven't had to do much myself, but I've helped a few people work through Nix issues just because I could :) I'm also one of the weirdos who uses NixOS on my work laptop instead of getting a macbook.
And, in parallel, I've been using NixOS on all my personal machines ever since I first discovered it. That was relatively shortly before joining Target, so, if I had to guess, late 2015 or early 2016. Coming up on 10 years shortly, even though it really does not feel like it...
pkulak
I still have a Thinkpad from 2015 for work that I run NixOS on. Company policy has since changed to Mac only, but I think I’ve got at least another decade on this ol’ gal. I suppose I could use Nix Darwin, but it wouldn’t be the same.
nylonstrung
That's so cool!
I love NixOS and was excited to see Mercury using it. What made Mercury choose it and what have the advantages been for a company like that?
danieldk
Interestingly, I found operating at this level to have been a necessity if I wanted to fix or write a derivation. Otherwise I would face random errors and be left clueless as to what the problem was.
I have used Nix on and off for work since 2018. We are using Nix at my current employer for two projects. I agree that if you want to go beyond some simple dev shells, for any realistically complex project you really need Nix Pills-level understanding (+ a good idea of how things work in nixpkgs).
It's a two-edged sword - the learning curve is very steep, but once you have that level of understanding, it's a superpower. We heavily use CUDA/ROCm machine learning stacks and Nix makes it much easier to ensure everyone is using the same development environment. Switching between different branches that may have different sets of dependencies/dependency versions always works. Also, it's much easier to stack custom patches on dependencies and caching avoids all the rebuilds.
gadtfly
Completely unrelated to anything I'm just taking this as an opportunity to yell this into the void while nix is on topic:
I have a theory that a problem for Nix understanding and adoption out of all apparent proportion is its use of ; in a way that is just subtly, right in the uncanny valley, different from what ; means in any other language.
In the default autogenerated file everyone is given to start with, it immediately hits you with:
environment.systemPackages = with pkgs; [ foo ];
angra_mainyu
To be fair, that is not problematic at all and most definitely not what I think is the issue with Nix adoption/learning curve.
Personally, it's the fact that there are 57698 ways of doing something and when you're new to Nix you're swarmed with all the options and no clear way of choosing where to go. For example, the docs still use a shell.nix for a dev shell but most have moved to a flake-based method...
I always recommend starting with devenv.sh from the excellent Domen Kozar and then slowly migrating to bare Nix flakes once you're more accustomed.
eru
> How is that supposed to read as a single expression in a pure functional language?
Well, in Haskell the following is technically single expression:
do with pkgs; [ foo ];
But extreme nerdery and nit-picking aside, I agree that the choice of syntax in nix unfortunate here.
mplanchard
Personally I think a bigger problem is the lack of discoverability of things in nixpkgs, which hits you as soon as you start writing anything remotely non-trivial. "Things" here means functions in 'lib', 'builtins', etc., as well as common idioms for solving various problems. I think this combines with the language's lack of types to make it hard to know what you can write, much less what you should write.
A language server with autocomplete and jump-to-definition would go a long way to making nix more accessible. As it stands, I generally have to clone nixpkgs locally and grep through it for definitions or examples of things related to what I'm doing, in order to figure out how they're used and to try to understand the idioms, even with 4 years of running NixOS under my belt and 3 years of using it for dev environments and packaging at work.
418tpot
I agree the syntax isn't perfect, but in case you're actually confused there's really only 3 places where semicolons go, and I would argue that two of the places make a lot of sense— as a terminator for attribute sets, and a terminator for `let` declarations.
Unfortunately it is also used with the somewhat confusing `with` operator which I personally avoid using. For those of you who aren't familiar, it works similar to the now deprecated javascript `with` statement where `with foo; bar` will resolve to `bar` if it is in scope, otherwise it will resolve to `foo.bar`.
chriswarbo
I actually prefer `with`, since it fits better with the language:
- It uses `;` in the same way as `assert`, whereas `let` uses a whole other keyword `in`.
- It uses attrsets as reified/first-class environments, unlike `let`, which lets us do `with foo; ...`.
- Since it uses attrsets, we can use their existing functionality, like `rec` and `inherit`; rather than duplicating it.
I've been using Nix for over a decade (it's even on my phone), and I've never once written a `let`.
(I agree that the shadowing behaviour is annoying, and we're stuck with it for back-compat; but that's only an issue for function arguments and let, and I don't use the latter)
wiktor-k
> now deprecated javascript `with` statement where `with foo; bar` will resolve to `bar` if it is in scope, otherwise it will resolve to `foo.bar`.
Technically, in JavaScript it's `with (foo) bar`.
Source: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
pkulak
Your p,k,g,s keys must be worn to nubs.
chombier
What I found hard with Nix is the sheer amount of things I had to get familiar with before it started to really click:
- nix, the command-line tool
- nix, the language
- nixpkgs with the general API/idioms (overriding, overlays)
- individual nixpkgs packages that sometimes deviate from common practices
- flakes (which I haven't properly looked into yet)
The Nix pills series [1] and nixpkgs documentation [2] do help a lot, but that is quite a lot to process.
angra_mainyu
Controversial, but I found just going through the docs, attempting to write my own flake, throwing it at ChatGPT and asking it to review it and give me feedback was enough to get started.
I never ask it for code, just to review code - keeps me sharp and makes the iteration cycles faster.
pxc
ChatGPT is generally pretty bad at writing Nix code IME, and has a much greater tendency to hallucinate than when writing code in other languages.
But it's better at advice and review than writing code.
And if you want to ask it about Nixpkgs conventions or the behavior of some Nix codebase, it does much better if you dump the relevant source files in the chat for it to consume.
It's important with Nix to seriously understand what you're doing, so I think the way you've described is the most appropriate way to use an LLM with Nix. Use it to point you to docs, ask it questions about some code and then read the code closely, ask it which docs to read, ask it to point you to example repos on GitHub. If you use it this way it can actually be helpful.
If you have a longer time horizon, interacting with real community members is even better, and of course can also enrich the corpus for LLMs like ChatGPT. Discourse.nixos.org has a lot of smart and helpful people whose insight is more reliable than the advice of an LLM, so it's worth visiting as well.
woile
they are a lot of things, but it's hard to avoid otherwise. Overall I'm glad it's relatively cohesive how it all works.
kennethallen
I walk away from every article on or attempt to use Nix more mystified
p1necone
Yeah Nix seems like an insanely useful concept (declarative, reproducable, diffable, source controllable definitions of linux environments). But actually using it is a nightmare.
I think there's some point on the spectrum between "commit your dotfiles to git" and Nix that would be useful, I don't know what it is though. Containerization is kinda like this, but they're entirely imperatively defined rather than declarative and I wouldn't really want to use a docker container as my main system environment.
tombert
I run NixOS on every computer I'm allowed to install it and I really don't think it's hard to use, just different. Adapting to a new workflow is hard, but I don't know that NixOS is intrinsically more difficult than any other Linux.
I wouldn't want a docker container as my main environment, but I do like having NixOS managing my main environment for a few reasons.
First, the declarative nature of everything makes it clear and easy to know what is actually installed on my computer. I can simply look at my configuration file and see if a program is installed, and if it's not. If I want to uninstall something, I delete the program from the configuration.nix and rebuild. This might not seem insignificant, but to me it's game changing. How many times have you uninstalled things in Ubuntu or something and had a ton of lingering transitive dependencies? How many times have you been trying to debug a problem, end up installing a million things, and then painstakingly having to track down every unnecessary dependency and transitive dependency that you ended up installing, only to miss something for months? Maybe most people here are better at this than I am, but these things happened to me all the time.
Second, the declarative nature of NixOS makes snapshotting trivial. Again, this is game-changing for me, and it makes fixing stuff on my computer more fun and less scary. If I break something, all I have to do is reboot and choose the last generation, then fix it.
This might not seem like a big deal, and again maybe for people smarter than me it's not, but for me it completely changed the way I deal with computers. When I first started using Ubuntu, when I would do something like break the video driver or the WiFi driver, I would end up having to nuke the system and start again, because I would get into a state where I didn't know how to fix it. I probably could fix these things now, I've been doing this stuff for awhile, but even still, it's nice to be able to not ever have to worry about getting into a state like that.
ChocolateGod
My biggest gripe with Nix (from real world experience) is that my .nix files randomly break due to changes and I have to spend my time going through Github commits to see what changed in the settings I used to fix it.
That and when things do error, the error messages may as well be generated from /dev/random
dymk
What you describe with building and rebuilding and keeping a clean environment is exactly what I use Docker containers for, eg devcontainers. I know it’s not reproducible in the same way, but the learning curve is so, so much lower for something 90% as good and with much more documentation and online support.
sepositus
> I run NixOS on every computer I'm allowed to install it and I really don't think it's hard to use, just different.
I work at a place that uses Nix for almost everything. Despite that, most developers do not like it (and usually create tickets asking the "experts" to fix things). The above quote is basically exactly what the experts are always telling the developers. That, along with "you just need to try harder." As if it's not valid that someone can think Nix isn't ergnomic and often sucks to use.
I personally don't mind it all that much, although nowadays I just use it for home-manager. But I've seen people go from disliking it to hating it because of the way some experienced Nix people have treated them.
eru
> [...] but they're entirely imperatively defined rather than declarative [...]
The conceptual problem with Docker isn't imperative vs declarative. It's that Docker doesn't even try to be reproducible. Executing the same Dockerfile twice doesn't necessarily build you the same container.
(Imperative vs declarative is still an interesting problem to think about, it's just independent of reproducibility in the abstract.)
darthrupert
I'm also one of the people who have tried Nix(OS) a couple of times and found it too much of a hassle, but nightmare is exaggerating a bit, I feel.
Nix's strength and weakness is that it wants to take over everything, and if you want to do something without it, you might be in a world of pain. And after doing more or less standard unixy things for 20+ years, it's difficult to hand over control to a new thing like that.
chubot
I honestly wonder why there are such divergent opinions between say git and Nix? (I've never used Nix, but I use git all the time) Is Nix so much harder to use than git?
git also has a clear model (the Merkle tree of file and directory nodes) but a famously unfriendly UI (git checkout does 5 or more unrelated things)
Some people do not like git, but I get the feeling that most people just use it, and get on with their day.
Why the difference with Nix? Maybe because building packages is inherently slower. Whereas you can quickly get yourself intro trouble with git, but you can also quickly get out of trouble (rm -rf, git clone)
Maybe Nix is more stateful? Although the git index/staging area I find to be a fabulously odd piece of state, and honestly breaks all the rules when I think about CLIs
Also Nix does rely on a very big global repo, whereas git doesn't
It also seems that Nix's model is less clean, and perhaps doesn't match the problem domain as well ... there are disagreements on whether it is "reproducible", etc.
Or maybe it's just a harder problem
pxc
You may not use Nix, but some pretty cool Nix tools use your work. :)
Just today I was working on some integrations with a runtime dependency resolver for shell programs that uses OSH for parsing on the job. We use Nix to manage our development environments, and we use it to make some small wrappers and tools written in Bash into something portable and reproducible that we can include in development environments that run both on Linux and macOS.
Historically we've just included them inline in our Nix files, but thanks to resholve, we're switching to a nicer system so that they live in separate files that are more pleasant to edit. The "source code" of the scripts lives in the repo as normal bash scripts, but when they get built into the development environment, all command invocations get automatically replaced with hard-coded references to the paths of the relevant executables in the Nix store, the shebang gets pinned to a specific bash version in the same way, and they also get run through ShellCheck. Now we have not only a really nice and quick way to define portable wrappers in our Nix code, but a sane way to manage longer scripts without any portability issues.
So thanks for your cool shell and associated tools and libs!
> Or maybe it's just a harder problem
Fwiw, I think this is true.
> Some people do not like git, but I get the feeling that most people just use it, and get on with their day.
I think one of the other devs on my team probably feels this way. He thinks it's conceptually cool but is somewhat horrified by the complexity and UI. He makes simple uses of it in ways that have precedent on our team, but never really dives in.
> Why the difference with Nix?
Nix didn't have a celebrity author to spur adoption early on, in some ways it can be slow, and I think maybe it isn't as much better than entrenched alternative stacks than Git was as opposed to SVN. The pain of SVN was very acute for the average developer. I'm not sure that any of the pains of dependency hell, stateful configuration management, distribution of portable executables, etc., are quite as acute for the average operator or developer as that. People who feel Nix makes their professional lives easier tend to have come to it after their career has inflicted more specialized pain upon them.
Using Git also doesn't generally (ever?) involve writing code in a Turing-complete language, but to make the best use of Nix, you do have to do that. The paradigm of that language is not very mainstream, either, and although it's generally suited to its domain imo, it certainly has some warts.
amarshall
Git doesn’t really care at all about the content. Nix does. Really they’re not reasonably comparable at all.
abecedarius
Maybe it's better now, but what I ran into in trying twice is that if you're not into installing by "curl | sh", then trying to build from source was an awful experience. It had out of date instructions for installing a whole lot of dependencies. I'd figure out one problem only to run into another, and another. Gave up both times, a few years in between.
rgoulter
> I honestly wonder why there are such divergent opinions between say git and Nix?
Nix is:
1. difficult & different, where similar-ish tools are straightforward & familiar, (e.g. cloud-init, ansible, vs NixOS; or devcontainers using Dockerfiles, vs nix shells).
2. demands a significant amount of understanding, even for tasks which you'd expect to be easy.
e.g.: git is difficult, but you can get by with rote memorizing 5 commands (& copy-pasting the repo if you mess up). Emacs is difficult, but you're not required to use it. Haskell can be difficult to work with, etc.
I'd say that running `direnv allow` & using nix that someone else has written is unlikely to be difficult. But, having to write your own Nix code can be quite high friction.
> It also seems that Nix's model is less clean
I think nixpkgs is cluttered with organic mess of inconsistent designs.. but I think there's also friction where Nix's ideal package is built with `./configure && make && make install`, and many packages aren't quite that clean.
xorcist
> some point on the spectrum between "commit your dotfiles to git" and Nix
That would be configuration management tools such as Salt/Puppet/Ansible/Chef.
They were popular ten years ago, and gained a lot of exposure as the devops movement gained ground, but they never stopped being useful.
Having your non-running state defined declarative is powerful, and if you can define a single source of truth for entire distributed systems, that suddenly makes you able to reason about state on whole systems.
XorNot
What I can't currently figure out with Nix though is how I kill off dependency explosions though.
I want to reach into a big tree of derivations and force library and compiler versions to a single common one so we don't, for example, have 6 rust compilers being used.
o11c
I'm pretty sure the best middle ground that currently exists is the various "immutable, snapshot to upgrade" distros out there.
danieldk
Yes and no. It has some of the same advantages, but not the declarative definition of a system.
For me (even though I use NixOS the desktop) there is a difference between desktops and servers. I can set up a desktop pretty quickly - install a bunch of flatpaks, checkout a bunch of dotfiles and I don't do it often.
Also, popular desktops like Fedora tend to be far more polished than the NixOS desktop, which has all kinds of glitches. Just to name two current ones: (1) gdm login will fail if you are too fast and log in before WiFi is up (usually you are thrown back in gdm, sometimes the session freezes up completly); (2) fwupd firmware updates usually fail.
On the other hand, on servers and remote development VMs, the setup work is annoying because I spin up/down machines far more frequently and managing them as pets gets old pretty quickly. So NixOS is much nicer, because you can have a system up and identical in 5 minutes. You could of course approximate it with something like Ansible on non-NixOS.
Though, I think the differences will become smaller since Fedora-based immutable systems will switch from OSTree to bootable containers soon [1].
Of course, you can use Nix on another immutable distro than NixOS.
[1] https://docs.fedoraproject.org/en-US/bootc/getting-started/
pxc
Depends on your use cases. I use Nix all the time at work but I don't use NixOS there at all. (I'd like to, but there are barriers and it's not a priority.) Distros like that don't address my use cases at all.
amarshall
For what it’s worth, nothing in this article is really necessarily for general usage of Nix, as the derivation format is mostly abstracted-away, like how the OCI image format is irrelevant to everyday authoring of Dockerfiles.
drowsspa
Yeah, it's like those famous posts comparing monads to burritos or something
setheron
What a fun read. Thanks for the nice call out on my blog.
If you're also interested I (Farid) also had a follow up where I follow up on how the hashes are calculated. (Using the error to get the hashes also bugged me)
I use it to also create vanity hashes :)
Joker_vD
Ah, don't you just like it when projects that use hashsums, calculate them in such a way that you can't actually recalculate them on your own? And when you start digging, you find not only that it uses the same basic design of XML-DSig while barely escaping its fatal flaws, it also uses some bizarre data encodings nobody else uses for anything. And then the resulting hash is not even the truncation of the actual hash, it has an additional (again, entirely undocumented) strange post-processing step for unspecified reasons.
pveierland
Nix derivations are pretty neat!
I've been building a Nix store navigator for MyNixOS v2. It can help you get a feel for how derivations are connected in Nix:
https://v2.mynixos.com/nix/store/lsk1c4v03y4lmpxdcwal99nm5nw...
PS: The controls to the upper right can be used to limit what is downloaded.
klysm
What a bizarre and arcane incantation
danlitt
c.f. guix, where the command is simply `guix hash`.
Mayora13
Now we are really interested in why they do this—probably for backward compatibility with a 20‑year‑old implementation. It’s not messy, just particular.
klysm
Is there any technical pathway for changing the underlying hash function? Similar to how git is kind of stuck on SHA1, what would it take to get this onto SHA512?
chriswarbo
I think it could be done in a backwards-compatible way, since existing hashes fit a known pattern; e.g. (silly example) store paths using SHA512 could be distinguished using prefix like 'sha512_'. One problem would be cache misses and duplication of files.
I'm not completely up to date on Git's hash situation, but it seems to me like SHA256 is usable; but only for new repos. I imagine lots of existing infrastructure assumes SHA1 hashes though, and would break for silly reasons like different lengths.
Relatedly, Nix has a new feature called `git-hashing` which can use Git tree hashes to validate fixed-output derivations, rather than requiring two separate hashes; e.g. the following `fetchTreeFromGitHub` function uses a single `tree` argument for a Git tree ID, rather than needing separate `rev` and `hash` arguments:
{ owner, repo, tree }: (pkgs.fetchFromGitHub {
inherit owner repo;
rev = tree;
hash = builtins.convertHash {
hash = tree;
hashAlgo = "sha1";
toHashFormat = "sri";
};
}).overrideAttrs (_: { outputHashMode = "git"; })
null
For context, this particular article is a cool deep dive into how Nix works, but it doesn't represent what using Nix + Nixpkgs is like in practice. I've been using Nix personally and professionally for almost 10 years now (yikes has time passed quickly!) and I have never needed to operate at the level of derivations like this.