How Jane Street accidentally built a better build system for OCaml
48 comments
·January 30, 2025tombert
I have to respect Jane Street for proving that HM-typed functional programming can be production-ready and extremely fast when given enough love.
I still occasionally hear things about how the more academic-styled functional languages can't work in production, but Ocaml shows that it absolutely can work, even with high performance requirements.
kyawzazaw
> HM-typed functional programming
Mercury also uses Haskell for their backend https://mercury.com/
tombert
Also very cool!
I was fortunate enough to work at Jet.com before Walmart completely destroyed it, which was an F# shop. I really liked it, and I never really felt "limited" by it.
The stuff I was working on didn't have nearly the same requirements as high-frequency trading like Jane Street does though. I never did any super low-latency stuff with F#, so it's tough for me to say how well it fair with that kind of environment.
Lyngbakr
They did an episode of Signals and Threads (Jane Street's technical podcast) about their build systems back in 2020¥. Highly recommended, as are the other episodes, especially The Future of Programming with Richard Eisenberg° and What is an Operating System? with Anil Madhavapeddy§.
¥https://signalsandthreads.com/build-systems/
triyambakam
Very interesting footnote system. Is it just for fun and aesthetic or have a deeper origin or purpose?
dgfitz
The only thing worse than a complex codebase is a complex build system. I think CMake is an abomination. “Hey go learn this new language to compile this other language!”
munificent
The only thing worse than a complex build system is a simple build system that can't build what you need to build. :(
dgfitz
This means the architecture is also bad, and that is what needs to be fixed.
gpderetta
> CMake is an abomination. “Hey go learn this new language to compile this other language!”
"go learn this awful new language"
tom_
I don't mind CMake. It's awful, but I'm already using C or C++, so the bar is set pretty low. It does do a few things right, and at least some of the things it does wrong are manageable.
I'll eat at least a bit of shit if it means I can get more than one platform's-worth of build process out of a single set of human-editable configuration files.
queuebert
Have you tried meson? I used it once many years ago, and it seemed slightly better.
kevin_thibedeau
The ideas embodied in modern target-oriented CMake are worthwhile. It's all hobbled by a dreadful script language but that could be improved with sufficient motivation.
cmrdporcupine
It's hobbled not just by that but by the fact that it's not the thing doing the building. It's a build system for build systems, and suffers from brittleness and confusion as a result.
tom_
I think what annoys me about it the most is that for every good idea in there there's one piece of tedious crap you have to work around, one actively bad misfeature that doesn't work properly, and one terrible scripting language misfeature.
(And two minor annoyances that people determined to hate CMake will never shut up about. But then, if you're determined to hate something, there are worse things than CMake to do it to! So I can't be too critical.)
codr7
I spent several years in Haskell land a while back, my first functional experience after Lisp.
Ocaml never clicked for me, I have a rare form of semicolon allergy and Haskell just looked a lot nicer to me.
But then I recently tried Reason and enjoyed it A LOT, so everything Ocaml is suddenly interesting.
cwmoore
“If your intention is to compile or interpret files as scripts, double semicolons can and should be avoided when writing in OCaml. Leaving them does not raise errors, but they are useless. The compiler tolerates them to allow copy-paste from UTop to a file without having to remove them.“ https://ocaml.org/docs/toplevel-introduction#loading-librari...
https://reasonml.github.io/ looks cool, OCaml with javascript.
cmrdporcupine
It's funny because the thing that always turned me off from Haskell was that it feels like a stream of whitespace and identifies without good visual cues as to structure.
I'm sure once you get the zen of it, it's fine. Like Lisp, I guess, you learn to think in its structure. But looking at a screenful of Haskell to me is intimidating.
Of the bunch I found SML/NJ to be the most readable.
whimsicalism
i wish the reason syntax was the main one
n144q
I wonder if anyone has used Jane Street's open source OCaml libraries and tools for somewhat serious projects.
I took a brief look at these things, and my impression is that their stuff isn't "ready" for anyone outside Jane Street, even though they put a lot of effort in building the ecosystem and open source their code.
giraffe_lady
I do! Their standard lib replacement was used on the project where I originally learned ocaml, so I have just always used it. You have to be fairly sophisticated at reading fn signatures to find things sometimes but otherwise no complaints with it. It has some very nice features too for example convert functions to/from sexps for all base types can save you a ton of time if you know it's there and plan for it.
I've used some of their other libraries too, their logging and unit test ppx are common maybe even de facto standards as much as the ocaml world has such a thing. I've also used, off the top of my head, their code formatter, one of their test frameworks, their implementations of some advanced data structures.
Sometimes you do run into one like the other commenter said, where that shit just does not work. It depends on an undocumented something they shipped separately, or needs a secret bit of config or whatever. These aren't malicious, I open a ticket and come back in a year or two often they'll be working.
It's not zero frustration but I appreciate their approach of just throwing everything over rather than spending more resources testing and polishing fewer releases. Their code quality is generally very high and even if I can't get something working directly, it provides a rigorous & vetted example implementation.
Kuinox
I tried to use their magic-trace tool but I get at least 4 differents crash when using it on trivial dotnet programs.
zokier
kinda surprising that it still makes sense to develop your own new build system, and then do the migration to it, instead of adopting an off-the-shelf one like buck2 (or bazel/etc...). Not saying that these fancy build tools are easy to adopt, but compared to building your own? is there something remarkable that makes dune much different?
wk_end
Without getting into any specifics of it - I'm sure there's people with much more experience with these tools who can comment - I'll point out that neither buck nor bzl existed when JS decided to start building their own tool in 2012. Bazel's first release was in 2015, Buck's was in 2013.
JS does have a bit of a NIH culture, but I'm not sure if that was really at play here. There just...weren't very many good build tools available at the time, particularly for a company using an unorthodox tech stack.
zokier
> I'll point out that neither buck nor bzl existed when JS decided to start building their own tool in 2012. Bazel's first release was in 2015, Buck's was in 2013.
But Dune started (according to this blog post) in 2016 and JS started seriously improving and adopting it last year. So to me Jenga sounds like a reasonable step in 2012, but pouring significant effort into migrating from Jenga to Dune (and improving Dune) in 2024 sounds more weird
yawaramin
Jenga and dune are the same thing, it was just renamed.
mhh__
I don't know ocaml particularly well but my understanding is that only with buck2 (of those mentioned) do you have a build system with the semantics to actually compile ocaml properly.
In particular I was under the impression one needed to be able to run ocamldep before hand (or compile twice) - buck2 can do this, bazel needs hacks iirc.
rwmj
I loath autotools with a passion, and it's awkward to set it all up, but it handles OCaml code just fine. We use it for multi-language projects (including OCaml) like nbdkit & libguestfs.
wiseowise
Dune is the single best name of a build system out there, nothing even comes close to.
708733454927516
Indeed.
"He who controls the [build system] controls the universe."
fire_lake
What percentage of JS programmers actually work on this stuff I wonder?
firesteelrain
Man I must be under a rock. Never heard of Dune!
Cmake mostly
xiaodai
but why?
rwmj
I respectfully have to disagree. As Fedora packager for OCaml packages, dune's configuration is yet another one-off, with a weird Lisp syntax, hard to understand, difficult to patch. I'd far rather people stuck to using more familiar Linux build systems. As an upstream OCaml developer, the whole thing falls down the minute you need to integrate other programming languages into your build (or OCaml code into a code base written in another language).
(This rant more or less equally applies to other language-specific build systems.)
munificent
> more familiar Linux build systems
"Familiar" is not a property of any system. It's a relation between a system and a user.
Some Linux build systems maybe be more familiar to some users, but will be less familiar to others. When picking a build system, you can't just look at the system itself and declare it familiar or not.
It's not even enough to look at the total number of users familiar with some thing. Hindi is one of the most familiar languages in the world, but you're probably gonna have a bad time if you use it for the menu in a cafe in rural Texas.
You have to look at your actual cohort of users (and potential future users) and see what's familiar to them. This is one of the key reasons why usability is actually a deeply hard problem. So much of usability hinges on familiarity, but familiarity is a human-specific highly variable property.
nicoburns
As a user of Linux, I wish Linux distributions would abandon their build systems which really only work well for C and maybe C++, and also only work on one platform (not even the whole of linux!) meaning that packaging has to be duplicated for every OS, and stuck to using more familiar language-specific build systems that work everywhere.
(this rant more or less equally applies to all os-or-distro-specific build systems)
---
This rant is only semi-serious. I do see some value in the Linux distribution style packaging. In particular, I do sympathise with the need to do cross-language builds. But goodness are they a pain to work with, and probably the biggest barrier to me shipping software on Linux.
My hope is that eventually an evolution on build systems like bazel/buck2 will lead to a truly universal build system that is both cross-platform and cross-language. But unfortunately it doesn't look like it's coming soon.
Hasnep
If you only care about Linux and MacOS then Nix can be that universal cross language build tool.
yawaramin
> more familiar Linux build systems
What's a 'familiar Linux build system'? make?
blobbers
They likely mean 'generally used with other languages'. make is an example.
UncleOxidant
I haven't been doing much OCaml in recent years, but I do sometimes check in to see what's going on in OCaml-land, and it seems like dune is what everyone is using now - what are the alternatives?
mhitza
Using, probably, to be deprecated ocamlbuild, with _tags file which is another made up syntax.
But I'm a new OCaml user, and actively starter using ocamlbuild because dune added layers of indirection really tripped me up at first.
lmm
> This rant more or less equally applies to other language-specific build systems.
And yet those language-specific build systems are overwhelmingly winning, in pretty much every language.
> As Fedora packager for OCaml packages,...
I honestly think traditional Linux packaging is in the wrong here and the problems are essentially self-inflicted (not in the sense that individual maintainers are doing something wrong, but in the sense that the policy that traditional Linux distributions are following is inherently unsustainable. It's designed for a pre-CPAN world)
> As an upstream OCaml developer, the whole thing falls down the minute you need to integrate other programming languages into your build (or OCaml code into a code base written in another language).
True up to a point, but frankly the worst case is falling back to a terrible C-style build, and "always do terrible C-style builds in case you need to integrate with C code" is not a proposition that has much appeal.
Much as I wish the whole world would standardise on Maven or Cargo, I can't see a realistic path to there without first eliminating C, because the C people are never going to agree to follow a standard for package repositories.
null
Dune's crash message:
> I must not segfault. Uncertainty is the mind-killer. Exceptions are the little-death that brings total obliteration. I will fully express my cases. Execution will pass over me and through me. And when it has gone past, I will unwind the stack along its path. Where the cases are handled there will be nothing. Only I will remain.