Jemalloc Postmortem

I worked on the Superdome servers back in the day. What a weird product. I still can't believe it was a profitable division (at my time circa 2011).

HP was going through some turbulent waters in those days.

Itanium did its most most important job: it killed everything but ARM and POWER.

Sunk by the Great Itanic ?

SGI and HP! Intel should have a statue of Rick Belluzzo on they’r campus.

Why was the sinking of SGI great?

Thanks for sharing the insight!

As I observed when I was at Google: tcmalloc wasn't a dedicated team but a project driven by server performance optimization engineers aiming to improve performance of important internal servers. Extracting it to github.com/google/tcmalloc was complex due to intricate dependencies (https://abseil.io/blog/20200212-tcmalloc ). As internal performance priorities demanded more focus, less time was available for maintaining the CMake build system. Maintaining the repo could at best be described as a community contribution activity.

> Meta’s needs stopped aligning well with those of external uses some time ago, and they are better off doing their own thing.

I think Google's diverged from the external uses even long ago:) (For a long time google3 and gperftools's tcmalloc implementations were so different.)

Everything from Google is an absolute pain to work with unless you're in Google using their systems, FWIW. Anything from the Chromium project is deeply intangled with everything else from the Chromium project as part of one gigantic Chromium source tree with all dependencies and toolchains vendored. They do not care about ABI what so ever, to the point that a lot of Google libraries change their public ABI based on whether address sanitizer is enabled or not, meaning you can't enable ASAN for your code if you use pre-built (e.g package manager provided) versions of their code. Their libraries also tend to break if you link against them from a project with RTTI enabled, a compiler set to a slightly different compiler version, or any number of other minute differences that most other developers don't let affect their ABI.

And if you try to build their libraries from source, that involves downloading tens of gigabytes of sysroots and toolchains and vendored dependencies.

Oh and you probably don't want multiple versions of a library in your binary, so be prepared to use Google's (probably outdated) version of whatever libraries they vendor.

And they make no effort what so ever to distinguish between public header files and their source code, so if you wanna package up their libraries, be prepared to make scripts to extract the headers you need (including headers from vendored dependencies), you can't just copy all of some 'include/' folder.

And their public headers tend to do idiotic stuff like `#include "base/pc.h"`, where that `"base/pc.h"` path is not relative to the file doing the include. So you're gonna have to pollute the include namespace. Make sure not to step on their toes! There's a lot of them.

I have had the misfortune of working with Abseill, their WebRTC library, their gRPC library and their protobuf library, and it's all terrible. For personal projects where I don't have a very, very good reason to use Google code, I try to avoid it like the plague. For professional projects where I've had to use libwebrtc, the only reasonable approach is to silo off libwebrtc into its own binary which only deals with WebRTC, typically with a line-delimited JSON protocol on stdin/stdout. For things like protobuf/gRPC where that hasn't been possible, you just have to live with the suffering.

..This comment should probably have been a blog post.

Wow. That does sound quite unpleasant.

Thanks again. This is far outside my regular work, but it fascinates me.

I’ve successfully used LLMs to migrate Makefiles to bazel, more or less. I’ve not tried the reverse but suspect (2) isn’t so bad these days. YMMV, of course, but food for thought

This. When step one is "install our weird build system," I'll immediately look for something else that meets my needs. All build systems suck, so everyone thinks they can write a better one, and too many people try. Pretty soon you end up having to learn a majority of this (https://en.wikipedia.org/wiki/List_of_build_automation_softw...) to get your code to compile.

malloc in uCRT just calls HeapAlloc, though? You can see the code in ucrt\heap\malloc_base.cpp if you have the Windows SDK installed.

Programs can opt in to the _segment_ heap in their manifest, but it’s not necessarily any faster.

But I'd like to know exactly what that means.

How can I find out if Facebook's focus is aligned with my own needs?

React, PyTorch, and RocksDB are all extremely significant. Not to mention them being one of the biggest contributors to the Linux kernel.

There’s something to that but it is victim blaming if you’re not acknowledging the larger trends. There are a lot of places whose MBAs are attending the same conferences, getting the same recommendations from consultants, and hearing the same demands from investors. The push against remote work, for example, was all driven by ideology against most of the available data but it affected a huge number of jobs.

> Both jemalloc and tcmalloc evolved and were refined in antagonistic multitenant environments without one overwhelming application. They are optimal for that exact thing.

They were mostly optimised on Facebook/Google server-side systems, which were likely one application per VM, no? (Unlike desktop usage where users want several applications to run cooperatively). Firefox is a different case but apparently mainline jemalloc never matched Firefox jemalloc, and even then it's entirely plausible that Firefox benefitted from a "selfish" allocator.

It's possible that they were referring to something specific about their platform and its system allocator, but like I said it was an anecdote about one engineer's statement. I just remember thinking it sounded fair at the time.

> I could easily imagine it being left out of glibc because [...]

... its license is BSD-2-Clause ;)

hence "political"

Why is that? Doesn't Linux use SIMD for the crypto operations?

jemalloc is used enough at Amazon that it would make sense for them to maintain it, but that's not really their style.

“Software is just done sometimes” is a common refrain I see repeated among communities where irreplaceable software projects are often abandoned. The community consensus has a tendency to become “it is reliable and good enough, it must be done”.

It's been 14 years since THP got added to the kernel[1], surely we're past calling that "recent" :)

https://www.kernelconfig.io/config_transparent_hugepage

Another example is rseq (which was originally implemented for tcmalloc).

This number of years depending on your allocation profile could be something like -10 years easily. New allocators constantly crop up

Presumably then the performance impact of any switch will be positive.

glibc does NOT return memory to the OS just fine.

In my experience it delays it way too much, causing memory overuse and OOMs.

I have a Python program that allocates 100 GB for some work, free()s it, and then calls a subprocess that takes 100 GB as well. Because the memory use is serial, it should fit in 128 GB just fine. But it gets OOM-killed, because glibc does not turn the free() into an munmap() before the subprocess is launched, so it needs 200 GB total, with 100 GB sitting around pointlessly unused in the Python process.

This means if you use glibc, you have no idea how much memory your system will use and whether they will OOM-crash, even if your applications are carefully designed to avoid it.

Similar experience: https://news.ycombinator.com/item?id=24242571

I commented there 4 years ago the glibc settings MALLOC_MMAP_THRESHOLD_ and MALLOC_TRIM_THRESHOLD_ should fix that, but I was wrong: MALLOC_TRIM_THRESHOLD_ is apparently bugged and has no effect in some situations.

A bug I think might be involved: "free() doesn't honor M_TRIM_THRESHOLD" https://sourceware.org/bugzilla/show_bug.cgi?id=14827

Open since 13 years ago. This stuff doesn't seem to get fixed.

The fix in general is to use jemalloc with

    MALLOC_CONF="retain:false,muzzy_decay_ms:0,dirty_decay_ms:0"

So in jemalloc, the settings to control this behaviour seem to actually work, in contrast to glibc malloc.

(I'm happy to be proven wrong here, but so far no combination of settings seem to actually make glibc return memory as written in their docs.)

From this perspective, it is frightening to see the jemalloc repo being archived, because that was my way to make sure stuff doesn't OOM in production all the time.

They're not really correct, glibc will return stuff back to the OS. It just has some quirks about how and when it does it.

First, some background: no allocator will return memory back to the kernel for every `free`. That's for performance and memory consumption reasons: the smallest unit of memory you can request from and return to the kernel is a page (typically 4kiB or 16kiB), and requesting and returning memory (typically called "mapping" and "unmapping" memory in the UNIX world) has some performance overhead.

So if you allocate space for one 32-byte object for example, your `malloc` implementation won't map a whole new 4k or 16k page to store 32 bytes. The allocator probably has some pages from earlier allocations, and it will make space for your 32-byte allocation in pages it has already mapped. Or it can't fit your allocation, so it will map more pages, and then set aside 32 bytes for your allocation.

This all means that when you call `free()` on a pointer, the allocator can't just unmap a page immediately, because there may be other allocations on the same page which haven't been freed yet. Only when all of the allocations which happen to be on a specific page are freed, can the page be unmapped. In a worst-case situation, you could in theory allocate and free memory in such a way that you end up with 100 1-byte allocations allocated across 100 pages, none of which can be unmapped; you'd be using 400kiB or 1600kiB of memory to store 100 bytes. (But that's not necessarily a huge problem, because it just means that future allocations would probably end up in the existing pages and not increase your memory consumption.)

Now, the glibc-specific quirk: glibc will only ever unmap the last page, from what I understand. So you can allocate megabytes upon megabytes of data, which causes glibc to map a bunch of pages, then free() every allocation except for the last one, and you'd end up still consuming many megabytes of memory. Glibc won't unmap those megabytes of unused pages until you free the allocation that sits in the last page that glibc mapped.

This typically isn't a huge deal; yes, you're keeping more memory mapped than you strictly need, but if the application needs more memory in the future, it'll just re-use the free space in all the pages it has already mapped. So it's not like those pages are "leaked", they're just kept around for future use.

It can sometimes be a real problem though. For example, a program could do a bunch of memory-intensive computation on launch requiring gigabytes of memory at once, then all that computation culminates in one relatively small allocated object, then the program calls free() on all the allocations it did as part of that computation. The application could potentially keep around gigabytes worth of pages which serve no purpose but can't be unmapped due to that last small allocation.

If any of this is wrong, I would love to be corrected. This is my current impression of the issue but I'm not an authoritative source.

When `free()` is called, the allocator internally marks that specific memory area as unused, but it doesn't necessarily return that area back to the OS, for two main reasons:

1. `malloc()` is usually called with sizes smaller than the sizes by which the allocator requests memory from the OS, which are at least page-sized (4096 bytes on x86/x86-64) and often much larger. After a `free()`, the freed memory can't be returned to the OS because it's only a small chunk in a larger OS allocation. Only after all memory within a page has been `free()`d, the allocator may, but doesn't have to, return that page back to the OS.

2. After a `free()`, the allocator wants to hang on to that memory area because the next `malloc()` is sure to follow soon.

This is a very simplified overview, and different allocators have different strategies for gathering new `malloc()`s in various areas and for returning areas back to the OS (or not).