Apple’s Darwin OS and XNU Kernel Deep Dive

> Has anyone found effective tools for maintaining this synchronization between documented architecture and implemented code?

Yes, it's called structure, discipline, and iterative improvement.

Keep the documentation alongside the code. Think in BSD terms: the OS is delivered as a whole; if I modify /bin/ls to support a new flag, then I update the ls.1 man page accordingly, preferably in the same commit/PR.

The man pages are a good reference if you already have familiarity with the employed concepts, so it's good to have an intro/overview document that walks you through those basics. This core design rarely sees radical changes, it tends to evolve - so adapt the overview as you make strategic decisions. The best benchmark is always a new hire. Find out what is it that they didn't understand, and task them with improving the document.

The claim is inaccurate and mixes together multiple different things:

- The Mach microkernel originally supported true userland paging, like mmap but with an arbitrary daemon in place of the filesystem. You can see the interface here:

https://web.mit.edu/darwin/src/modules/xnu/osfmk/man/memory_...

But I'm not sure if Darwin ever used this functionality; it certainly hasn't used it for the last ~20 years.

- dynamic_pager never used this interface. It used a different, much more limited Mach interface where xnu could alert it when it was low on swap; dynamic_pager would create swap files, and pass them back into the kernel using macx_swapon and macx_swapoff syscalls. But the actual swapping was done by the kernel. Here is what dynamic_pager used to look like:

https://github.com/apple-oss-distributions/system_cmds/blob/...

But that functionality has since moved into the kernel, so now dynamic_pager does basically nothing:

https://github.com/apple-oss-distributions/system_cmds/blob/...

- The vast majority of kernel memory is wired and cannot be paged out. But the kernel can explicitly ask for pageable memory (e.g. with IOMallocPageable), and yes, that memory can be swapped to disk. It's just rarely used.

Still, any code that does this needs to be careful to avoid deadlocks. Even though userland is no longer involved in "paging" per se, it's still possible and in fact common for userland to get involved one or two layers down. You can have userland filesystems with FSKit (or third-party FUSE). You can have filesystems mounted on disk images which rely on userland to convert reads and writes to the virtual block device into reads and writes to the underlying dmg file (see `man hdiutil`). You can have NFS or SMB connections going through userland networking extensions. There are probably other cases I'm not thinking of.

EDIT: Actually, I may be wrong about that last bit. You can definitely have filesystems that block on userspace, but it may not be supported to put swap on those filesystems.

https://github.com/apple-oss-distributions/xnu

Sadly, that is not entirely correct.

Whilst all three BSDs (386BSD, FreeBSD, and NetBSD; there was no OpenBSD in the beginning) did inherit the legacy Mach 2.5-style design, it did not live on in FreeBSD, whose core team started pretty quickly replacing all remaining vestiges of the Mach VM[0] with a complete, modern, and highly performant rewrite of the entire VM. FreeBSD 4 had none of the original Mach code left in the kernel codebase, and that happened in the late 1990s. Therefore, FreeBSD can't be referenced in a relationship to Mach apart from the initial separation/very early foundation stage.

NetBSD (and OpenBSD) went on for a while but also quickly hit the wall with the Mach design (performance, SMP/scalability, networking) and also set out on a complete rewrite with UVM (unified virtual memory) designed and led by Chuck Cranor, who wrote his dissertation on the UVM. OpenBSD later borrowed and adopted the UVM implementation, which remains in use today.

So out of all living BSD's[1], only XNU/Darwin continues to use Mach, and not Mach 2.5 but Mach 3. There have been Mach 2.5, 3 and 4 (GNU/Hurd uses Mach 4) in existence, and the compatibility between them is rather low, and remains mostly at the overall architectural level. They are better to be treated as distinct design with shared influence.

[0] Of which there were not that many to start off with.

[1] I am not sure whether DragonBSD is dead or alive today at all.

Ohhh interesting! I’ll update the post to include this soon, thanks!

There was never a right time for Apple to make such a switch. NeXTSTEP predates Linux, and when it was adapted into Mac OS X, Apple couldn't afford a wholesale kernel replacement project on top of everything else, and Linux in the late 1990s was far from being an obviously superior choice. Once they were a few versions in to OS X and solidly established as the most successful UNIX-like OS for consumer PCs, switching to a Linux base would have been an expensive risk with very little short-term upside.

Maybe if Apple had been able to keep classic MacOS going five years longer, or Linux had matured five years earlier, the OS X transition could have been very different. But throwing out XNU in favor of a pre-2.6 Linux kernel wouldn't have made much sense.

Back in the days when Apple acquired NeXT, Linux was undergoing lots of development and wasn't well established. Linux being a monolithic kernel didn't offer the levels of compartmentalization that Mach did.

As things now stand, FreeBSD represents many of the benefits of Darwin and the open source nature of Linux. If you seek a more secure environment without Apple's increasing levels of lock-in, then FreeBSD (and the other BSDs) merit consideration for deployment.

Interestingly enough, Apple did contribute to porting Linux to PowerPC Macs in the mid-1990s under the MkLinux project, which started in 1996 before Apple’s purchase of NeXT later that year:

https://en.m.wikipedia.org/wiki/MkLinux

I don’t think there was any work done on bringing the Macintosh GUI and application ecosystem to Linux. However, until the purchase of NeXT, Apple already had the Macintosh environment running on top of Unix via A/UX (for 68k Macs) and later the Macintosh Application Environment for Solaris and HP-UX; the latter ran Mac OS as a Unix process. If I remember correctly, the work Apple did for creating the Macintosh Application Environment laid the groundwork for Rhapsody’s Blue Box, which later became Mac OS X’s Classic environment. It is definitely possible to imagine the Macintosh Application Environment being ported to MkLinux. The modern FOSS BSDs were also available in 1996, since this was after the settlement of the lawsuit affecting the BSDs.

Of course, running the classic Mac OS as a process on top of Linux, FreeBSD, BeOS, Windows NT, or some other contemporary OS was not a viable consumer desktop OS strategy in the mid 1990s, since this required workstation-level resources at a time when Apple was still supporting 68k Macs (Mac OS 8 ran on some 68030 and 68040 machines). This idea would’ve been more viable in the G3/G4 era, and by the 2000s it would have be feasible to give each classic Macintosh program its own Mac OS process running on top of a modern OS, but I don’t think Apple would have made it past 1998 without Jobs’ return, not to mention that the NeXT purchase brought other important components to the Mac such as Cocoa, IOKit, Quartz (the successor to Display PostScript) and other now-fundamental technologies.

Why would we want more of a monoculture? We've put so many eggs in one basket already. I hope we see more diversity in kernels, not further consolidation.

Taken a different way, it feels similar to suggesting Apple should rebase safari on chromium.

> Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux

XNU is only partially open sourced – the core is open sourced, but significant chunks are missing, e.g. APFS filesystem.

Forking Linux might have legally compelled them to make all kernel modules open source–which while that would likely be a positive for humanity, isn't what Apple wants to do

1) We are talking about the late 90s, well before Ubuntu, where Desktop Linux was pretty poor in terms of features and polish.

2) Apple had no money or time to invest in rewriting NeXTStep for a completely new kernel they had no experience in. Especially when so many of the dev team was involved in sorting out Apple's engineering and tech strategy as well as all the features needed to make it more Mac like.

3) Apple was still using PowerPC at the time which NeXTStep supported but Linux did not. It took IBM a couple of years to get Linux running.

Control is important. Apple has never had to fight with Torvalds or IBM or Microsoft over getting something added to the kernel. Just look at the fiasco when Microsoft wanted to add a driver for their virtualization system to the kernel.

Also, one thing you'll notice about big companies - they know that not only is time valuable, worst-case time is important too. If someone in an open-source ecosystem CAN delay your project, that's almost as bad as if they regularly DO delay your project. This is why big companies like Google tend to invent everything themselves, I.E. Google may have "invented Kubernetes" (really, an engineer at Google uninvolved with the progenitor of K8s - Borg - invented it based on Borg), but they still use Borg, which everyone Xoogler here likes to say is "not as good as k8s". Yet they still use it. Because it gives them full control, and no possibility of outsiders slowing them down.

Keep in mind they were also looking at BeOS which is more real time and notably not unix/Linux. I wish I lived in the timeline that they went with it as I'm a huge Be fan.

Hard to imagine Torvalds working on a microkernel, of all people

Many drivers are gradually moving back into userspace via DriverKit: https://developer.apple.com/documentation/driverkit

I think you’re misunderstanding “unified memory”. That term refers to whether the GPU has its own onboard memory chips which must be populated by a DMA transfer. It doesn’t refer to whether the system has an MMU.

IOKit C++ is running in the kernel, so it's not really related to any of the technologies you mentioned which are all userland-only.

I do not think that "security" is the appropriate name for such features.

In my opinion "security" should always refer to the security of the computer owners or users.

These Apple features may be used for enhancing security, but the main purpose for which they have been designed is to provide enhanced control of the computer vendor on how the computer that they have sold, and which is supposed to no longer belong to them, is used by its theoretical owner, i.e. by allowing Apple to decide which programs are run by the end user.

I remember reading back in 2007-2008 that Apple was interested in bringing ZFS support to Mac OS X, but discussions ended once Oracle purchased Sun. This was a bummer; I would’ve loved ZFS on a Mac.

After a cursory Google search, I found this article:

https://www.zdnet.com/article/zfs-on-snow-leopard-forget-abo...

IIRC that was something they had been working on, but it got axed when ZFS changed hands and licensing became potentially thorny. My memory may be failing me though.

Around the time of Snow Leopard, it was rumored. I assume the Oracle buyout of Sun around the same time had a big part in killing that particular idea.

Supporting ZFS in a UNIX kernel requires excessively extensive modifications to the design and implementation of the VMM, namely:

  1. Integration of the kernel's VM with ZFS's adaptive replacement cache which runs in user space – memory pressure cooperation, page accounting and unified memory management. It also requires extensive VM modifications to support ZFS's controlled page eviction, fine-grained dirty page tracking, plus other stuff.

  2. VMM alignment with the ZFS transactional semantics and intent logs – delayed write optimisations, efficient page syncing.

  3. Support for large memory pages and proper memory page alignment – support for superpages (to reduce the TLB pressure and to efficiently map large ZFS blocks efficiently) and I/O alignment awareness (to ensure proper alignment of memory pages to avoid unnecessary copies).

  4. Memory-mapped I/O: different implementation of mmap and support for lazy checksumming for mmap pages.

  5. Integration with kernel thread management and scheduling, co-opertation with VMM memmory allocators.

  6. … and the list goes on and on.