Apple's MLX adding CUDA support

You're comparing theoretical maximum memory bandwidth. It's not enough to only look at memory bandwidth because you're a lot more likely to be compute limited when you have a lot of memory bandwidth available. For example, M1 had so much bandwidth available that it couldn't make use of even when fully loaded.

Apple is just being stupid, handicapping their own hardware so they can sell the fixed one next year or the year after

This is time tested Apple strategy is now undermining their AI strategy and potential competitiveness

tl;dr they could have done 1600GB/s

The inference side is fine, nowadays. llama.cpp has had a GPU-agnostic Vulkan backend for a while, it's the training side that tends to be a sticking point for consumer GPUs.

nit: it doesn’t really copy data - it’s more about migrating pages between memory spaces. The core problem is that the GPU’s BAR aperture isn’t wide enough to expose all device memory, leading to an overcommitted mapping situation.

The driver manages collections of contiguous pages as units for BAR aperture mapping. Since the GPU’s BAR window is significantly smaller than total device memory (typically 256MB-4GB vs 80GB-140GB of VRAM), the driver maintains these memory segments to track which page ranges are currently mapped through the aperture. They may or may not correspond whatsoever to the mappings as seen from userspace.

When the GPU attempts to access memory that’s not currently mapped, the driver’s fault handler intervenes, unmaps an existing memory segment from the BAR window, and maps in the segment containing the required pages. It looks more like a software-managed TLB for the BAR aperture, and less like traditional page faulting.

This segmented approach reduces mapping overhead since you’re managing larger chunks rather than individual 4KB pages. It’s a delicate balance because if you work in too big of chunks, you make the thrashing worse. This entire remapping process under the hood can be really expensive.

eg: 2P Threadripper and EPYC systems. Every BAR remapping triggers TLB shootdowns that must propagate across the infinity fabric to invalidate stale entries, on both sockets. Since these systems all lack DDIO, the GPU cannot directly access the CPU’s LLC, so every memory access requires a round trip through main memory. This creates a vicious cycle where frequent BAR thrashing causes continuous cross-socket TLB invalidations, each one stalling cores across the entire NUMA topology while you wait on IPIs, which block reads on the GPU of anything in host memory. The lack of cache coherency between GPU and CPU memory hierarchies means you’re consuming bandwidth on both the PCIe bus and memory controllers, while cores (host and device) remain idle waiting for TLB flushes to complete.

It’s an absolute nightmare to debug and profile, because most all of this^ is invisible to your standard tools, especially if aren’t running bare metal and can’t use zen-specific debuggers and perf counters.

This seems like it would be slow…

There is also NVLink c2c support between Nvidia's CPUs and GPUs that doesn't require any copy, CPUs and GPUs directly access each other's memory over a coherent bus. IIRC, they have 4 CPU + 4 GPU servers already available.

See also: https://www.kernel.org/doc/html/v5.0/vm/hmm.html

The Jetsons[1] have unified memory[2].

[1] https://www.nvidia.com/en-us/autonomous-machines/embedded-sy... [2] https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s...

The physical memory is not be unified, but on modern rack scale Nvidia systems, like Grace Hopper or NVL72, the CPU and the GPU(s) share the same virtual address space and have non-uniform memory access to each other's memory.

The servers don't, but the Jetsons do

MacBooks used to use Nvidia GPUs, then Apple had a falling out with Nvidia and the beef stands to this day (Apple didn’t use Nvidia hardware when training it’s own LLMs for Apple Intelligence).

I wouldn’t be surprised if within the next few years we see a return of Nvidia hardware to the Mac, probably starting with low volume products like the MacPro, strictly for professional/high-end use cases.

Won’t work. No driver support.

That particular MBP model had a high rate of GPU failure because it ran too hot.

I imagined the convo between Steve Jobs and Jensen Huang went like this:

S: your GPU is shit

J: your thermal design is shit

S: f u

J: f u too

Apple is the kind of company that hold a grudge for a very long time, their relationships with suppliers are very one way, their way or the highway.

On Apple Silicon, writing to memory on a PCIe / Thunderbolt device will generate an exception. ARM spec says you're allowed to write to devices as if they were memory but Apple enforces that all writes to external devices go through a device memory mapping[0]. This makes using an external GPU on Apple Silicon[1] way more of a pain in the ass, if not impossible. AFAIK nobody's managed to write an eGPU driver for Apple Silicon, even with Asahi.

[0] https://developer.arm.com/documentation/102376/0200/Device-m...

[1] Raspberry Pi 4's PCIe has the same problem AFAIK

> M1 was nice but that was all Jobs doing and planning

M1 was launched 9 years after Jobs died. You're saying they had everything ready to go back then and just sat on their asses for a decade?

It would be funny if you were typing out your response on an iPhone that has been running for 36 hours without recharging.

Yeah, nice to have MLX-opencl or MLX-amd-whatever

If you implement compatible apis, are you prohibited from calling it cuda?

Oh bummer. Almost got excited.