I want a good parallel computer

One of the most important steps of my career was being forced to write code for an 8051 microcontroller. Then writing firmware for an ARM microcontroller to make it pretend it was that same 8051 microcontroller.

I was made to witness the horrors of archaic computer architecture in such depth that I could reproduce them on totally unrelated hardware.

> I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with.

So the designers of the Cell processor made some mistakes and therefore the entire concept is bunk? Because you've seen a concept done badly, you can't imagine it done well?

To be clear, I'm not criticising those designers, they probably did a great job with what they had, but technology has moved on a long way from then... The theoretical foundations for memory models, etc. are much more advanced. We've figured out how to design languages to be memory safe without significantly compromising on performance or usability. We have decades of tooling for running and debugging programs on GPUs and we've figured out how to securely isolate "users" of the same GPU from each other. Programmers are as abstracted from the hardware as they've ever been with emulation of different architectures so fast that it's practical on most consumer hardware.

None of the things you mentioned are inherently at odds with more parallel computation. Whether something is a good idea can change. At one point in time electric cars were a bad idea. Decades of incremental improvements to battery and motor technology means they're now pretty practical. At one point landing and reusing a rocket was a bad idea. Then we had improvements to materials science, control systems, etc. that collectively changed the equation. You can't just apply the same old equation and come to the same conclusion.

> What amazes me more is the comment section—full of people waxing nostalgic for architectures they clearly never had to ship stable software on.

Isn't it much more plausible that the people who love to play with exotic (or also retro), complicated architectures (with in this case high performance opportunities) are different people than those who love to "set up or work in an assembly line for shipping stable software"?

> I really believe I need to spend 2-weeks requiring students write code on an Amiga, and the programs have to run at the same time. If anyone of them crashes, they all will fail my course. A new found appreciation may flourish.

I rather believe that among those who love this kind of programming a hate for the incompetent fellow student will happen (including wishes that these become weed out by brutal exams).

> There's a whole generation of engineers that don't seem to realize why we architected things this way in the first place.

Nobody teaches it, and nobody writes books about it (not that anyone reads anymore)

I loved and really miss the cell. It did take quite a bit of work to shuffle things in and out of the SPUs correctly (so yeah, it took much longer to write code and greater care), but it really churned through data.

We had a generic job mechanism with the same restrictions on all platforms. This usually meant if it ran at all on Cell it would run great on PC because the data would generally be cache friendly. But it was tough getting the PowerPC to perform.

I understand why the PS4 was basically a PC after that - because it's easier. But I wish there was still SPUs off the side to take advantage of. Happy to have it off die like GPUs are.

Those students would all drop out and start meditating. That would be a fun course. Speed run developing for all the prickly architectures of the 80s and 90s.

Don't worry, with LLMs, we're moving away from anything that remotely looks like "stable software" :)

Also, yeah, I recall the dreaded days of cooperative multitasking between apps. Moving from Windows 3.x to Linux was a revelation.

> They forget why we moved on. Modern systems are built with constraints like memory protection, isolation, and stability in mind. You can’t just “flatten address spaces” and ignore the consequences.

Is there any reason why GPU-style parallelism couldn't have memory protection?

>What I would like is a CPU with a highly parallel array of "worker cores" all addressing the same memory and speaking the same goddamn language that the CPU does.

For "embarrassingly parallel" jobs vector extensions start to eat tiny bits of the GPU pie.

Unfortunately, just slapping thousands of cores works poorly in practice. You quickly get into the synchronization wall caused by unified memory. GPUs cleverly work around this issue by using numerous tricks often hidden behind extremely complex drivers (IIRC CUDA exposes some of this complexity).

The future may be in a more explicit NUMA, i.e. in the "network of cores". Such hardware would expose a lot of cores with their own private memory (explicit caches, if you will) and you would need to explicitly transact with the bigger global memory. But, unfortunately, programming for such hardware would be much harder (especially if code has to be universal enough to target different specs), so I don't have high hopes for such paradigm to become massively popular.

What I want is a Linear Algebra interface - As Gilbert Strang taught it. I'll "program" in LinAlg, and a JIT can compile it to whatever wonky way your HW requires.

I'm not willing to even know about the HW at all, the higher level my code the more opportunities for the JIT to optimize my code.

What I really want is something like Mathematica that can JIT to GPU.

As another commenter mentioned all the API's assume you're a discrete GPU off the end of a slow bus, without shared memory. I would kill for an APU that could freely allocate memory for GPU or CPU and change ownership with the speed of a pagefault or kernel transition.

> The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths.

To me it feels somewhat like programming for the segmented memory model with its near and far pointers, back in the old days. What a nightmare.

You can have that today. Just go out and buy more CPUs until they have enough cores to equal the number of SMs in your GPU (or memory bandwidth, or whatever). The problem is that the overhead of being general purpose -- prefetch, speculative execution, permissions, complex shared cache hierarchies, etc -- comes at a cost. I wish it was free, too. Everyone does. But it just isn't. If you have a workload that can jettison or amortize these costs due to being embarrassingly parallel, the winning strategy is to do so, and those workloads are common enough that we have hardware for column A and hardware for column B.

Larrabee was something like that, didn't took off.

IMHO, the real issue is cache coherence. GPUs are spared from doing a lot of extra work by relaxing coherence guarantees quite a bit.

Regarding the vendor situation - that's basically how most of computing hardware is, save for the PC platform. And this exception is due to Microsoft successfully commoditizing their complements (which caused quite some woe on the software side back then).

I really would like you to sketch out the DX you are expecting here, purely for my understanding of what it is you are looking for.

I find needing to write seperate code in a different language annoying but the UX of it is very explicit of what is happening in the memory which is very useful. With really high performance compute across multiple cores ensuring you don't get arbitrary cache misses is a pain. If we could address CPUs like we address current GPUs( well you can but it's not generally done) it would make it much much simpler.

Want to alter something in parallel, copy it to memory allocated to a specific core which is guaranteed to only be addressed by that core and the do the operations on it.

To do that currently you need to be pedantic about alignment and manually indicate thread affinity to the scheduler etc. Which ia entirely as annoying as GPU programming.

> What I would like is a CPU with a highly parallel array of "worker cores" all addressing the same memory...

I too am very interested in this model. The Linux kernel supports up to 4,096 cores [1] on a single machine. In practice, you can rent a c7a.metal-48xl [2] instance on AWS EC2 with 192 vCPU cores. As for programming models, I personally find the Java Streams API [3] extremely versatile for many programming workloads. It effectively gives a linear speedup on serial streams for free (with some caveats). If you need something more sophisticated, you can look into OpenMP [4], an API for shared-memory parallelization.

I agree it is time for some new ideas in this space.

[1]: https://www.phoronix.com/news/Perf-Support-2048-To-4096-Core...

[2]: https://aws.amazon.com/ec2/instance-types/c7a/

[3]: https://docs.oracle.com/en/java/javase/24/docs/api/java.base...

[4]: https://docs.alliancecan.ca/wiki/OpenMP

Yep, and those printers are proprietary and mutually incompatible, and there are buggy mutually incompatible serial drivers on all the platforms which results in unique code paths and debugging & workarounds for app breaking bugs for each (platform, printer brand, printer model year) tuple combo.

(That was idealized - actually there may be ~5 alternative driver APIs even on a single platform each with its own strengths)

The key transformation required to make any parallel architecture work is going to be taking a program that humans can understand, and translating it into a directed acyclic graph of logical Boolean operations. This type of intermediate representation could then be broken up into little chunks for all those small CPUS. It could be executed very slowly using just a few logic gates and enough ram to hold the state, or it could run at FPGA speeds or better on a generic sea of LUTs.

Yep, transputers failed miserably. I wrote a ton a code for them. Everything had to be solved in a serial bus, which defeated the purpose of the transputer.

Could you elaborate on this? How does many-small-CPUs make for a weirder programming model than a GPU?

Im no expert, but I’ve done my fair share of parallel HPC stuff using MPI, and a little bit of Cuda. And to me the GPU programming model is far far “weirder” and harder to code for than the many-CPUs model. (Granted, I’m assuming you’re describing a different regime?)

Im guessing you just dont have the computational power to compete with a real GPU. It would be relatively easy for a top end graphics programmer to write the front end graphics API for your chip. Im guessing that if they did this you would just end up with a very poor performing GPU.

While acknowledging that it's theoretically possible other approaches might succeed, it seems quite clear the author agrees with you.

my take from reading this is more about programming abstractions than any particular hardware instantiation. the part of the Connection Machine that remains interesting is not building machines with CPUS with transistor counts in the hundreds running off a globally synchronous clock, but that there were a whole family of SIMD languages and let you do general purpose programming in parallel. And that those language were still relevant when the architecture changed to a MIMD machine with a bunch of vector units behind each CPU.

Reminds me of Itanium

Picochip?

Possibly compilation and linking. That's very slow for big programs like Chromium. There's really interesting work on GPU compilers (co-dfns and Voetter's work).

Optimization problems like scheduling and circuit routing. Search in theorem proving (the classical parts like model checking, not just LLM).

There's still a lot that is slow and should be faster, or at the very least made to run using less power. GPUs are good at that for graphics, and I'd like to see those techniques applied more broadly.

A big one is video encoding. It seems like GPUs would be ideal for it but in practice limitations in either the hardware or programming model make it hard to efficiently run on GPU shader cores. (GPUs usually include separate fixed-function video engines but these aren't programmable to support future codecs.)

2D rendering is harder in fact, because antialiased curves are harder than triangle soup.

It's an issue of code complexity, not fill rate

https://faultlore.com/blah/text-hates-you/

I think a dynamic, fully vector-based 2D interface with fluid zoom and transformations at 120Hz+ is going to need all the GPU help it can get. Take mapping as an example: even Google Maps routinely struggles with performance on a top-of-the-line iPhone.

> Other than as an exercise, it's not clear why someone would write a massively parallel 2D renderer that needs a GPU. Modern GPUs are overkill for 2D. Now, 3D renderers, we need all the help we can get.

A ton of 2D applications could benefit from further GPU parallelization. Games, GUIs, blurs & effects, 2D animations, map apps, text and symbol rendering, data visualization...

Canvas2D in Chrome is already hardware accelerated, so most users get better performance and reduced load on main UI & CPU threads out of the box.

Fast light transport is an incredibly hard problem to solve.

Raytracing (in its many forms) is one solution. Precomputing lightmaps, probes, occluder volumes, or other forms of precomputed visibility are another.

In the end it comes down to a combination of target hardware, art direction and requirements, and technical skill available for each game.

There's not going to be one general purpose renderer you can plug into anything, _and_ expect it to be fast, because there's no general solution to light transport and geometry processing that fits everyone's requirements. Precomputation doesn't work for dynamic scenes, and for large games leads to issues with storage size and workflow slow downs across teams. No precomputation at all requires extremely modern hardware and cutting edge research, has stability issues, and despite all that is still very slow.

It's why game engines offer several different forms of lighting methods, each with as many downsides as they have upsides. Users are supposed to pick the one that best fits their game, and hope it's good enough. If it's not, you write something custom (if you have the skills for that, or can hire someone who can), or change your game to fit the technical constraints you have to live with.

> Nobody has a good solution to this yet. What does the renderer need to know from its caller? A first step I'm looking at is something where, for each light, the caller provides a lambda which can iterate through the objects in range of the light. That way, the renderer can get some info from the caller's spatial data structures. May or may not be a good idea. Too early to tell.

Some games may have their own acceleration structures. Some won't. Some will only have them on the GPU, not the CPU. Some will have an approximate structure used only for specialized tasks (culling, audio, lighting, physics, etc), and cannot be generalized to other tasks without becoming worse at their original task.

Fully generalized solutions will be slow be flexible, and fully specialized solutions will be fast but inflexible. Game design is all about making good tradeoffs.

At Vulkanised 2025 someone mentioned it is an HAL for writing GPU drivers, and they have acknowledge it has gotten as messy as OpenGL and there is now a plan in place to try to sort the complexity mess.

> Other than as an exercise, it's not clear why someone would write a massively parallel 2D renderer that needs a GPU. Modern GPUs are overkill for 2D.

Depends on how complicated your artwork is.

> Modern GPUs are overkill for 2D

That explains why modern GUI are crap: because they are not able to draw a bloody rectangle, and fill it with colour. /s

I've always wondered if you could use iGPU compute cores with unified memory as "transparent" E-cores when needed.

Something like OpenCL/CUDA except it works with pthreads/goroutines and other (OS) kernel threading primitives, so code doesn't need to be recompiled for it. Ideally the OS scheduler would know how to split the work, similar to how E-core and P-core scheduling works today.

I don't do HPC professionally, so I assume I'm ignorant to why this isn't possible.

Isn't Xeon Phi just an instance of Larrabee?

You can. There are API extensions for persistently mapping memory, and it's up to you to ensure that you never write to a buffer at the same time the GPU is reading from it.

At least for Vulkan/DirectX12. Metal is often weird, I don't know what's available there.

Unified memory doesn't mean unified address space. It frustrates me when no one understands unified memory.