Introduction to CUDA programming for Python developers

Yes. They are largely unrelated. Just go to Nvidia's site and find the docs. Or there are several books (look at amazon).

A "background in AI" is a bit silly in most cases these days. Everyone is basically talking about LLMs or multimodal models which in practice haven't been around long. Sebastian Raschka has a good book about building an LLM from scratch, Simon Prince has a good book on deep learning, Chip Huyen has a good book on "AI engineering". Make a few toys. There you have a "background".

Now if you want to really move the needle... get really strong at all of it, including PTX (nvidia gpu assembly, sort of). Then you can blow people away like the deep seek people did...

The math isn't that difficult. The transformers paper (https://proceedings.neurips.cc/paper_files/paper/2017/file/3...) was remarkably readable for such a high impact paper. Beyond the AI/ML specific terminology (attention) that were thrown out

Neural networks are basically just linear algebra (i.e matrix multiplication) plus an activation function (ReLu, sigmoid, etc.) to generate non-linearities.

Thats first year undergrad in most engineering programs - a fair amount even took it in high school.

May I plug-in with ClojureCUDA, a high-level library that lets you write CUDA with almost no overhead, but write it in the interactive Clojure REPL.

https://github.com/uncomplicate/clojurecuda

There's also tons of free tutorials at https://dragan.rocks And a few books! (not free) at https://aiprobook.com

Everything from scratch, interactive, line-by-line, and each line is executed in the live REPL.

Not a stupid question at all! Imo, you can definitely dive deep into CUDA and GPU architecture without needing to be a math whiz. Think of it like this: you can be a great car mechanic without being the engineer who designed the engine.

Start with understanding parallel computing concepts and how GPUs are structured for it. Optimization is key - learn about memory access patterns, thread management, and how to profile your code to find bottlenecks. There are tons of great resources online, and NVIDIA's own documentation is surprisingly good.

As for the data engineering side, tbh, it's tougher to get into MLE without ML knowledge. However, focusing on the data pipeline, feature engineering, and data quality aspects for ML projects might be

I suggest having a look at https://m.youtube.com/@GPUMODE

They have excellent resources to get you started with Cuda/Triton on top of torch. It also has a good community around it so you get to listen to some amazing people :)

It's definitely possible to focus on the CUDA/GPU side without diving deep into the math. Understanding parallel computing principles and memory optimization is key. I've found that focusing on specific use cases, like optimizing inference, can be a good way to learn. On that note, you might find https://github.com/codelion/optillm useful – it optimizes LLM inference and could give you practical experience with GPU utilization. What kind of AI applications are you most interested in optimizing?

> Math side of AI but still drill deeper into the lower level of CUDA or even GPU architecture

CUDA requires clear understanding of mathematics related to graphics processing and algebra. Using CUDA like you would use traditional CPU would yield abysmal performance.

> MLE or AI Data Engineering without knowing AI/ML

It's impossible to do so, considering that you need to know exactly how the data is used in the models. At the very least you need to understand the basics of the systems that use your data.

Like 90% of the time spent in creating ML based applications is preparing the data to be useful for a particular use case. And if you take Google ML Crash Course, you'll understand why you need to know what and why.

I will provide general advice that applies here, and elsewhere: Start with a project, and implement it, using CUDA. The key will be identifying a problem that is SIMD in nature. Choose something you would normally use a loop for, but that has many (e.g. tens of thousands or more) iterations, which do not depend on the output of the other iterations.

Some basic areas to focus on:

  - Setting up the architecture and config
  - Learning how to write the kernels, and what makes sense for a kernel
  - Learning how the IO and synchronization between CPU and GPU work.

thank you!

They didn’t. They used PTX, which is what CUDA C++ compiles down to, but which is part of the CUDA toolchain. All major players have needed to do this because the intrinsics for the latest accelerators are not actually exposed in the C++ API, which means using them requires inline PTX at the very minimum.

They basically ditched CUDA and went straight to writing in PTX, which is like GPU assembly, letting them repurposing some cores for communication to squeeze out extra performance. I believe that with better AI models and tools like Cursor, we will move to a world where you can mold code ever more specific to your use case to make it more performant.

lol. i guess this tutorial is about cutting out guido ;)

Yes, it is great for key concepts but a bit outdated. Hence we added an LLM/FA section in the linked post!

Rust-Cuda is broken and has been for years.`cudarc` is the [only?] working one.

this looks really cool and i love rust. just a matter of time until everything runs on rust.

Wasn’t this a bunch of kernels that didn’t work?

Disclaimer: I work on pyspur

I'd recommend pyspur if you seek

1) More AI-native features eg. Evals, RAG, or even UI decisions like seeing outputs directly on the canvas when running on the agent 2) Truly open-source Apache license 3) Python-based (in the sense that you can run and extend it via python)

On the other hand, n8n is 1) more mature for traditional workflows 2) offering overall more integrations (probably every single integration you can think of) 3) TypeScript based and runs on Node.js

pyspur is apache 2. it is free to self-host.

Always been possible, but now the time cost of moving data between the GPU and CPU memory is too high to ignore. Branching may be slower on the GPU but it's still faster than moving data to the CPU for a time then back. The maturation of direct GPU-GPU transfers over the network also helped enable GPU-only MPI codes.

interesting, you mean they are less obscure?

[dead]

Triton sits between CUDA and PyTorch and is built to work smoothly within the PyTorch ecosystem. In CUDA, on the other hand, you can directly manipulate warp-level primitives and fine-tune memory prefetching to reduce latency in eg. attention algorithms, a level of control that Triton and PyTorch don't offer AFAIK.

Triton is somewhat limited in what it supports, and it’s not really Python either.

or use Hidet compiler (open source)