Surprisingly fast AI-generated kernels we didn't mean to publish yet

> LLMs don't understand. It's mind-boggling to me that large parts of the tech industry think that.

I think you might be tied to a definition of "understanding" that doesn't really apply.

If you prompt a LLM with ambiguous instructions, it requests you to clarify (i.e., extend prompt to provide more context) and once you do the LLM outputs something that exactly meets the goals of the initial prompt, does it count as understanding?

If it walks like a duck and quacks like a duck, it's a duck,or something so close to a duck that we'd be better off calling it that.

I don't believe the user meant "understand" in the classical biological and philosophical sense, or were otherwise attempting to anthropomorphize the systems. They were speaking from the practical experience of "this thing takes a somewhat ambiguous input with unique constraints and implements the ask more-or-less as intended".

They understand. Anything able to reason about any arbitrary request and form a plan tailored to that request understands well enough to qualify for the verb. The mechanism behind it may feel hollow or fake. But if its responses reliably show understanding, the LLM understands - by any ordinary measure.

Nearly every argument like this has the same fatal flaw, and it's generally not the critique of the AI, but the critique reflected back on to humans.

Humans also don't understand and are frequently faking understanding, which for many tasks is good enough. There are fundamental limits to what humans can do.

The AI of a few months ago before OpenAI's sycophancy was quite impressive, less so now which means it is being artificially stunted so more can be charged later. It means privately it is much better than what is public. I can't say it "understands," but I can say it outclasses many many humans. There are already numbers of tasks based around understanding where I would already choose an LLM over a human.

It's worth looking at bloom's taxonomy (https://en.wikipedia.org/wiki/Bloom%27s_taxonomy): In the 2001 revised edition of Bloom's taxonomy, the levels were renamed and reordered: Remember, Understand, Apply, Analyze, Evaluate, and Create. In my opinion it is at least human competitive for everything but create.

I used to be very bearish on AI, but if you haven't had a "wow" moment when using one, then I don't think you've tried to explore what it can do or tested it's limits with your own special expertise/domain knowledge, or if you have then I'm not sure we're using the same LLMs. Then compare that experience to normal people, not your peer groups. Compare an LLM to people into astrology, crystal healing, or homeopathy and ask which has more "understanding."

meh. I feel this is just a linguistic shortcut, similar to how _trained_ biologists can talk about a species or organism evolving some trait. Of course the organism isn't _really_ evolving with any goal in mind, but that's clear to the speaker and audience. Whether or not LLMs understand (very unlikely), it's clear what we mean by an LLM "understanding": has the context + prior training to make reasonable predictions. But no one wants to write that each time.

How do you know?

What is the limit my system will reach?

Thats an interesting word to pick on. Understanding still means something here in a relative sense.

I would be annoyed along with you if I thought the post was true.

Thanks!!! I decided not to build it, that space is already too busy, there is a startup with $25MM in stealth, who else is in stealth? On top of that, this method will get stale very very quickly, foundation model businesses are just too hard to work around right now, it's a silly way to do business. My magic is I've build a startup from scratch to over 400 people and watched what they do, it won't be long till that isn't worth much.

Goods and services are a byproduct of business, business is primarily concerned with systems and processes that facilitate value exchange, so my tool, can work with a user, to build a business, not a product or a service. If you bake cupcakes, my tool can get you 100 people at your door, it cannot open the door or provide the cakes.

I would love to know how to do it another way if you have any ideas, I'm sadly not experienced or intelligent enough to think of another way to do it.

The only investor is me. I build it on my own over a weekend, on my own. I just wanted to confirm it can be done therefore will exist, that is all. Personally, I decided not to peruse it because I am old and lazy and don't want to compete against a16z and sequoia funded adderall filled teenagers.

> I designed something for a business and found I needed 4 major sub-systems (like a real business) - insight/data, cognition, meta cognition and execution, and if you don't define all 4, the system is junk.

Might it be just another realization of Conway's law?

https://en.wikipedia.org/wiki/Conway%27s_law

Might it be possible that the only reason you're assuming a system is junk is just that it doesn't resemble the systems you know and expect? There are so many ways to skin a cat, and certainly no business process represents the optimal process.

For those who don't want to read the article: The previous best was 49 scalar multiplications.

While I think the OP did not mean the compilation process is nondeterministic, I won't be surprised if it is actually non-deterministic. A lot of algorithms and data structures rely on nondeterminism for performance or security (by default). It is too easy to introduce nondeterminism accidentally, and it is tempting to use that to speed up algorithms. Also, if it relies on floating point, results on different machines and environments may be different (depending on libm and hardware implementation), which is, in some sense, nondeterministic.

The running time of a CUDA kernel is apparently impossible to determine except by experiment and measurement, and might be nondeterministic. By contrast for a more typical CPU, there's a compiler whose assembly output you can examine, and there's a processor manual that gives the cycle timing of each instruction. So you can compute the running time at least of inner loops that stay in cache, and that sort of thing.

The point was about being able to write an optimal program with certainty, not about just getting the thing to operate.

>The implication was that the FP32 versions of these kernels have lagged behind the more popular versions.

Help me understand this 'cause I'm a bit slow these days ...

Does that mean optimized FP32 versions of these kernels were already there or not?

I was blown away by some debugging results I got from o3 early on and have been using it heavily since. The early results that caught my attention were from a couple cases where it tracked down some problematic cause through several indirect layers of effects in a way where you'd typically be tediously tracing step-by-step through a debugger. I think whatever's behind this capability has some overlap with really solid work it'll do in abstract system design, particularly in having it think through distant implications of design choices.

In the context of LLMs, what do you mean by "reason"? What does reasoning look like in LLMs and how do you recognize it, and more importantly, how do you invoke it? I haven't had much success in getting LLMs to solve, well, basically any problem that involves logic.

Chain of thought at least introduces some skepticism, but that's not exactly reasoning. It makes me wonder what people refer to when they say "reason".

Very likely. Larger context is significantly beneficial to the LLMs when they can maintain attention, which was part of my point. Imagine being able to hold the word for word text of your required reading book while you are taking a test, while older models were more like a couple chapters worth of text. Two years ago.

SRE / DevOps / coding mostly in the Azure and .NET ecosystems.

The problems I have to solve tend to be the horrible ones that nobody has answers to, anywhere on the Internet, so unsurprisingly the AIs aren't good at it either.

The trick has been to use the AIs for what they are good that, which used to be "nothing" for me at least, but now I can use them productively for certain "spot" tasks.

Random examples:

- Cross-language and cross-platform benchmarking of a bunch of different database clients to see how they stack up. I gave the AI a working example in one language and got it to whip up a series of equivalents with other DB drivers and languages. Sure, it's trivial, but it's way faster than doing it myself!

- Crash dump analysis using WinDbg. I read somwhere that "vibe debugging" of kernel dumps totally works, so when I had an actual crash I gave it a go for laughs. With AI help I managed to extract the name of the specific file that had NTFS corruption and was crashing the server. Deleted the file, restored it from backups, and the server was good to go again!

- If you ever watch the top mechanical engineers on YouTube, they all make their own tools instead of just buying them. Jigs, extenders, unusual sizes, etc... IT work is the same. As a recent example, I got Gemini to make me a code-AST rewriter for a specific issue I wanted to clean up in bulk across a huge code base. Using the Roslyn compiler SDK is a bit fiddly, but it spat out a working tool for me in under an hour. (This is not something you can solve with a script full of regex, it needed a proper parser to handle commented-out blocks and the like.)

Theres zero days in obscure parts of the kernel nobody uses every other day. (It also of course found 100 other things that were not zero days or vulnerabilities, yet professed they were, which is why this trash even on Gemini 9000 Pro keeps spamming security mails)

That's an entirely different idea that may or may not work. This is not evidence of that.