Semantic unit testing: test code without executing it

It's a common idea, all the way back to Hoare logic. There was a time when people believed in the future, people would write specifications instead of code.

The problem with it takes several times more effort to verify code than to write it. This makes intuitive sense if you consider that the search space for the properties of code is much larger than the code for space. Rice theorem's states that all non trivial semantic properties of a program are undeniable.

> That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently (...)

The problem with your assertion is that it fails to understand that today's software, where every single line of code was typed in by real flesh-and-bone humans, already fails to have adequate test coverages, let alone be validated.

The main problem with output from LLMs is that they were trained with the code written by humans, and thus they accurately reflect the quality of the code that's found in the wild. Consequently, your line of reasoning actually criticizes LLMs for outputing the same unreliable code that people write.

Counterintuitively, LLMs end up generating a better output because at least they are designed to simplify the task of automatically generating tests.

This particular person seems to be using LLMs for code review, not generation. I agree that the problem is compounded if you use an LLM (esp. the same model) on both sides. However, it seems reasonable and useful to use it as an adjunct to other forms of testing, though not necessarily a replacement for them. Though again, the degree to which it can be a replacement is a function of the level of the technology, and it is currently at the level where it can probably replace some traditional testing methods, though it's hard to know which, ex-ante.

edit: of course, maybe that means we need a meta-suite, that uses a different LLM to tell you which tests you should write yourself and which tests you can safely leave to LLM review.

LLM-based coding only really works when wrapped in structured prompts, constrained outputs, external checks etc. The systems that work well aren’t just 'LLM take the wheel' architecture, they’re carefully engineered pipelines. Most success stories are more about that scaffolding than the model itself.

If you are working with natural language, it is by definition 'fuzzy' unless you reduce it to simple templates. So to evaluate whether an output is a semantically e.g. a reasonable answer to an input where non-templated natural verbalization is needed, you need something that 'tests' the output, and that is not going to be purely 'logical'.

Will that test be perfect? No. But what is the alternative?

I agree. I want this as a code review tool to check if people forgot to update comments - "it looks like this now adds instead of multiplies, but the comment says otherwise; did you forget to update it?".

Seems of dubious value as unit tests. LLMs don't seem to be quite smart enough for that in my experience, unless your bugs are really as trivial as adding instead of multiplying, in which case god help you.

Couldn't put it better myself.

I have the toughest time trying to communicate why f(x) should equal f(x) in the general case.

Hm... I think you have a good point.

Maybe the non-determinism can be reduced by caching: Just reevaluate the spec if the code actually changes?

I think there are also other problems (inlining a verbal description makes the codebase verbose, writing a precise, non-ambiguous verbal description might be more work than writing unit tests)

Many good and prolific approaches are non deterministic such as fuzzing or property-based testing,

this. Let’s not confuse meanings. There are multiple ways to improve quality of code. Testing is one, code review is another. this belongs to the latter

Yeah this sounds like a good way to detect out of date comments. I would have focused on that.

Agree, it's not testing. The problem is here: "In a typical testing workflow, you write some basic tests to check the core functionality. When a bug inevitably shows up—usually after deployment—you go back and add more tests to cover it. This process is reactive, time-consuming, and frankly, a bit tedious."

This is exactly the problem that TDD solves. One of the most compelling reasons for test-first is because "Running the code in your head" does not actually work well in practice, leading to the above-cited issues. This is just another variant of "Running the code in your head" except an LLM is doing it. Strong TDD practices (don't write any code without a test to support it) will close those gaps. It may feel tedious at first but the safety it creates will leave you never wanting to go back.

Where this could be safe and useful: Find gaps in the test-set. Places where the code was never written because there wasn't a test to drive it out. This is one of the hardest parts of TDD, and where LLMs could really help.

> But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this.

I firmly believe that the group of people who laud dynamically typed languages as efficient time-savers, that help shed drudge work involving typing, is tightly correlated with the group of people who fail to establish any form of quality assurance or testing, often using the same arguments to justify their motivation.

Dan Luu looked at the literature and concluded that the evidence for the benefit of types is underwhelming:

https://danluu.com/empirical-pl/

>But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this.

OK, but if the alternative to tests is spending more time on a reliability method (type annotations) which buys you less reliability compared to writing tests... it's hardly a win.

It fundamentally seems to me that there are plenty of bugs that types can simply never catch. For example, if I have a "divide" function and I accidentally swap the numerator and divisor arguments, I can't think of any realistic type system which will help me. Other methods for achieving reliability, like writing tests or doing code review, don't seem to have the same limitations.

> On statically typed languages this happens for free at compile time.

If only that were true I wouldn't be a tiny bit as good at tracking down segfaults as I've become over the years...

I think at least some people who say this think of Java-esque type systems. And there I agree: it is a boilerplate nightmare.

> Did the author do any analysis of the effectiveness of their tool on something beyond multiplication? Did they look to see if it caught any bugs in any codebases? What's the false positive rate? False negative?

I would also add the concern on whether the tests are actually deterministic.

The premise is also dubious, as docstring comments typically hold only very high-level descriptions of the implementation and often aren't even maintained. Writing a specification of what a function is expected to do is what writing tests is all about, and with LLMs these are a terse prompt away.

if that is the only testing you do I agree. However to test that the code works as the docs say is valuable as well. The code often will do more, but it needs to do at least what the docs say.