Code and Trust: Vibrators to Pacemakers
7 comments
·July 6, 2025mrheosuper
> How do we get to a point to `trust` it?
You don't, similar to you don't trust code written by human. The code needs to be audited, line-by-line.
tzs
> This program will vibrate with increase frequency using the Fibonacci numbers 2, 3, 5, 8, 13, 21, 34..
No it won't. The sequence it follows is 2, 1, 1, 1, ...
After spotting that I was curious if LLMs would also spot it. I asked Perplexity this and gave it the code:
> This is Python code someone posted for a hypothetical vibrator. They said it will vibrate with increasing frequency following Fibonacci numbers: 2, 3, 5, 8, ...
> Does the frequency really increase following that sequence?
It correctly found the actual sequence, explained why the code fails, and offered a fix.
kh_hk
Weird, after the fix vibrator sleeps the longer it is used. Also, at some point it burst into flames
MangoToupe
> To be honest, my threshold is quite low, I think in 1 year we will be able to take any piece of code and audit it with `claude`, and I will put my life on the line using it.
Bold! I wonder what leads to this sort of confidence.
> Radical transparency is the only answer. I am convinced that open code, specs and procesees must be requirement going forward.
Yes. Transparency is the only foundation of trust.
null
pragmatic
Brakes!!
AI can’t even spell/grammar check this article lol.
I'm trivialising, but a lot of software in medical devices is turning a GPIO pin on/off in response to another pin, then announcing that it did so. The piece missing from the article is that the assumed probability of software/firmware (or anything really) failing is 1.0. Everything is engineered around the assumption that things (_especially_ software) WILL fail and minimising the consequences when they do. LLM's writing the code will happen soon, it's a GPIO pin control after all. LLM's proving the code is as safe as possible and that they have thought about the failure modes will be a while.