O3-mini simulated scikit calculations

There's a methodological flaw here—you asked the model to generate an example, and then simulate the results on that example. For all you know, that specific (very simple) example and its output are available on the internet somewhere. Try it with five strings of other random words and see how it does.

When I ran these prompts, I saw in the chain of thought

  Hmm, I need to run some code. I'm thinking I can use Python, right? There’s     this Python tool I can simulate in my environment since I can’t actually execute   the code. I’ll run a TfidfVectorizer code snippet to compute some outcomes.

This gave me a thought on the future of consumer-facing LLMs though. I was speaking to my nephew about his iPhone, he hadn't really considered that it was "just" a battery, a screen, some chips, a motor, etc.. all in a nice casing. To him, it was a magic phone!

Technical users will understand LLMs are "just" next token predictors that can output structured content to interface with tools all wrapped in a nice UI. To most people they will become magic. (I already watched a video where someone tried to tell the LLM to "forget" some info...)

Well, the code in question is also written by the same LLM, so it could just output something it knows the answers to already. On its own, this result doesn't really seem to prove anything.

OK, this is wild. I just saw o3-mini (regular) to precisely simulate (calculate?) output of quite complicated computations. Well, at least for a human… and no, it didn’t use code interpreter.