AI LLMs can't count lines in a file

This is true, in general LLMs are not great at counting because they don't see individual characters they see tokens.

Imagine you spoke perfect English, but you learned how to write English using Mandarin characters, basically using the closest sounding Mandarin characters to write in English. Then someone asks you how many letter o's are in the sentence "Hello how are you?". Well you don't read using English charaters, you read using Mandarin characters so you read it as "哈咯，好阿优？" because using Mandarin letters that's the closest sounding way to spell "Hello how are you?"

So now if someone asks you how many letter o's are in "哈咯，好阿优？", you don't really know... you are familiar conceptually that the letter o exists, you know that if you spelled the sentence in English it would contain the letter o, and you can maybe make an educated guess about how many letter o's there are, but you can't actually count out how many letter o's there are because you've never seen actual English letters before.

The same thing goes for an LLM, they don't see characters, they only see tokens. They are aware that characters do exist, and they can reason about their existence, but they can't see them so they can't really count them out either.

Why would you expect them to be able to given how they work? [1]

Someone where I work was trying to get an LLM to evaluate responses to an internal multiple-choice quiz (A, B or C), putting people into different buckets based on a combination of the total number of correct responses and having answered specific questions correctly. They spent a week "prompt engineering" it back and forth, with subtle changes to their instructions on how the scoring should work, with no appreciable effect on accuracy or consistency.

That's another scenario where I felt someone was asking for something with no mechanical sympathy for how it was supposed to happen. Maybe a "thinking" model (why do "AI" companies always abuse terms like this? (rhetorical)) would have been able to get enough stuff into the context for it to be able to get closer to a better outcome, but I took their prompt asked it to write code instead, and got it translated into some overly-commented but simple-enough code which would do the job perfectly every time, including a comment that the instructions they'd provided had a gap where people answering with a certain combination of answers wouldn't fall into any bucket.

[1] https://www.youtube.com/watch?v=7xTGNNLPyMI

LLMs don’t reason or count. They predict and output next tokens. “Reasoning” models mostly just have another layer of validating actual output against predictions. Newer models, if provided with programming tools (as they are in the ChatGPT interface), will predict the tokens that make up short scripts and then call those scripts to achieve numeric results for things like counting lines or letters.

Yeah, they're still bad at counting.

Tools like Claude Code work around this by feeding code into the LLMs with explicit line numbers - demo of that here: https://static.simonwillison.net/static/2025/log-2025-06-02-... - expand out some of the "tool result" panels until you see it, more notes on where I got that trace from here: https://simonwillison.net/2025/Jun/2/claude-trace/

Both GPT-4o and o4-mini got it right for me. They both wrote and executed a small Python program:

    # Let's read the file and get line 27
    file_path = '/mnt/data/code.py'
    line_27 = None
    try:
        with open(file_path, 'r') as f:
            lines = f.readlines()
            if len(lines) >= 27:
                line_27 = lines[26].rstrip('\n')
            else:
                line_27 = None
    except FileNotFoundError:
        line_27 = None

The thing is I imagine the LLM would be able to write a code that counted the lines and outputted what is on line 27. It seems inevitable (in a way that scares me) that a good model in the near future would know enough to write that file and execute on its own.

My understanding is early LLMs were bad at math (for similar reasons) but then got better once the model was hooked up to a calculator behind the scenes.

I also noticed that they struggle reversing strings. Ask it to "generate a list of the 30 biggest countries together with their name in reverse". Most of the results will be correct but you'll likely find some weird spelling mistakes.

It's not something they can regurgitate from previously seen text. Models like Claude with background code execution might get around that.

How good do you think a human brain is at doing this if you simply provided the contents of the file as a string of characters (i.e. not in a text editor with line breaks rendered, etc.)?

This prompt works fine with Qwen2.5-Coder-32B-Instruct-Q4_K_M:

    Add a line number prefix to each line, stopping at line 27. What's on line 27 of this program?

They're very bad at geometry. GPT4 (first ed.) tried convincing me a line intersects a sphere 3 times. Completely clueless at comparing volumes of various polyhedra.

I tried it in gpt-4.1, o4-mini, and claude-4-sonnet, and all got the right answer.

ChatGPT o4-mini got it right

https://chatgpt.com/share/683f9f73-42d8-8010-9cbc-27ad396a55...

ChatGPT 4o (the product not the LLM) got it right with a little additional prompting

https://chatgpt.com/share/683f9fd4-e61c-8010-99be-81d25264ba...