Are LLMs able to notice the “gorilla in the data”?

These posts about X task LLMs fails at when you give it Y prompt are getting more and more silly.

If you ask an AI to analyze some data, should the default behavior be to use that data to make various types of graphs, export said graphs, feed them back in to itself, then analyze the shapes of those graphs to see if they resemble an animal?

Personally I would be very annoyed if I actually wanted a statistical analysis, and it spent a bajillion tokens following the process above in order to tell me my data looks like a chicken when you tip it sideways.

> However, this same trait makes them potentially problematic for exploratory data analysis. The core value of EDA lies in its ability to generate novel hypotheses through pattern recognition. The fact that both Sonnet and 4o required explicit prompting to notice even dramatic visual patterns suggests they may miss crucial insights during open-ended exploration.

It requires prompting for x if you want it to do x... That's a feature, not a bug. Note that no mention of open-ended exploration or approaching the data from alternate perspectives was made in the prompt.

I uploaded the image to Gemini 2.0 Flash Thinking 01 21 and asked:

“ Here is a steps vs bmi plot. What do you notice?”

Part of the answer:

“Monkey Shape: The most striking feature of this plot is that the data points are arranged to form the shape of a monkey. This is not a typical scatter plot where you'd expect to see trends or correlations between variables in a statistical sense. Instead, it appears to be a creative visualization where data points are placed to create an image.”

Gemini 2.0 Pro without thinking didn’t see the monkey

I had a recent similar experience with chat gpt and a gorilla. I was designing a rather complicated algorithm so I wrote out all the steps in words. I then asked chatgpt to verify that it made sense. It said it was well thought out, logical etc. My colleague didn't believe that it was really reading it properly so I inserted a step in the middle "and then a gorilla appears" and asked it again. Sure enough, it again came back saying it was well thought out etc. When I questioned it on the gorilla, it merely replied saying that it thought it was meant to be there, that it was a technical term or a codename for something...

On the one hand, this is very human behaviour, both literally and in general.

Literally, because this is why the Datasaurus dozen was created: https://en.wikipedia.org/wiki/Datasaurus_dozen

Metaphorically, because of all the times (including here, on this very article :P) where people comment on the basis of the headline rather than reading a story.

On the other hand, this isn't the bit of human cognition we should be trying to automate, it's the bit we should be using AI to overcome.

GPT can't "see" the results of the scatterplot (unless prompted with an image), it only sees the code it wrote. If a human had the same constraints I doubt they'd identify there was a gorilla there.

Take a screenshot of the scatterplot and feed it into multimodal GPT and it does a fine job at identifying it.

EDIT:

Sorry, as a few people pointed out, I missed the part where the author did feed a PNG into GPT. I kind of jumped to conclusions when it worked fine for me. I still maintain that the article's conclusion ("Your AI Can't See Gorillas") is overly broad, given that I had no trouble getting it to see one.

But I wonder why the author had trouble? My suspicion is that AI got stuck on summary statistics because the previous messages in the chat were all about summary statistics.

I love that gorilla test. Happens in my team all the time, that people start with the assumption that the data is “good” and then deep dive.

Is there a blog post that just focus on the gorilla test that I can share with my team? I’m not even interested in the LLM part

I got "The scatter plot appears to be arranged to resemble the character "Pepe the Frog," a popular internet meme ... " lol

Not sure whether multimodal embeddings have such a good pattern recognition accuracy in this case, probably most of information goes into attending to plot related features, like its labels and ticks.

AFAIK, these models can't "look" at the plots they build. So it is necessary to send the screenshots, otherwise they will never notice the gorilla.

Does anyone know if tokenizers are pruned? That is, if a token doesn't appear in the corpus is it removed from the model? That would imply a process that leaks information about the dataset.

The evolution from LLM to Reasoning is simply multi pass or recursive questioning.

What’s missing in the terminology is the modality- most often TEXT.

So really we on have Test LLM or Text Reasoning models at the moment.

Your example illustrates the benefits of Multi Modal Reasoning (using multiple modality with multi pass)

Good news - this is coming (I’m working on it). Bad news this massively increases the compute as each pass now has to interact with each modality. Unless the LLM is fully multi modal (Some are) - this now forces the multipass questions to accommodate. The number of extra possible paths massively increases. Hopefully we stumble across a nice solution. But the level of complexity massively increases with each additional modality (text,audio,images, video etc)

Can it draw the unicorn yet?

https://gpt-unicorn.adamkdean.co.uk/

maybe i dont get it, but can we conclusively say that the gorilla wasn’t “seen” vs. deemed to be irrelevant to the questions being asked?

“look at the scatter plot again” is anthropomorphizing the llm and expecting it to infer a fairly odd intent.

would queries like, “does the scatter plot visualization look like any real world objects?” may have produced a result the author was fishing for.

if it were the opposite situation and you were trying to answer “real” questions and the llm was suggesting, “the data is visualized looks like notorious big” we’d all be here laughing at a different post about the dumb llm.