The Fractured Entangled Representation Hypothesis

I can't really take this paper seriously because they are severely lacking references to the linear representation hypothesis. Much work on neural network interpretability lately has shown individual neurons are polysemantic, and therefore practically useless for explainability. My hypothesis is fitting linear probes (or a sparse autoencoder) would reveal linearly semantic attributes.

It is unfortunate because they briefly mention Neel Nanda's Othello experiments, but not the wide array of experiments like the NeurIPS Oral "Linear Representation Hypothesis in Language Models" or even golden gate Claude.

Did you investigate other search processes besides SGD? I'm thinking of those often termed "biologically plausible" (e.g. forward-forward, FA). Are their internal representations closer to the fractured or unified representations?

> Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance.

Scaling up almost always leads to better performance. If you're only getting linear gains though then there is absolutely nothing to be excited about. You are in a dead end.

Tweet: https://x.com/kenneth0stanley/status/1924650124829196370 Arxiv: https://arxiv.org/abs/2505.11581