Model intelligence is no longer the constraint for automation

Providing more context is difficult for a number of reasons. If you do it RAG style you need to know which context is relevant. LLMs are notorious for knowing that a factor is relevant if directly asked about that factor, but not bringing it up if it's implicit. In business things like people's feelings on things, historical business dealings, relevance to trending news can all be factors. If you fine tune... well... there have been articles recently about fine tuning on specific domains causing overall misalignment. The more you fine tune, the riskier.

Same same human problems. Regardless of their inherent intelligence...humans perform well only when given decent context and clear specifications/data. If you place a brilliant executive into a scenario without meaningful context.... an unfamiliar board meeting where they have no idea of the company’s history, prior strategic discussions, current issues, personel dynamics...expectations..etc etc, they will struggle just as a model does surly. They may still manage something reasonably insightful, leveraging general priors, common sense, and inferential reasoning... their performance will never match their potential had they been fully informed of all context and clearly data/objectives. I think context is the primary primitive property of intelligent systems in general?

Author IMO correctly recognizes that access to context needs to scale (“latent intent” which I love), but I’m not sure I’m convinced that current models will be effective even if given access to all priors needed for a complex task. The ability to discriminate valuable from extraneous context will need to scale with size of available context, it will be pulling needles from haystacks that aren’t straightforward similarity. I think we will need to steer these things.