Vintage Large Language Models

The talk focuses for a bit on having pure data from before the given date. But it doesn't consider that the data available from before that time may be subject to strong selection bias, based on what's interesting to people doing scholarship or archival work after that date. E.g. have we disproportionately digitized the notes/letters/journals of figures whose ideas have gained traction after their death?

The article makes a comparison to financial backtesting. If you form a dataset of historical prices of stocks which are _currently_ in the S&P500, even if you only use price data before time t, models trained against your data will expect that prices go up and companies never die, because they've only seen the price history of successful firms.

I love the ideas about how we might use historical LLMs to inquire into the past!

I imagine that (the author hints at this), to do this rigorously, spelling out assumptions etc, you’d have to build off theoretical frameworks used to inductively synthesize/qualify interviews and texts, currently around in history and the social sciences.

Very cool! I’ve been wanting to do this do a long time!