The Anthropic Economic Index

Very wary. Not weary.

Yeah. reminds me of the ancient okcupid data analysis blogs and not the creepy one by sleep8. The group I'm surprised not to see represented in their analysis is "personal", where people I know use ChatGPT as a therapist/life coach/sms analysis&editor. and of course they crucially but understandably left off the denominator. 35% of a million requests is different than 35% of a billion. and also how many of the conversations had 1 message, indicating "just testing" vs 10 or 100 messages.

I think the problem is the data.

Software engineering is a weird niche that is both a high paying job and something you can almost self-teach from widely available free online content. If not self-teach, you can rely on free online content for troubleshooting, examples, etc.

A lot of other industries/jobs are more of an apprenticeship model, with little data and even less freely available on open internet.

I would agree that the products coming out so far lack imagination, but hard disagree on the impact. LLMs have completely transformed multiple industries already. In SWE, I would estimate that junior positions shrank by 70-80%, but even that is less extreme than what is going on in other industries.

In marketing, the entire low-end to mid-tier market is gone. Instead of having teams working on projects for small to mid-sized companies, there's now a single Senior managing projects with the help of LLMs. I know multiple agencies who cut staff by 80-90% without dropping revenue.

Translation (of books, articles, subtitles) was never well paid, even for very complex and demanding work. My partner did it a bit on the side, mostly justifying the low pay with some moral bla about spreading knowledge across cultures... With LLMs you can completely cut out the grunt part. You define the hard parts (terms that don't translate well), round out the edges and edge out the fluff, and every good translator becomes two to ten times more productive. Since work is usually paid by the page, people in the industry got a very decent (at least temporary) pay jump, I would imagine around 100%.

Support is probably the biggest one though. It is important to remember that outsourcing ot India only works for English speaking countries. And even that isn't super cheap. Here in Germany, if you don't have back-up wealth, it is your constitutional right to get some support from the state (~1400 euro), but you are obligated to find a job as soon as possible, and they will try to help you find a role. Support was always one of the biggest industries to funnel people towards. I talked to a friend working there, and according to them the complete industry basically stopped advertising new positions, the only ones that are left are financial services. The rest went all in on LLMs and just employ a fraction of the support stuff to deal with things escalating enough.

And that's not even touching on all the small things. How much energy is spent on creating pitch decks, communicating proposals, writing documentation etc? It probably goes up as far as 50% of work in large Orgs, and even if you can just save 5% of your time by using LLMs to phrase or organize, there is a decent ROI for companies to pay for them.

We’re trying to use it for industrial apps. Been over a year of R&D. Some good but often mixed results. Adherence to prompts is a big issue for us. It’s most useful not as a chatbot but to give explained descriptions of what the user is seeing, so they don’t need to dig down into 20 graphs and past history. That necessitates being able to refer to things with URIs which works 95% of the time but the 5% is killer since it’s difficult to detect issues and leads to dead links.

But "AI" tools have more or less seeped into every mainstream product... this is a strong "defensive move" for companies in anticipation of more to come.

We aren't leaving MS Office or Adobe because they already pushed out some minimal innovation. But what about the products you don't even know about? For lawyers, doctors, logistics, sales, marketing, wood workers, handymen? In Europe or Asia?

New product by bringing true innovation could easily push out legacy business by "shiny new thing"(AI) and better UX alone. A lot of software in these areas simply hasn't improved for 10 years - with a great idea and a dedicated team it's a landslide waiting to happen.

Maybe spend and better models will help this (I’ve not used the deep research models so maybe we are there already). But even day to day coding, the LLMs are great helpers but giving them anything more than a slightly complicated prompts and it seems like these models become completely helpless. You just constantly need a human in the loop because these models are too dumb or lack the context to understand the big picture.

Maybe these models will get better as they’re given more context and can understand the full stack but for now they cannot.

And this is just with code where it already has billions of examples. Nevermind any less data-rich fields. The models still need to get smarter.

Daniel Rock has done some interesting work on the ROI of AI in general (also, I believe two of his papers are referenced in this study). Note that this doesn't explicitly restrict itself to covering LLMs, but... still a very interesting body of work.

https://www.danielianrock.com/research

My term for this is “Whitey’s goin’ to the data center”. We are looking at an arms race, where there really is genuine new technology and it will make a difference - but at the 1-2% per annum of an economy level - compounded over fifty years that is geo political dominance yes, but it’s not “machines of loving grace” level growth.

We already have thousands of geniuses working across our economies and teaching our youth. The best of our minds have every year or so been given a global stage in Nobel speeches. We still ignore their arses and will ignore it when AI tells us to stop fighting or whatever.

The real issue here is that wafer scale chips give 900,000 cores, and nothing but embarrassingly parallel code can use it - and frankly no coder I know writes code like that - we have to rethink our whole approach now Moores law is over. Only AI has anything like ability to use the processing ability being built today - the rest of us can stick to cores from 2016 and nothing would change.

Throwing hundreds of billions at having a bad way to program 1 million cores because we have not rethought software and businesses to cope seems wrong - both because “Whitey” can spend it on better things but also because it is an opportunity - imagine being 900,000 times faster than your competitors - what’s does that even mean?

Edit: Trying to put it another way - there are two ways AI can help us - it can improve cancer treatments at every stage of medical care, through careful design and creation of medical AI models that can slowly ratchet up diagnosis, treatment and even research and analysis. This is human organisations harnessing and adapting around a new technology

Or AI can become so smart it just invents a cure for cancer.

I absolutely think the first is going to happen and will benefit the denizens of the first world first. The second one requires two paradigm shifting leaps in the same sentence. Ten years ago I would have laughed in Anthropics face. Today I just give it a low probability multipled by another low probability- and that is an incredible shift.

Claude is indeed far more familiar amongst software engineers.

Google Gemini integration into their docs/sheets/slides and Gmail perhaps will show different demographics in a few months, and that is yet before we heard from OpenAI.

Anthropic is a Public Benefit Corporation whose governance is very different from a typical company in that it doesn’t put shareholder ROI above all else. A majority of its board seats are reserved for people who hold no equity whatsoever and whose explicit mandate is to look out for humanity.

https://www.anthropic.com/news/the-long-term-benefit-trust

https://time.com/6983420/anthropic-structure-openai-incentiv...

The exact opposite. Relative to ChatGPT Anthropic has an enormous "brand problem." What they should be doing is exclusive deals like this, but with deals with large publishers on a recurring basis and figure out how to teach consumers who they are and how to use them best. For like 99% of the use cases all these products are parody and the real business gains are finding a way into consumers lives.

Semi-relevant sidenote: ChatGPT, spent $8m on a super bowl commercial yesterday just to show cool visualizations instead of any emotional product use case to an ultra majority audience has never had a direct experience with the product.

These companies would be best served building a marketing arm away from the main campus in a place like LA or NY to separate the gen pop story from that of the technology.

I disagree. I think Anthropic, like the other big players, is trying to get some of that government money. Releasing policy-adjacent papers seems like a way to alert government officials that Anthropic ought to be in the room when the money starts changing hands.

I am inclined to agree. If you’re at the precipice of automating or transforming knowledge work and the value for being the first is nearly infinite (due to “flywheel effects”), why would you dedicate any energy to studying the impact of AI on jobs directly? The thesis is everything changes.

I think AI in its current iteration is going to settle into being like a slightly worse version of Wikipedia morphed with a slightly better version of stackoverflow.

Understating who is using your product is wheel spinning?

idk, the models themselves are quickly becoming a commodity. it makes sense to spend money figuring out go to market rather than just improve the models themselves.

I don't see it. This is just an analysis of how Anthropic customers are using the product and what investment areas seem most promising in the future - why wouldn't they want that?

I would argue this is within their overall objective. It’s not like Stripe creating a publisher (??)

They also call out how it's more used for augmentation rather than full automation which will address some concerns the public has.

I honestly don’t know who the audience could be, other than “people who like to tell others they’re in the know because they read AI companies’ press releases.”

At no point do I see an actual elevator pitch/tl;dr/summary of what the frak this index actually is, except that it’s part of some effort to track AI adoption. It just rains down figures about which industries are using how much AI without first grounding the new concept they’re introducing.

When you say you have a new economic index, you need to give me a number, how I should interpret that number, and where it comes from. I don’t see that.

GDP: measure if a country’s total economic output by adding up end product purchases.

CPI: general price level by taking a weighted average of prices throughout the economy

Big Mac index: how expensive goods are in a country relative to the US by reference to the local cost of a Big Mac, converted through the exchange rate.

Here I expect something like “the economic output-weighted fraction of production taken over by AI”, but instead it’s just a list of AI adoption by industry.

Why introduce an index and not headline with a definition of an index? Which audience prefers that?

I expect more 'automation' to happen through the API than 'augmentation'.

Tools like cursor have to be huge chunk of their traffic. I mean Cursor and Sonnet are like two peas in a pod.

Even if they don’t train on the data they could break it down by user agent / API client ID and infer something about cursor traffic.

haven't they committed to not training on / using data submitted through API?

I laughed out loud at their chart showing that Sonnet had a higher share of coding questions, whereas Opus had more writing.

If they would just look at their product, they'd see that it literally says it in the model description that Opus is better for writing. If you advertise one of your models as geared for task X, the insight that people use it more for task X isn't really an insight.

Read it as that as well, I think this mixes the "there is no penetration in that market" with "we have not been able to get these people to use our tools".

Does OpenAI have similar data available to compare to?

Honestly, simply releasing a graph showing the trendline of their own metrics on e.g. inference would speak volumes. They have this data already, 100%, its on some grafana dashboard the on-calls are watching every day. My suspicion is that some AI providers (OAI/Google) are watching these metrics go up-and-to-the-right quite consistently, but Anthropic's isn't doing that.

Uh, this isn't true at all. Did you actually look at the repo[1]? Only the metadata (i.e., LLM-generated task names and interaction classifications) have been made available.

[1] https://huggingface.co/datasets/Anthropic/EconomicIndex