Canaries in the Coal Mine? Recent Employment Effects of AI [pdf]
73 comments
·August 28, 2025majormajor
LLMs are very useful tools for software development, but focusing on employment does not appear to really dig into if it will automate or augment labor (to use their words). Behaviors are changing not just because of outcomes but because of hype and expectations and b2b sales. You'd expect the initial corporate behaviors to look much the same whether or not LLMs turn into fully-fire-and-forget employee-replacement tools.
Some nits I'd pick along those lines:
>For instance, according to the most recent AI Index Report, AI systems could solve just 4.4% of coding problems on SWE-Bench, a widely used benchmark for software engineering, in 2023, but performance increased to 71.7% in 2024 (Maslej et al., 2025).
Something like this should have the context of SWE-Bench not existing before November, 2023.
Pre-2023 systems were flying blind with regard to what they were going to be tested with. Post-2023 systems have been created in a world where this test exists. Hard to generalize from before/after performance.
> The patterns we observe in the data appear most acutely starting in late 2022, around the time of rapid proliferation of generative AI tools.
This is quite early for "replacement" of software development jobs as by their own prior statement/citation the tools even a year later, when SWE-Bench was introduced, were only hitting that 4.4% task success rate.
It's timing lines up more neatly with the post-COVID-bubble tech industry slowdown. Or with the start of hype about AI productivity vs actual replaced employee productivity.
dathinab
> like this should have the context of SWE-Bench not existing before November, 2023.
Given the absurdly common mal practice(1) of training LLMs on/for tests i.e. what you could describe as training on the test set any widely used/industry standard test to evaluate LLMs is not really worth half of what it claims it is.
(1): Which is at least half intend, but also to some degree accident due to web scrabbling, model cross training etc. having a high chance to accidentally sneak in test data.
In the end you have to have your own tests to evaluate agent/LLM performance, and worse you have to not make them public out of fare of scientific malpractice turning them worthless. Tbh. that is a pretty shitty situation.
eru
Yes, even if the underlying AI stops advancing today, it will take a while for the economy to digest and adjust to the new systems. Eg a lot of the improvements in usefulness in the last few quarters came from better tooling, not necessarily better models.
But with progress continuing in the models, too, it's an even more complicated affair.
trhway
Offshoring was similar - i.e. companies discovered that expensive labor here can be performed inexpensively there while senior laborers/PMs here would perform the overseeing role - and we can look at it how long it took to digest it and adjust to it. While 15-20 years ago it was all the rage, today it is just an established well understood and efficiently utilized, where applicable, practice.
eru
Yes.
However it wasn't just noticing the difference in wages. That had been known since forever and didn't wake a genius. Figuring out how to produce efficiently in the cheaper places and get the goods to rich markets took more smarts and experimentation.
Container shipping played a big role in that, and so did modern communication and cheaper flights.
hochstenbach
One would expect that if such studies indeed indicate that AI has an effect on early-career workers in AI-exposed occupations, that this would be a global effect. I wonder if there are good comparable non-US studies available.
ludicrousdispla
It's funny the lengths to which companies will go in order to avoid work.
godelski
It's also funny how much they'll spend to save a few pennies
wazoox
Well...
> The contest between the capitalist and the wage-labourer dates back to the very origin of capital. It raged on throughout the whole manufacturing period. [112] But only since the introduction of machinery has the workman fought against the instrument of labour itself, the material embodiment of capital. He revolts against this particular form of the means of production, as being the material basis of the capitalist mode of production.
> [...]
> The instrument of labour, when it takes the form of a machine, immediately becomes a competitor of the workman himself. [116] The self-expansion of capital by means of machinery is thenceforward directly proportional to the number of the workpeople, whose means of livelihood have been destroyed by that machinery. The whole system of capitalist production is based on the fact that the workman sells his labour-power as a commodity. Division of labour specialises this labour-power, by reducing it to skill in handling a particular tool. So soon as the handling of this tool becomes the work of a machine, then, with the use-value, the exchange-value too, of the workman’s labour-power vanishes; the workman becomes unsaleable, like paper money thrown out of currency by legal enactment.
Karl Marx, The Capital, Book I Chapter 15.
https://www.marxists.org/archive/marx/works/1867-c1/ch15.htm
NitpickLawyer
> Hard to generalize from before/after performance.
While this is true, there are ways to test (open models) on tasks created after the model was released. We see good numbers there as well, so something is generalising there.
ath3nd
> LLMs are very useful tools for software development
That's an opinion many disagree with. As a matter of fact, the only limited study up to date showed that LLMs usage decrease productivity for experienced developers by roughly 19%. Let's reserve opinions and link studies.
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
My anecdotal experience, for example, is that LLMs are such a negative drain on both time and quality that one has to be really early in their career to benefit from their usage.
ftmootnomoat
This kind of blanket statement smells of the same dogmatism as the AI hype train in reverse.
LLMs are a just a simple tool, if people misuse it it's on them.
ChrisMarshallNY
Probably depends on how it’s being used.
I use LLMs every day. They are useful to me (quite useful), but I don’t really use them for coding. I use them as a “fast reference” source, an editor, or as a teacher.
I’ve been at this since 1983, so I’ve weathered a couple of sea changes.
manmademagic
I wouldn't call myself an 'experienced' developer, but I do find LLMs useful for once-off things, where I can't justify the effort to research and implement my own solution. Two recent examples come to mind:
1. Converting exported data into a suitable import format based on a known schema 2. Creating syntax highlighting rules for language not natively support in a Typst report
Both situations didn't have an existing solution, and while the outputs were not exactly correct, they only needed minor adjustments.
Any other situation, I'd generally prefer to learn how to do the thing, since understanding how to do something can sometimes be as important as the result.
CuriouslyC
People who suck at typing are better off writing by hand as well. I don't need to argue, I'll let history pick a winner.
9rx
> decrease productivity for experienced developers by roughly 19%.
Seems about right when trying to tell an LLM what to code. But flipping the script, letting the LLM tell you what to code, productivity gains seem much greater. Like most programmers will tell you: Writing code isn't the part of software development that is the bottleneck.
jopsen
There are many ways to use an LLM.
Writing code is a bit crazy, maybe writing tedious test case variations.
But asking an LLM questions about a well established domain you're not expert in is a fantastic use case. And very relevant for making software. In practice, most software requires you to understand the domain your aiming to serve.
ChrisMarshallNY
Not sure why your post was dinged. I have used them for exactly this, and it has been amazingly effective.
yakshaving_jgt
I’m 15 years into my career and I write Haskell every day. I’m getting a massive productivity boost from using an LLM.
black_knight
How do you find the quality of the Haskell code produced by LLM? Also, how do you use the LLM when coding Haskell? Generating single functions or more?
null
ardit33
LLMs help a lot in doing 'well defined' tasks, and things that you already know you want, and they just accelerate the development of it. You still have to re-write some of it, but they do the boring stuff fast.
They are not great if your tasks are not well defined. Sometimes, they suprise you with great solutions, sometimes they produce mess that just wastes your time and deviates from your mission.
To, me LLMs have been great accelerants when you know what you want, and can define it well. Otherwise, they can waste your time by creating a lot of code slop, that you will have to re-write anyways.
One huge positive sideffect, is that sometimes, when you create a component, (i.e. UI, feature, etc), often you need a setup to test, view controllers, data, which is very boring and annoying / time wasting to deal. LLM can do that for you within seconds (even creating mock data), and since this is mostly test code, it doesn't matter if the code quality is not great, it just matters to get something in the screen to test the real functionality. AI/LLMs have been a huge time savers for this part.
Terr_
I get the impression that the software scenarios where LLMs do the best on both reliability and time-saving are places where a task was already ripe (or overdue) to be be abstracted away: Turned into a reusable library; as as a default implementation or setting; expressed as a shorter DSL; or a template/generator script.
When it's a problem lots of people banged their head against and wrote posts about similar solutions, that makes for good document-prediction. But maybe we should've just... removed the pain-point.
borzi
The 10% reduction in hiring for young workers is entirely because industry (software, manufacturing) at least in the united states (and probably the world) is contracting and in recession, while the services and government sector has been the main sector growing since a long time now - completely due to economic and geopolitical reasons, nothing to do with AI.
elric
Some anecdata: I have a buddy who runs a smallish consulting firm, and another buddy who's a software recruiter. Both are telling me that they've stopped hiring developers except for senior developers. From this limited sample size, the market seems to be convinced that it no longer needs or wants junior developers because their tasks can be outsourced to LLMs.
Outside of tech, my eulogy writer friend got fired and replaced by ChatGPT. So when gramma dies, someone will now read a page of slop at her funeral instead of something that a person with empathy wrote.
sokoloff
I’m sorry for your friend’s job loss, but reading a page of fairly generic eulogy written by someone who knew nothing of grandma beyond what the family said and basic obituary facts doesn’t seem massively different from a page of fairly generic eulogy written by an AI based on the same inputs.
If you can find a workable way to put the family in the improvements loop, the AI eulogy could be far better at expressing the family’s sentiment about grandma. (I’m not going to want to go 3 rounds of edits over 2-3 days with a human to get it just right, but going 8 rounds with an AI in a 20-30 minute sitting is appealing and would give a better result in a lot of cases.)
Under those conditions, how much more am I willing to pay for a human-written eulogy? $0 at most, and probably a negative amount.
johnfn
Genuine curiosity: how do you know that software is in a recession? What measures do you use to determine this? And how do you know that the recession is not AI driven? I don't think it is either, but it's more of a feeling; I'm not sure how I would make that argument more grounded.
borzi
Well, the best measurement is hiring slowdown and bankruptcies. Bankruptcies in the US are up 13.1%.
Traditionally, you wouldn't look at the release of a productivity tool coinciding with a hiring slowdown and assume that it's automation causing the hiring slowdown, your first instinct would be that the sector is not doing well.
sameermanek
Check sum of free cash flow of major tech co.s. you'll soon find out that the cash needle of industry as a whole is not moving that much.
They are just round tripping the cash that was sitting in their accounts through investments that make their way back through advertising channels or compute channels.
Once you see the bigger picture, you'll realise its all just a Fugazi post covid
whatever1
To me it seems that LLMs are a tool that only increase productivity for given headcount in dimensions that were neglected in the past.
For example, everyone now writes emails with perfect grammar in a fraction of a time. So now the expectation for emails is that they will have perfect grammar.
Or one can build an interactive dashboard to visualize their spreadsheet and make it pleasing. Again the expectation just changed. The bar is higher.
So far I have not seen productivity increase in dimensions with direct sight to revenue. (Of course there is the niche of customer service, translation services etc that already were in the process of being automated)
manmademagic
It's an interesting dilemma, since if I know that an email was written mostly with AI, it feels to me like the author didn't put effort in, and thus I won't put much effort into reading the email.
I had a conversation with my manager about the implications of everyone using AI to write/summarise everything. The end result will most likely be staff getting Copilot to generate a report, then their manager uses Copilot to summarise the report and generate a new report for their manager, ad inifinitum.
Eventually all context is lost, busywork is amplified, and nobody gains anything.
osn9363739
My experience has been interesting. I have been sending super short, mostly dot point emails since before LLMS. Pre ChatGPT I used to cop a bit of shit about it. Now thought, people love it.
chii
> Eventually all context is lost, busywork is amplified
why not fire everyone in between the top-most manager and the actual "worker" doing the work, as the report could be generated with the correct level of summary?
manmademagic
Mostly because there are different depths of reporting required depending who you’re creating said reports for. Often it’s unnecessary bureaucracy, but also often the ones doing the “actual work” don’t have a full understanding of how what they’re working on interacts with other parts of a system. (I mean this broadly, and not just related to software development)
Middle management can sometimes be good at this, because they may actually have the time to step back and take a holistic look at things. It’s not always easy to do that when you’re deep in the weeds with clients, managers, colleagues, or direct reports bugging you about misc things.
Overall I think (or hope) the more useless reporting will die a slow death, but I also think there’ll be a loooooong period of AI slop before we reach the point where everyone says “why are we actually doing this?”
wiz21c
as google seems to do (seen that other HN post today)
https://www.cnbc.com/2025/08/27/google-executive-says-compan...
Wololooo
I'm sad to see this for several reasons because I do not expect or want everyone up use a LLM to converse with me via mail, the whole point is to exchange information, with everyone using a LLM as output and input, now the whole thing becomes a game of telephone.
You do not need to build a spreadsheet visualiser tool there are plenty of options that exist and are free and open source.
I'm not against advances, I'm just really failing to see what problem was in need of solving here.
The only use I can get behind is the translation, which admittedly works relatively well with LLMs in general due to the nature of the work.
rsynnott
> For example, everyone now writes emails with perfect grammar in a fraction of a time.
And absolutely bloody _hideous_ style, if they are using our friends the magic robots to do this.
sschueller
I don't have time to read paragraphs of AI slop emails. Please keep them short and to the point. No need to send it through an LLM.
dumbfoundded
Corporations will require everything going through an LLM to meet company standards.
user94wjwuid
Is 2 years or so big enough sample size for any conclusions? You’re also seeing massive money movements; last quarter larger than consumer spending (!!!) This money isn’t going to junior head counts(labor), it’s going to compute(capital) Also what’s everyone’s balance sheet really saying? Will this money movement to capital rather than labor ACTUALLY pay off ? I think that would take a 10year hind sight to prove no?
monster_truck
I've got a few buddies over at Microsoft, they've all said something along the lines of "I really hate using copilot. They at least let us use pre-approved models in VSCode, we get most that come out. But all AI metrics are tracked and there are layoffs every quarter. I have kids now man. Strange times. I know you would have quit months ago" and they're right.
mandeepj
Hopefully they have racked up a few million.
https://www.fool.com/investing/2024/11/29/this-magnificent-s...
digitcatphd
I had a team of developers and essentially told them all 'either learn to code with Claude' or you're out. What I found is the more junior developers started 'vibe coding' resulting in a net decrease in performance, where the more senior ones used it to accelerate their speed cautiously and selectively.
My conclusion was senior engineers were better because they were used to managing developers and taking on more managerial tasks building 'LLM Soft Skills' and also frankly fixing mistakes, the junior developers were pressured for speed and had their managers to correct them.
Within 12 months, despite extensive attempts, only the mid level team members remained.
Etheryte
This is literally the difference between a junior or senior though? You could've just as well said to go faster or you're out and you would've gotten the same result. LLMs are irrelevant to what you've done here.
bgwalter
The senior developers pretended to use Claude Steal to comply with your rude and ignorant request. They are better at programming and the soft skills required to appease an incompetent manager.
You just used "AI" to get rid of people you didn't want anyway, but since it was your order it had to be a "success story".
ggm
Any Board which supports management hollowing out future profits by either firing, or not hiring junior staff deserves to have their bonus rescinded.
Think like a forestry investor, not a cash crop next season.
joshdavham
It's going to be extremely interesting to see what the field of software dev will look like in a few years given how few juniors are getting hired recently.
rsynnott
Same as a few years after the dot com crash and after the financial crisis (assuming that this is a real phenomenon; I’m not currently convinced); a critical skills shortage.
(This isn’t unique to IT; this cyclical underinvest-shortage-panic pattern happens in a lot of industries.)
camillomiller
Corporate incentives are usually not pushing in this direction.
indymike
Now that bs work has next to no cost, I see a lot more bs work being done, and often on pointless bureaucratic activities involving generating questionnaires and answering them. It's as if the activities add up to a big net zero.
softwaredoug
I think mRNA vaccines and green energy are equally transformative economic opportunities. In the US though we are becoming a one trick pony. Instead of investing in all 3, we will prioritize AI because Silicon Valley sucked up to Trump in the recent election.
All to say we could have quite a bit more resilience as an economy, but we decided to sacrifice our leadership in these areas.
>early-career workers (ages 22-25) in the most AI-exposed occupations have experienced a 13 percent relative decline in employment even after controlling for firm-level shocks.
As a professor of software engineering, I'm feeling an existential crisis coming on. Are we preparing students in vain? Last term was the first time I had senior project students who didn't have a job lined up in the Fall of their final year. Maybe it's time to retire.