Accumulation of Cognitive Debt When Using an AI Assistant for Essay Writing Task
83 comments
·June 17, 2025mjburgess
Al-Khwarizmi
> In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to say.
But what fraction of communication is "worthwhile"?
I'm an academic, which in theory, should be one of the jobs that requires the most thinking. And still, I find that over half of the writing I do are things like all sorts of reports, grant applications, ethics/data management applications, recommendation letters, bureaucratic forms, etc. Which I wouldn't class as "worthwhile" in the sense that they don't require useful thinking, and I don't care one iota whether the text sounds like me or not as long as I get the silly requirement done. For these purposes, LLMs are a godsend and probably actually help me think more because I can devote more time to actual research and teaching, which I do in person.
codeduck
> In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people
I believe that one of the most underappreciated skills in business is the ability to string a coherent narrative together. I attend many meetings with extremely-talented engineers who are incapable of presenting their arguments in a manner that others (both technical and non-technical) can follow them. There is an artistry to writing and speaking that I am only now in my late forties beginning to truly appreciate. Language is a powerful tool, the choice of a single word can sometimes make or break an argument.
I don't see how LLMs can do anything but significantly worsen this situation overall.
bonoboTP
> I believe that one of the most underappreciated skills in business is the ability to string a coherent narrative together. I attend many meetings with extremely-talented engineers who are incapable of presenting their arguments in a manner that others (both technical and non-technical) can follow them.
Yes, but the arguments they need to present are not necessarily the ones they used to convince themselves, or their own reasoning history that made them arrive at their proposal. Usually that is an overly boring graph search like "we could do X but that would require Y which has disadvantage Z that theoretically could be salvaged by W, but we've seen W fail in project Q and especially Y would make such a failure more likely due to reason T, so Y isn't viable and therefore X is not a good choice even if some people argue that Y isn't a strict requirement, but actually it is if we think in a timeline of several years and blabla" especially if the decision makers have no time and no understanding of what the words X, Y, Z, W, Q, T etc. truly mean. Especially if the true reason also involves some kind of unspeakable office politics like wanting to push the tools developed by a particular team as opposed to another or wanting to use some tech for CV reasons.
The narrative to be crafted has to be tailored for the point of view of the decision maker. How can you make your proposal look attractive relative to their incentives, their career goals, how will it make them look good and avoid risks of trouble or bad optics. Is it faster? Is it allowing them to use sexy buzzwords? Does it line up nicely with the corporate slogan this quarter? For these you have to understand their context as well. People rarely announce these things, and a clueless engineer can step over people's toes, who will not squarely explain the real reason for their pushback, they will make up some nonsense, and the clueless guy will think the other person is just too dumb to follow the reasoning.
It's not simply about language use skills, as in wordsmithing, it's also strategizing and putting yourself in other people's shoes, trying to understand social dynamics and how it interacts with the detailed technical aspects.
mjburgess
To give a brief example of this -- a college asked why an exec had listened to my argument but not theirs recently, despite "saying the same thing". I explained that my argument contained actual impacts: actual delays, actual costs, an actual timeline when the impact would occur -- rather than nebulous "there will be problems".
Everyone comes to execs with hypothetical problems that all sound like people dressing up minor issues -- unless you can give specific details, justifications, etc. they're not going to parse properly.
This would be one case where a person asking an LLM for help is not even aware of the information they lack about the person they're trying to talk to.
We could define expertise this way: that knowledge/skill you need to have to formulate problems (, questions) from a vague or unknown starting point.
Under that definition, it becomes clear why LLMs "in the large" pose problems.
je42
I find that LLM extremely good in training such language skills by using following process:
a) write a draft yourself.
b) ask the LLM to correct your draft and make it better.
c) newer LLMs will explicitly mention the things they corrected (otherwise ask for being explict about the changes)
d) walk through each of the changes and apply the ones you feel that make the text better
This helped me improving my writing skills drastically (in multiple languages) compared to the times where I didn't have access to LLMs.
darkwater
Done this as well. But after the initial "wow!" moment, the "make it better" part became a "actually I don't like how you wrote it, it doesn't sound like me".
There is a thin line between enhancing and taking over, and IMO the current LLMs cross it most of the time.
bayindirh
I use Grammarly sometimes to check my more serious texts, but there's a gotcha. If you allow all of its stylistic choices, your writing becomes very sterile and devoid of any soul.
Your word and structural choices adds a flair of its own, makes something truly yours and unique. Don't let the tool kill that.
jddj
> another technology that society will inoculate itself against
I like the optimism. We haven't developed herd immunity to the 2010s social media technologies yet, but I'll take it.
Ekaros
> I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people
Now one like me might go and ask how much of communication is actually worthwhile? Sometimes I consider that there is lot of communication that might not actually be. It is still done, but if no one actually reads it, why not automate generation.
Not to say there is not significant amount of stuff you actually want to get right.
mjburgess
It's not about getting it right, its about having thought about it. Authoring means thinking-thru, owning, etc.
There's a tremendous hollowing-out of our mental capacities caused by the computer science framing of activities in terms of input->output, as-if the point is to obtain the output "by any means".
It would not matter if the LLM gave exactly the same output as you had written, and always did. Because you still have to act in the world with thoughts that you needed have when authoring it.
supriyo-biswas
> It's not about getting it right, its about having thought about it. Authoring means thinking-thru, owning, etc.
So much this.
At my current workplace, I was asked to write up a design doc for a software system. The contents of the document itself weren't very relevant as the design deviated significantly based on constraints and feedback that could be discovered only after beginning the implementation, but it was the act of putting together that document, thinking about the various cases, etc. that lead to the formation of a mental model that helped me work towards delivering that system.
lazyasciiart
“Plans are worthless, planning is invaluable”
kibibu
> It is still done, but if no one actually reads it, why not automate generation.
There's a reason the real-estate industry has been able to go all-in on using AI to write property listings with almost no consumer kickback (except when those listings include hallucinated schools).
We're already used to treating them with skepticism, and nobody takes them at face value.
CuriouslyC
Shallow take. LLMs are like food for thought -- the right use in the right amounts is empowering, but too much (or uncritical use) and you get fat and lazy, metaphorically speaking.
You wouldn't go around crusading against food because you're obese.
Another neat analogy is to children who are too dependent on their parents. Parents are great and definitely help a child learn and grow but children who rely on their parents for everything rather than trying to explore their limits end up being weak humans.
mjburgess
> You wouldn't go around crusading against food because you're obese.
My eateries I step into are met with revulsion at the temples to sugary carbohydrates they've become.
> about 40.3% of US adults aged 20 and older were obese between 2021 and 2023
Prey your analogy to food does not hold, or else, we're on track for 40% of americans to acquiring mental disabilities.
NilMostChill
Shallow take.
Your analogies only work if you don't take in to account there are different degrees of utility/quality/usefulness of the product.
People absolutely crusade against dangerous food, or even just food that has no nutritious benefit.
The parent analogy also only holds up on your happy path.
hansmayer
It's all already there. When you converse with a junior-engineer about their latest and greatest idea (over a chat platform), and they start giving you real-time responses which are a page long and structured into bullet points...it's not even that they are using chatgpt to avoid thinking, it is the fact that they think either no-one will notice, or that this is how grown-ups actually converse with each other, is terrifying.
jstummbillig
I see it as more of a calibration, revolving around understanding what an AI is inherently not able to do – decide what YOU want – and stopping to be weird about that. If you chose to stop being involved in a process and mold it, then your relationship to that process and the outcome will necessarily change. Why would we be surprised by that?
As soon as we stop treating AI like mind readers things will level out.
barrell
It’s been my experience that most people opinions on AI is inversely proportional to the timescale they have been using it.
Using AI is kind of like having a Monika Closet. You just push all the stuff you don’t know to the side until it’s out of view. You then think everything is clean, and can fool yourself into thinking so for a while.
But then you need to find something in that closet and just weep for days.
je42
> All participants were then reassured that though 20 minutes might be a rather short time to write an essay, they were encouraged to do their best.
Given that the task has been under time pressure, I am not sure this study helps gauging the impact of LLMs in other contexts.
When my goal is to produce the result for a specific short term task - I maximize tool usage.
When my goal is to improve my personal skills - I use the LLM tooling differently optimizing for long(er) term learning.
einrealist
"I"? You should treat yourself as an anecdotal exception.
You are reading on HN. You are probably more aware about the advantages and shortcomings of LLMs. You are not a casual user. And that's the problem with our echo chamber here.
greekanalyst
"...the LLM group's participants performed worse than their counterparts in the Brain-only group at all levels: neural, linguistic, scoring."
That's not surprising but also bleak.
fhd2
Appears to align with good old Ironies of Automation [1]. If humans just review and rubber stamp results, they do a pretty terrible job at it.
I've been thinking for a while now that in order to truly make augmented workflows work, the mode of engagement is central. Reviewing LLM code? Bah. Having an LLM watch over my changes and give feedback? Different story. It's probably gonna be difficult and not particularly popular, but if we don't stay in the driver's seat somehow, I guess things will get pretty bleak.
tuatoru
Didn't realise the pedigree of the idea went back to 1983.
I read about this in a book "Our Robots, Ourselves". That talked about airline pilots' experience with auto-land systems introduced in the late 1990s/ early 2000s.
As you'd expect after having read Ironies of Automation, after a few near misses and not misses, auto-land is not used any more. Instead, pilot augmentation with head-up displays is used.
What is the programming equivalent of a head-up display?
fhd2
Certainly a relatively tight feedback loop, but not too tight. Syntax errors are very tight, but non negotiable: Fix it now.
Test failures are more explicit, you run tests when you want to and deal with the results.
Code review often has a horrible feedback loop - often days after you last thought about it. I think LLMs can help tighten this. But it can't be clippy, it can't interrupt you with things that _may_ be problems. You have to be able to stay in the flow.
For most things that make programmers faster, I think deterministic tooling is absolutely key, so you can trust it rather blindly. I think LLMs _can_ be really helpful for helping you understand what you changed and why, and what you may have missed.
Just some random ideas. LLMs are amazing. Incorporating them well is amazingly difficult. What tooling we have now (agentic and all that) feels like early tech demos to me.
stevage
>What is the programming equivalent of a head-up display?
Syntax highlighting, Intellisense, and the millions of other little features built into modern editors.
pantalaimon
> We must negate the machines-that-think. Humans must set their own guidelines. This is not something machines can do. Reasoning depends upon programming, not on hardware, and we are the ultimate program! Our Jihad is a "dump program." We dump the things which destroy us as humans!
amunozo
What I still wonder is whether using LLMs is helpful in some ways, or it is, as other users say, just useful for man-made problems such as corporate communication or bureaucracy. I use it for coding and it makes me confident to tackle new things.
I try to use it to understand the code or to implement changes I am not familiar with, but I tend to overuse them a lot. Would it be better, if used ideally (i.e. only to help learning and guiding), to just try it harder before using this or using a search engine? I wonder what's the most optimal use of LLMs in the long run.
Magmalgebra
Well... yes? Essays are tools to force students to structure and communicate thinking - production of the essay forces the thinking. If you want an equivalent result from LLMs you're going to need a much more iterative process of critique and iteration to get the same kind of mental effort out of students. We haven't designed that process yet.
bayindirh
I mean, they found brain atrophy. If this doesn't get someone worried, I don't know what would.
I joked that "I don't do drugs" when someone asked me whether I played MMORPGs, but this joke becomes just too real when we apply it to generative AI of any soırt.
Magmalgebra
As someone who used to teach, this does not worry me (also, they mention skill atrophy - inherently less concerning).
Putting ChatGPT in front of a child and asking them to do existing tasks is an obviously disasterous pedagogical choice for the reasons the article outlines. But it's not that hard to create a more constrained environment for the LLM to assist in a way that doesn't allow the student to escape thinking.
For writing - it's clear that finding the balance on how much time you ordering your thoughts and getting the LLM to write things is its own skillset, this will be its own skill we want to teach independent of "can you structure your thoughts in an essay"
falcor84
> I mean, they found brain atrophy.
Where did you get that from? While the article mentions the word "atrophy" twice, it's not something that they found. They just saw less neural activation in regards to essay writing in those people who didn't write the essay themselves. I don't anything there in regards to the brain as a whole.
bayindirh
If physical exercise builds muscle mass, mental work and exercise builds more connections in your brain.
Like everything, not using something causes that thing to atrophy. IOW, if you depend on something too much, you'll grow dependency on it, because that part of your body doesn't do the work that much anymore.
Brain is an interconnected jungle. Improvement in any ability will improve other, adjacent abilities. You need to think faster to type faster. If you can't think faster, you'll stagnate, for example.
Also, human body always tries to optimize itself to reduce its energy consumption. If you get a chemical from outside, it'll not produce it anymore, assuming the supply will be there. Brain will reduce its connections in some region if that function is augmented by something else.
Same for skill atrophies. If you lose one skill, you lose the connections in your brain, and that'll degrade adjacent skills, too. As a result, skill atrophy is brain atrophy in the long run.
eru
> I joked that "I don't do drugs" when someone asked me whether I played MMORPGs, [...]
I thought WoW was an off-label contraceptive?
HPsquared
LLMs are the tip of the iceberg when it comes to this stuff.
dzonga
I worry about the adverse effects of LLM on already disfranchised populations - you know the poor etc - that usually would have to pull themselves up using hard work etc studying n reading hard.
now if you don't have a mentor to tell you in the age of LLM you still have to do things the hard / old school way to develop critical thinking - you might end up taking shortcuts and have the LLMs "think" for you. hence again leaving huge swaths of the population behind in critical thinking which is already in shortage.
LLMs are bad that they might show you the sources but also hallucinate about the sources. & most people won't bother going to check source material and question it.
eru
LLMs are great for the poor!
If you are rich, you can afford a good mentor. (That's true literally, in the sense of being rich in money and paying for a mentor. But also more metaphorically for people rich in connections and other resources.)
If you are poor, you used to be out of luck. But now everyone can afford a nearly-free mentor in the form of an LLM. Of course, at the moment the LLM-mentor is still below the best human mentors. But remember: only rich people can afford these. The alternative for poor people was essentially nothing.
And AI systems are only improving.
supriyo-biswas
If people are using it to critically question their beliefs and thinking, that is.
However, most of the hype around LLMs is that they take out the difficult task of thinking and allow the creation of the artifact (documents, code or something else) that is really dangerous.
eru
How is it any worse than the status quo for the disenfranchised?
rglullis
People could in theory also get a college-level education by watching videos on YouTube, but in practice the masses just end up watching Mr. Beast.
15 years ago, people were sure that the Khan Academy and Coursera would disrupt Ivy League and private schools, because now one good teacher could reach millions of students. Not only this has not happened, the only movement I'm observing against credentialism is that I have good amount of anecdata showing kids preferring to go to trade school instead of university.
> pull themselves up using hard work etc studying n reading hard.
Where are you from? "The key to success is hard work" is not exactly something part of the Gen Z and Zoomers core values, at least not in the Americas and Western Europe.
jonplackett
I think we need to shift our idea of what LLMs do and stop thinking they are ‘thinking’ in any human way.
The best mental description I have come up with is they are “Concept Processors”. Which is still awesome. Computers couldn’t understand concepts before. And now they can, and they can process and transform them in really interesting and amazing ways.
You can transform the concept of ‘a website that does X’ into code that expresses a website X.
But it’s not thinking. We still gotta do the thinking. And actually that’s good.
panstromek
Concept Processor actually sounds pretty good, I like it. That's pretty close to how I treat LLMs.
eru
Are you invoking a 'god of the gaps' here? Is 'true' thinking whatever machines haven't mastered yet?
jonplackett
Not at all, I don’t think humans are magic at all.
But I don’t think even the ‘thinking’ LLMs are doing true thinking.
It’s like calling pressing the autocomplete buttons on your iPhone ‘writing’. Yeah kinda. It mostly forms sentences. But it’s not writing just because it follows the basic form of a sentence.
And an LLM, though now very good at writing is just creating a very good impression of thinking. When you really examine what it’s outputting it’s hard to call it true thinking.
How often does your LLM take a step back and see more of the subject than you prompted it to? How often does it have an epiphany that no human has ever had?
That’s what real thinking looks like - most humans don’t do tonnes of it most of the time either - but we can do it when required.
panstromek
Interesting. This says a different thing than what I thought from the title. I thought this will be about cognitive overload from having to process and review all the text the LLM generates.
I had to disable copilot for my blog project in the IDE, because it kept bugging me, finishing my sentences with fluff that I'd either reject or heavily rewrite. This added some mental overhead that makes it more difficult to focus.
MasihMinawal
I'm curious to see how the EEG measurements might change if someone uses LLM extensively over a longer period of time (fe about a year).
ozgune
This was on HN's front page yesterday. Here's the discussion:
kotaKat
People getting dumber using an LLM as their daily crutch? Say it isn't so!
kaelandt
One thing that is also truly unappreciated is most of us humans actually enjoy thinking, and people are trying to make llms strip us from a fundamental thing we enjoy doing. Look at all the people that enjoy solving problems for the sake of it
It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to say. Authoring is two-thirds of the point of most communication.
Before this, of course, will be a dramatic "shallowness of thinking" shock that will have to occur before its ill-effects are properly inoculated against. It seems part of the expert aversion to LLMs -- against the credulous lovers of "mediocrity" (cf. https://fly.io/blog/youre-all-nuts/) -- is an early experience of inoculation:
Any "macroscopic usage" of LLMs has, in any of my projects, dramatically impaired my own thinking, stolen decisions-making, and worsened my readiness for necessary adaptions later-on. LLMs are a strictly microscopic fill-in system for me, in anything that matters.
This isn't like calculators: my favourite algorithms for by-hand computation arent being "taken away". This is a system for substituting thinking itself with non-thinking, and radically impairs your readiness (, depth, adaptability, ownership) wherever it is used, on whatever domain you use it on.