The Gentle Singularity

No... the reason housing prices are high and rising is that not enough housing is getting build in the places people want to live. The main reason for that is that the people who already live in those places can block construction of new housing. That, and zoning.

Citation needed, my understanding was that housing prices are being driven up by real estate owners/agencies buying tons of property to either rent at extortionary prices or sit on until they sell for a higher price to the next sucker. Also stuff like the RealPage scandal where they simply illegally collude: https://news.ycombinator.com/item?id=41330007

I think the idea of a law that only allows a limited number of owned properties per person and requires them to actually be using those properties would be interesting to alleviate this.

That's not entirely accurate. Wages are real enough whether paid out as cash, or paid to a third party for the employee's benefit. Some medical costs are unreasonably inflated, but on the other hand much of the cost increase reflects greater capabilities. We can effectively treat many conditions today that would have been a death sentence in 1980, but some of those cutting edge treatments cost >$1M per patient. That doesn't directly benefit the average healthy consumer, but it can help a lot if you get sick.

I do agree that it makes no logical sense to couple medical insurance to employment. This system was created sort of accidentally as a side effect of wartime tax law and has persisted mainly due to inertia.

All costs. New roof on the house? $30k. Hail damage on the car? It’s actually totaled. Whatever is in the store? Inflation certainly hit there too.

How has inflation behaved since 1980?

No, that's a real wage increase. The healthcare system can effectively treat a lot more conditions than it could in 1980. That makes you richer as measured in terms of QALYs.

I think GDPPPP/capita is a better measure here, but the story is similar. From the IMF's 2025 numbers: USA $90k, China $29k, world average $26k.

Lovely tabulation of the data here: https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP)...

Purchasing power parity matters a lot too.

It varies a lot - check this out https://www.businessinsider.com/france-vs-us-real-wages-2015...

The US is kind of an outlier.

I guess if you have the opportunity to have your stuff made by cheap or free labour whether low paid Chinese or AI robots, societies have a choice how to distribute the benefits which has varied from everything to the rich in the US to fairly even in France say. Such choices will be important with AI.

They definitely do strain the neurology and thinking metaphors in that article. But the Dijkstra's algorithm and A* comparisons are the flipside of that same coin. They aren't trying to make it more effective. And definitely not trying to argue for anything AGI related.

Either way: They're tampering with the inference process, by turning circuits in the LLM on and off, in an attempt to prove that those circuits are related with a specific function. [0]

They noticed that circuits related to a token that is only relevant ~8 tokens forward were already activated on the newline token. Instead of only looking at the sequence of tokens that has been generated so far (aka backwards), and generating the next token based off of that information, the model is activating circuits related to tokens that are not relevant to the next token only, but to specific tokens a handful of tokens after.

So, information related to more than just the next upcoming token (including a reference to just one specific token) is being cached during a newline token. Wouldn't call that thinking, but I don't think calling it planning is misguided. Caching this sort of information in the hidden state would be an emergent feature, rather than a feature that was knowingly aimed at by following a specific training method, unlike with models that do test time compute. (DeepSeek-R1 paper being an example, with a very direct aim at turbocharging test time compute, aka 'reasoning'. [1])

The way they went at defining the function of a circuit, was by using their circuit tracing method, which is open source so you can try it out for yourself. [2] Here's the method in short: [3]

> Our feature visualizations show snippets of samples from public datasets that most strongly activate the feature, as well as examples that activate the feature to varying degrees interpolating between the maximum activation and zero.

> Highlights indicate the strength of the feature’s activation at a given token position. We also show the output tokens that the feature most strongly promotes / inhibits via its direct connections through the unembedding layer (note that this information is typically more meaningful for features in later model layers).

[0]: https://transformer-circuits.pub/2025/attribution-graphs/bio... [1]: https://arxiv.org/pdf/2501.12948 [2]: https://github.com/safety-research/circuit-tracer [3]: https://transformer-circuits.pub/2025/attribution-graphs/met...

> In both cases you're adding another layer to an existing algorithm in order to make it more effective. Doesn't make either AGI.

Yet. The human mind is a big bag of tricks. If the creators of AI can enumerate a large enough list of capabilities and implement those, then the product can be as good as 90% of humans, but at a fraction of the cost and a billion times the speed - then it doesn't matter if it's AGI or not. It will have economic consequences.

I guess that depends on what you think is coherent. A key finding is that the larger the network the more coherent the representation becomes. One example is larger networks merge the same concept across different languages into a single concept (as humans do). The addition circuits are also fairly easy to interpret.

Yea whenever we get into this sort of “what’s happening in the network is like what’s going on in your brain” discussion people never have concrete evidence of what they’re talking about.

The diversity is itself indicative, though, that intelligence isn't bound to the particularities of the human nervous system. Across different animal species, nervous systems show a radical diversity. Different architectures; different or reversed neurotransmitters; entirely different neural cell biologies. It's quite possible that "neurons" evolved twice, independently. There's nothing magic about the human brain.

Most of your critique is surface level: you can add all kinds of different structural diversity to an ML model and still find learning. Transformers themselves are formally equivalent to "fast weights" (suppression and control signals). Continuous learning is an entire field of study in ML. Or, for injury, you can randomly mask out half the weights of a model, still get reasonable performance, and retrain the unmasked weights to recover much of your loss.

Obviously there are still gaps in ML architectures compared to biological brains, but there's no particular reason to believe they're fundamental to existence in silico, as opposed to myelinated bags of neurotransmitters.

The bullet list is a good point, but:

> We also know that biological neural systems continuously learn and adapt, for example in the face of injury. Large models just don't do these things.

This is a deliberate choice on the part of the model makers, because a fixed checkpoint is useful for a product. They could just keep the training mechanism going, but that's like writing code without version control.

> Also this thing about deeper and deeper realities? C'mon, it's surface level association all the way down!

To the extent I agree with this, I think it conflicts with your own point about us not knowing how human minds work. Do I, myself, have deeper truths? Or am myself I making surface level association after surface level association, but have enough levels to make it seem deep? I do not know how many grains make the heap.

LLMs are not a linear combination of training data.

Some LLMs have interacted with the outside world, such as through reinforcement learning while trying to complete tasks in simulated physics environments.

Just because humans can describe it, doesn't mean they can understand (predict) it.

And the web contains a lot more than people's expressions: think of all the scientific papers with tables and tables of interesting measurements.

Your point is not wrong, but I'd clarify a couple things;

The primary cost of a traditional plant is fuel. However a nuclear plant needs a tad more (qualified) oversight than a coal plant.

In the same way that the navy has specialists for running nuclear propulsion systems versus crew needed for diesel engines. Not surprisingly nuclear engines cost "more than fuel".

That cost may end up being insignificant in the long run, but cost is not zero. And shortages of staff would matter (like it does with ATC at the moment.)

Construction cost should be much lower than it is, but I don't think it'll be as cheap as say coal or diesel. The nature of the fuel would always require some extras, if only because the penalty-for-failure is so high. There's a difference between a coal-plant catastrophe and Chernobyl.

So there are costs gor running nuclear, I don't think it necessarily gets "too cheap to measure".

It is worth noting that the effect is jaw-droppingly stark. The regulators managed to invert the learning curve [0] so the more power plants get built the more expensive it gets! It is one of the most stunning failures of an industrial society in the modern era; the damage this did to us all is huge. It is disheartening that our leadership/society chose to turn their backs on the future and we're all lucky that the Chinese chose a different tack to the West's policy of energy failure. At least there are still people who believe in industry.

[0] https://www.mdpi.com/1996-1073/10/12/2169 - Fig 3

I don't buy into the "red tape" argument. For me it's all due to a lack of a business case. I think building (LWR) nuclear plants is just freaking expensive, has always been freaking expensive, was bankrolled for a while by governments because they thought it would get cheaper, and then largely abandoned in most countries when it proved to be an engineering dead end.

Here's an MIT study that dug into the reasons behind high nuclear construction cost. They found that regulatory burdens were only responsible for 30% of cost increases. Most of the cost overruns were because of needing to adapt the design of the nuclear plant to the site.

https://news.mit.edu/2020/reasons-nuclear-overruns-1118

Now, you can criticize the methodology of that study, but then you have to bring your own study that shows precisely which regulatory burdens are causing these cost overruns, and which of those are in excess. Is it in excess that we have strict workplace safety regulation now? Is it in excess that we demand reactor containment vessels to prevent meltdowns from contaminating ground water supplies? In order to make a good red tape argument I expect detail in what is excess regulation, and I've never seen that.

Besides, if "red tape" and fossil industry marketing was really the cause of unpopularity, and the business case for nuclear was real when installing plants at scale, you would see Russia and China have the majority of their electricity production from nuclear power.

- Russia is the most pro-nuclear country in the world, and even they didn't get past 20% of electricity share. They claim levelized cost for nuclear on the order of that of solar and wind, but I am very skeptical of that number, and anyone who knows anything about the Russian government's relation to the truth will understand why. When they build nuclear plants in other countries (e.g. the bangladesh project) they are not that cheap.

- China sits at low single digit percentages of nuclear share, with a levelized cost that is significantly higher than Russia's and well above that of solar and wind. While they're planning to grow the nuclear share they assume it will be based on new nuclear technology that changes the business case.

Both Russia and China can rely on cheap skilled labor to bring down costs, a luxury western countries do not have.

And this is ultimately the issue: the nuclear industry has been promising designs that bring down costs for over half a century, and they have never delivered on that promise. It's all smoke and mirrors distracting from the fact that building nuclear plants is inherently freaking expensive. Maybe AI can help us design better and cheaper nuclear power plants, but as of today there is no proven nuclear plant design that is economical to build, and that is ultimately why you see so little new nuclear plant construction in the west.

Nuclear plants have other long tails costs, decommissioning and waste containment.

> So if you build enough nuclear plants to power the grid, charging per kWh for the electricity is pointless because the marginal cost of the fuel is so low.

Wouldn't there still be the OpEx of maintaining the power plants + power infrastructure (long-distance HVDC lines, transformer substations, etc)? That isn't negligible.

...I say that, but I do note that I live in a coastal city with a temperate climate near a mountain watershed, and I literally do have unmetered water. I suppose I pay for the OpEx of the infrastructure with my taxes.

> Unsurprisingly, this scared the crap out of the fossil fuel industry in the US and countries like Russia that are net exporters of fossil fuels, so they've spent decades lobbying to bind nuclear plant construction up in red tape to prevent them being built and funding anti-nuclear propaganda.

This is frankly nonsense, and my hope is that this nonsense is coming from a person too young to remember the real, valid fears from disasters like 3 Mile Island, Chernobyl, and Fukushima.

Yes, I fully understand that over the long term coal plants cause many more deaths, and of course climate change is an order of magnitude worse, eventually. The issue is that human fears aren't based so much on "the long term" or "eventualities". When nuclear power plants failed, they had the unfortunate side effect of making hundreds or thousands of square miles uninhabitable for generations. The fact that societies demand heavy regulation for something that can fail so spectacularly isn't some underhanded conspiracy.

Just look at France, the country with probably the most successful wide-scale deployment of nuclear. They are rightfully proud of their nuclear industry there, but they are not a huge country (significantly smaller than Texas), and thus understand the importance of regulation to prevent any disasters. Electricity there is relatively cheap compared to the rest of Western Europe but still considerably higher than the US average.

I always thought with nuclear electricity it would actually make more sense to charge by the max watts a customer can draw.

weird i didn't know they could build nuclear power plants (surely you are being sarcastic?)

also this is weird i thought electricity prices only get cheaper? https://fred.stlouisfed.org/series/APU000072610

this is not the same thing and all the AI CEOs would agree with me. very absurd statement to try to use for your argument.

but here is some data bro! https://fred.stlouisfed.org/series/APU000072610

Instead of always being 30 years in the future, it's now always 15 years in the future!

good luck w that

If any of that were true, then the llm would be actively involved in advancing themselves, or assisting humans in a meaningful way in the endeavor, which they’re not so far as I can tell.

For Sam Altman, honestly yes. The man has said some truly wild things for the VC

There's really no defensible way to call one "learning" and the other not. You can carry a half-full context window (aka prompt) with you at all times. Maybe you can't learn many things at once this way (though you might be surprised what knowledge can be densely stored in 1m tokens), but it definitely fits the GP's definition of (1) real-time and (2) based on a few examples.

Excellent link, thank you for sharing this!

> Do you have any proof whatsoever that non-programmers are using LLMs to write small bespoke apps for them successfully?

I’m radiologist, I’ve been paying for software that sped up my reporting like 200 usd per month. I’ve remade all the functionality I need in one evening with cursor and added some things that I’ve found missing from the original software.

My company is a non-tech company that is in manufacturing. I'm not a programmer and niether is anyone else here.

So far I have created 7 programs that are now used daily in production. One of them replaces a $1k/yr/usr CAD package, and another we used to bring in a contractor to write. The others a miscellaneous apps for automating/simplifying our in houses processes. None of the programs are more than 6k LOC.

And often you are still supervised by sometimes rude security guards who demand the receipts. So you have the worst of all worlds: You pay the supermarket, do the work and are treated like dirt.

On a tangent: what about self scanning? Here in Sweden that is the norm. For the last ten years I've brought my own bags, packed everything like I want it as I grab it - with a quick blip from the scanner - and then checkout is no line, no cashier, no unpacking and packing things again. Just scanning my card and going through a brief screen to check that the summary looks right.

Instead of a guard there's a scanner by the exit gate where you scan your ticket. In my case just a small stub since the actual ticket is sent digitally.

I think it works so smoothly, much better than before this system was introduced. My recent experience buying groceries in Belgium was very similar.

>That's great, but Becky at the farmer's market doesn't want to be re-inventing notepad on her computer to sell sourdough.

That's exactly why she asks an LLM to do it for her. A program like this would be <1k LOC, well within even the current LLM one-shot-working-app domain.

And yet I'd much rather self checkout than have a cashier do it for me.

IMO the problem with the current software landscape isn't the number of applications, it's the quality and availability of integrations. Everything is in it's own little kingdom, so the communication overhead is extreme and grows exponentially. And by overhead I mean human cognition - I need to remember what's in my notes, and then put that in my calendar, and then email it to myself or something, and on and on.

We have these really amazing computers with very high-quality software but still, so many processes are manual because we have too many hops from tool to tool. And each hop is a marginal increase in complexity, and an increase in risk of things going wrong.

I don't think AI will solve this necessarily, and it could make it worse. But, if we did want to improve it, AI can act as a sort of spongy API. It can make decisions in the face of uncertainty.

I imagine a world where I want to make a doctor's appointment and I don't open any apps at all, or pick up the phone. I tell my AI assistant, and it works it out, and I have all my records in one place. Maybe the AI assistant works over a phone line, like some sort of prehistoric amalgamation of old technology and new. Maybe the AI assistant doesn't even talk to a receptionist, but another AI assistant. Maybe it gives me spongy human information, but speaks to other's of it's kind in more defined interfaces and dialects. Maybe, in the future, there are no applications at all. Applications are, after all, a human construction. A device intended to replace labor, but a gross approximation of it.

What the android/apple appstores have taught me compared to the old symbian appstore is that we don't need more apps, we need decent ones. The current appstores have more.

The issue as I have observed it is that software is written to cover as many use cases for as many users as possible. Obviously executives want to sell their product to as many hands as possible.

But end users often only need a tiny fraction of what the software is fully capable of, leaving a situation where you need you need to purchase 100% to use 5%.

LLMs can break down this wall by offering the user the ability to just make a bespoke 5% utility. I personally have had enormous success doing this.

Micro-services are part of a system, so to create anything meaningful with them your context window has to include knowledge about the rest of the system anyway. If your system is large and complex, that means a huge context window even if your code itself is small.

Micro service systems are just huge sprawling code bases with more glue code. Calling something over a network instead of via a local function call is still calling something.

(average) Watt-hours would still be the right unit to measure that, if the question is about energy efficiency / environmental impact. You still care how much energy those agents consume to achieve a given task, and the time it takes for tasks is still variable, so you need to multiply by time.

Money is irrelevant to energy efficiency.

To be clear, I'm only saying the unit is correct, not whether the number of watt-hours is accurate or (assuming it is correct) that it's a good amount to spend on each query.

A watt wouldn't make any sense.

A watt needs a time unit to be converted to joules because a watt is a measure of power, while a joule is a measure of energy.

Think of it like speed and distance:

A watt is a RATE of energy use, just like speed (e.g., miles per hour) is a RATE of travel.

A joule is a total amount of energy, just like distance (e.g., miles) is a total amount of travel.

If you're traveling 60 mph but you don't specify for how long... You won't know how far you went, how much gas you used, etc.

Watt-hours is the only unit that makes sense here as we're describing the energy cost of some calculation. Watt, a rate unit, is not appropriate.

Watt-hours are what matter. If you want to boil a kettle, you need to use a certain minimum amount of watt-hours to do it, which corresponds to, say, burning a certain amount of propane. If you care about how many times you can boil a kettle on a bottle of propane, you care about the watt-hours expended. If you had perfect insulation you could boil a kettle by applying barely any watts for a long enough time and it would take the same amount of propane.

There are situations where Watts per se matter, eg if you build a datacenters in an area that doesn't have enough spare electricity capacity to generate enough watts to cover you, and you'd end up browning out the grid. Or you have a faulty burner and can't pump enough watts into your water to boil it before the heat leaks out, no matter how long you run it.

Surreal shitposting you say?

https://www.instagram.com/reel/DE3EhrZyCVd/

Why would you expect the average person to have exceptional taste to produce things you like? Go listen to a random Spotify track, you probably won't like that either. Art still needs to be curated and given some direction. Metas random AI feed created by the average FB user is indicative of nothing.