Measuring the environmental impact of AI inference

Here's what happened:

1. Google rolled our AI summaries on all of their search queries, through some very tiny model 2. Given worldwide search volume, that model now represents more than 50% of all queries if you throw it on a big heap with "intentional" LLM usage 3. Google gets to claim "the median is now 33x lower!", as the median is now that tiny model giving summaries nobody asked for

It's very concerning that this marketing puff piece is being eaten up by HN of all places as evidenced by the other thread.

Google is basing this all of "median" because there's orders of magnitudes difference betwen strong models (what most people think of when you talk AI) and tiny models, which Google uses "most" by virtue of running them for every single google search to produce the summaries. So the "median" will be whatever tiny model they use for those models. Never mind that Gemini 2.5 Pro, which is what everyone here would actually be using, may well consume >100x much.

It's absurdly misleading and rather obvious, but it feels like most are very eager to latch on to this so they can tell themselves their usage and work (for the many here in AI or at Google) is all peachy. I've been reading this place for years and have never before seen such uncritical adoption of an obvious PR piece detached from reality.

If you have a market for it, the hardware industry will aggressively dig in to try to deliver. Maximum performance and maximum efficiency. So I can imagine there is still more to go.

I'm sure the relatively clean directed computational graph + massively parallel + massively hungry workload of AI is a breath of fresh air to the industry.

Hardware gains were for the longest time doing very little for consumers because the bottlenecks were not in the hardware but instead in extremely poorly written software running in very poorly designed layers of abstraction that nothing could be done about.

What’s the cost of training vs inference?

If it’s like Marvel sequels every year then there is a significant added training cost as the expectations get higher and higher to churn out better models every year like clockwork.

Finally someone using drop not in a teenage slang sense.

There are two ways to make AI cheaper: make energy cheaper or make AI hardware and algorithms more efficient and use less energy that way. Google is investing in doing both. And that's a good thing.

I actually see growth in energy demand because of AI or other reasons as a positive thing. It's putting pressure on the world to deliver more energy cheaply. And it seems the most popular and straightforward way is through renewables + batteries. The more clean and cheap capacity like that is added, the more marginalized traditional more expensive solutions get.

The framing on this topic can be a bit political. I prefer to look at this through the lens of economics. The simple economic reality is that coal and gas plant construction has been bottle necked for years on a lot of things to the point where only very little of it gets planned and realized. And what little comes online has pretty poor economics. The cost and growth curves for renewables+battery paint a pretty optimistic picture here with traditional generation plateauing for a while (we'll still build more coal/gas plants, not a lot, and they'll be underutilized) and then dropping rapidly second half of the century as cost and availability of alternatives improves and completely steam roll anything that can't keep up. Fossil fuel based generation could be all but gone by the 2060s.

There are lots of issues with regulations, planning, approval, etc for fossil fuel based generation. There are issues with supply chains for things like turbines. Long term access to cooling water (e.g. rivers) is becoming problematic because of climate change. And there are issues with investors voting with their feet and being reluctant to make long term commitments in what could end up being very poor long term investments. A lot of this also impacts nuclear, which while clean remains expensive and hard to deliver. The net result of all this is that investments in new energy capacity are heavily biased towards battery + renewables. It's the only thing that works on short notice. And it's also the cheapest way to add new capacity. Current growth is already 80-90% renewable. It's not even close at this point. We're talking tens/hundreds of GW added annually.

Of course AI is so hungry for energy that there is a temporary increase in usage for coal/gas. That's existing underutilized plants temporarily getting utilized a bit more mainly because they are there and utilizing them a bit more is relatively easy and quick to realize. It's not actually cheaper and future cost reductions will likely come in the form of replacing that capacity with cheaper power generation as soon as that can be delivered.

Measurements for water consumption seems cherry-picked and incorrect to look better than they actually are. When asked about it, they doubled-down and incorrectly mentioned the study in question (to which they compared against) was incorrect. See https://www.linkedin.com/posts/shaolei-ren-68557415_today-go...

Cost/prompt is a ratio. "Prompt" is not a normalized metric that is stable over time. It can increase (as context lengths increase) or decrease (as google's product suite integrates llms).

How do you drop something by 33x? That is literally impossible unless they make money by purchasing energy.

This from quantizing their Gemini model?

There are a lot of anecdotal reports of quality differences following some Gemini 2.5 Pro releases earlier in the year.

They didn’t account for training. From the paper:

> LLM training & data storage: This study specifically considers the inference and serving energy consumption of an Al prompt. We leave the measurement of Al model training to future work.

This is disappointing, and no analysis is complete without attempting to account for training, including training runs that were never deployed. I’m worried these numbers would be significantly worse and that’s why we don’t have them.

[dupe] https://news.ycombinator.com/item?id=44972808

I'm on record as pretty stridently anti-AI Hype Bullshit (I was calling Altman a criminal back when that had real-world consequences, check the history).

But this is in the vanishing minority of frontpage AI threads where it's a really interesting concersation about quantifiable things: what quantization, what engagement metrics, what NDGC on downstream IR. People are complaining they gamed the number: that's an improvement! Normally they just lie. This is amenable to analysis and frankly an interesting one.

If it were up to me they'd flat regex ban "llm" and "ai" on HN, thats about the right ROC. But if we're going to have it? I'll take this over "How AI Saved My Vibecode Startup From Vibe Coding".