Project Aardvark: reimagining AI weather prediction

The World Meteorological Organization has data that national weather services exchange to make forecast. Apart from that, its on a country-by-country basis.

In France for instance Météo-France has released all of its historical data in January 2024: https://www.data.gouv.fr/fr/organizations/meteo-france/#/dat...

> Kind of like how fintech algos can be run against historic stock market data to evaluate them.

Backtesting, that's called.

> Is there a big Clearinghouse for this data?

The short answer is, "no." There are some projects like the "NNJA-AI" project at Brightband[1] which is attempting to create such a clearing house in order to focus research efforts across the community.

[1]: https://www.brightband.com/data/nnja-ai

> Are all the inputs (from buoys, weather balloons, stations, etc) from decades of history stored

I was also thinking about smartphones. They have barometric data, and while it might vary from phone to phone, I'm sure something like a kalman filter + historic data could do something there.

Think about gathering all the data from "stationary" phones, correlate that with weather sat data, and with real "ground truth" weather stations, and then go back 30 - 60 min / a day, and see what comes out.

It would also be nice to have a historical store of the weather predictions and not just instantaneous parameters. But I'm not aware of such record, perhaps because weathermen don't want a record of their (mis)predictions...

> Is there a big Clearinghouse for this data?

Many commercial and even third party governments rely on the data from NOAA and its archives, on top of that the EU runs its own EUMETSAT fleet of data, plus a ton of national services - unfortunately, the result is there's a looooot of datasets.

NOAA's dataset is public domain [1], EUMETSAT only requires attribution for most of its data [2]. On top of that you got the EU's Climate Data store [3], ECMWF [4], and ECA&D [5].

The service that many private weather services provide is to aggregate and weigh all of the publicly available datasets, and some also add in data from their own ground stations, commercially licensed "realtime" data from governmental services, and their own models as well.

The interesting question is what DOGE will do regarding NOAA - it is increasingly possible that NOAA will shut down, either to be turned into a pay-for-play model, replaced by private services, or just carelessly dropped in its entirety.

[1] https://www.ncei.noaa.gov/archive

[2] https://user.eumetsat.int/resources/user-guides/data-registr...

[3] https://cds.climate.copernicus.eu/

[4] https://www.ecmwf.int/en/forecasts/datasets

[5] https://www.ecad.eu/dailydata/

What data is used to train this AI? The article doesn't say anything about that (tough I have to admit I didn't read it super carefully). My first thought would be exactly all this historical data, but then you can't use that same data to test the AI's performance. Are different subsets of the available historical data used for training vs testing?

Is there an equivalent modelling approach for earthquake prediction? A data repository for it widely shared on the same line as the top comment would work

> Accurate (within 3deg F) weather predictions

Sniff test: ASTM E230 standard tolerances for the venerable Type K thermocouple is +/- 2.2°C or +/- 0.75%, whichever is greater.

Expectations are in need of recalibration if anyone thinks a single number is going to meaningfully achieve that level of accuracy across a volume representing any metro area at any given point in time, let alone two days out.

Well, take the actual weather observations for the past year. Take the actual weather observations for the last week. Overlay and slide this week of observations on top of the observations of the past year, until you find the window that matches "the best" — then take the day right after this window and predict that the weather will be just like that (or maybe try to tweak the values a bit).

I wonder how poorly this thing operates, and whether taking several years of history to look at would help much.

What are your use cases for such level of accuracy on such long time frames?

According to paper model grid resolution is 1.5 degrees. I do not think it can predict accurate weather in any location. It show global weather trends.

Indeed we have it tough out here. I like the excitement, though, I must say! Funny to find the sentiment here on HN, though!

Where in the city?

Let's name "they". It was Trump via DOGE.

I remember learning that humans trying to catch a ball are not actually able to predict where the ball will land, but rather, will move in a way that maintains the angle of movement constant.

As a result a human running to catch the ball over some distance (eg during a baseball game) runs along a curved path, not linearly to the point where the ball will drop (which would be evidence of having an intuition of the ball's destination).

> would be akin to how most humans see a ball in flight and just instinctively know (within reason) where the ball is going to go?

Up to a point .. and that point is more or less the same as the point where humans can no longer catch a spinning tennis raquet.

We understand the gravitional rainbow arc of the centre of mass, we fail at predicting the low order chaotic spin of tennis raquet mass distributions.

Other butterflies are more unpredictable, and the ones that land on a camels back breaking a dam of precariously balanced rocks are a particular problem.

* https://en.wikipedia.org/wiki/Tennis_racket_theorem

* Dzhanibekov effect demonstration in microgravity: https://www.youtube.com/watch?v=1x5UiwEEvpQ

* https://en.wikipedia.org/wiki/Horseshoe_map

I think “chain-of-thought” LLMs, with access to other tools like Python, already demonstrate two types of “thinking”.

We can query an LLM a simple question like “how many ‘r’ are in the word strawberry”, and an LLM with know access to tools will quite confidently, and likely incorrectly, give you an answer. There’s no actual counting happening, and any kind of understanding of the problem, the LLM will just guess an answer based on its training data. But that answer tends to be wrong, because those types of queries don’t make up a large portion of its training set, and if they do, there’s a large body of similar queries with vastly different answers, which ultimately results in confidentiality incorrect outputs.

But provide an LLM tools like Python, and a “chain-of-thought” prompt that allows it to recursively re-prompt itself, while also executing external code and viewing the outputs, and an LLM can easily get the correct answer to query “how many ‘r’ are in the word strawberry”. By simply writing and executing some Python to compute the answer.

Those two approaches to problem solving are strikingly similar to intuitive vs analytical thinking in humans. One approach is driven entirely by pattern matching, and breaks down when dealing with problems that require close attention to specific details, the other is much more accurate, but also slower because directed computation is required.

As for your hypothetical “weather AI”, I think it’s pretty easy to imagine an AGI capable of confidently predicting the weather tomorrow, not be capable of understanding how it computed the prediction, beyond a high level hand wavy explanation. Again, that’s basically what LLM do today, confidently make predictions of the future, with zero understanding of how or why they made those predictions. But you can happily ask an LLM how and why it made a prediction, and it’ll give you a very convincing answer, that will also be a complete and total deception.

> would be akin to how most humans see a ball in flight and just instinctively know (within reason) where the ball is going to go?

Generally no. If I show you a puddle of water, can you tell me what shape was the ice sculpture that it was melted from?

One is Newtonian motion and the other is a complex chaotic system with sparse measurements of ground truth. You can minimize error propagation but it’s a diminishing returns problem, (except in rare cases like for natural disasters where a 6h warning can make a difference).

To quote Iain M. Banks, probably not :)

> "Sma," the ship said finally, with a hint of what might have been frustration in its voice, "I'm the smartest thing for a hundred light years radius, and by a factor of about a million ... but even I can't predict where a snooker ball's going to end up after more than six collisions." [GCU Arbitrary in "The State of the Art"]

That's assuming we are actually tracking all the relevant indicators.

I think that's actually a pretty good way to frame how these deep learning-based forecasting models might evolve

> Would some hypothetical future AI just "know" that tomorrow it's going to be 79 with 7 mph winds, without understanding exactly how that knowledge was arrived at?

I think a consciousness with access to a stream of information tends to drown out the noise to see signal, so in those terms, being able to "experience" real-time climate data and "instinctively know" what variable is headed in what direction by filtering out the noise would come naturally.

So, personally, I think the answer is yes. :)

To elaborate a little more - when you think of a typical LLM the answer is definitely no. But, if an AGI is likely comprised of something akin to "many component LLMs", then one part might very well likely have no idea how the information it is receiving was actually determined.

Our brains have MANY substructures in between neuron -> "I", and I think we're going to start seeing/studying a lot of similarities with how our brains are structured at a higher level and where we get real value out of multiple LLM systems working in concert.

Arxiv version here https://arxiv.org/pdf/2404.00411

I think it’s replacing data assimilation. Ingesting observation data from lots of sources. Not replacing observations themselves.