Beyond the difficulty of running calculations (or even accurately measuring the current state), is there a reason to believe weather is unpredictable?
I would imagine we probably have a solid mathematical model of how weather behaves, so given enough resources to measure and calculate, could you, in theory, predict the daily weather going 10 years into the future? Or is there something inherently “random” there?
What you're describing is effectively how climate models work; we run a physical model which solves the equations that govern how the atmosphere works out forward in time for very long time integrations. You get "daily weather" out as far as you choose to run the model.
But this isn't a "weather forecast." Weather forecasting is an initial value problem - you care a great deal about how the weather will evolve from the current atmospheric conditions. Precisely because weather is a result of what happens in this complex, 3D fluid atmosphere surrounding the Earth, it happens that small changes in those initial conditions can have a very big impact on the forecast on relatively short time-periods - as little as 6-12 hours. Small perturbations grow into larger ones and feedback across spatial scales. Ultimately, by day ~3-7, you wind up with a very different atmospheric state than what you'd have if you undid those small changes in the initial conditions.
This is the essence of what "chaos" means in the context of weather prediction; we can't perfectly know the initial conditions we feed into the model, so over some relatively short time, the "model world" will start to look very different than the "real world." Even if we had perfect models - capable of representing all the physics in the atmosphere - we'd still have this issue as long as we had to imperfectly sample the atmosphere for our initial conditions.
So weather isn't inherently "unpredictable." And in fact, by running lots of weather models simultaneously with slightly perturbed initial conditions, we can suss out this uncertainty and improve our estimate of the forecast weather. In fact, this is what's so exciting to meteorologists about the new AI models - they're so much cheaper to run that we can much more effectively explore this uncertainty in initial conditions, which will indirectly lead to improved forecasts.
Say you had a massive array of billions of perfect sensors in different locations, and had all the computing power to process this data, would an N year daily forecast then be a solved problem?
For the sake of the argument I’m ignoring ”external” factors that could affect the weather (e.g meteors hitting earth, changes in man-made pollution, etc)
At that point you're slipping into Laplace's Demon.
In practical terms, we see predictability horizons get _shorter_ when we increase observation density and spatial resolution of our models, because more, small errors from slightly imperfect observations and models still cascade to larger scales.
Yes, this is effectively what 4DVar data assimilation is [1]. But it's very, very expensive to continually run new forecasts with re-assimilated state estimates. Actually, one of the _biggest_ impacts that models like GraphCast might have is providing a way to do exactly this - rapidly re-running the forecast in response to updated initial conditions. By tracking changes in the model evolution over subsequent re-initializations like this, one could might be able to better quantify expected forecast uncertainty, even moreso than just by running large ensembles.
Expect lots of R&D in this area over the next two years...
> Present understanding is that this chaotic behavior limits accurate forecasts to about 14 days even with accurate input data and a flawless model. In addition, the partial differential equations used in the model need to be supplemented with parameterizations for solar radiation, moist processes (clouds and precipitation), heat exchange, soil, vegetation, surface water, and the effects of terrain.
I think there is a hope that DL models wont have this problem.
AFAIK there's nothing random anywhere except near atomic/subatomic scale. Everything else is just highly chaotic/hard-to-forecast deterministic causal chains.
Cloud formation is affected by cosmic ray flux. It's effectively random.
But the real problem is chaos - which says that even with perfect data, unless you also have computations with infinite precision and time/spatial/temperature/pressure/etc resolution, eventually you wind up far from reality.
The use of ensembles reduces the effect of chaos a bit, although they tend to smooth it out - so your broad pattern 12 days out might be more accurately forecast than without them, but the weather at your house may not be.
Iterative DL models tend to smooth it faster, according to a recent paper.
I would imagine we probably have a solid mathematical model of how weather behaves, so given enough resources to measure and calculate, could you, in theory, predict the daily weather going 10 years into the future? Or is there something inherently “random” there?