Signature Kernel Scoring Rule: A Spatio-Temporal Diagnostic for Probabilistic Weather Forecasting

Archer Dodson, Ritabrata Dutta

Abstract: Modern weather forecasting has increasingly transitioned from numerical weather prediction (NWP) to data-driven machine learning forecasting techniques. While these new models produce probabilistic forecasts to quantify uncertainty, their training and evaluation may remain hindered by conventional scoring rules, primarily MSE, which are designed for single time point predictions and ignore the highly correlated data structures present in weather behaviour. This work introduces the signature kernel scoring rule to the domain of weather forecasting, which reframes weather variables as continuous paths to encode temporal and spatial dependencies through iterated integrals. Validated as strictly proper through the use of path augmentations to guarantee uniqueness, the signature kernel provides a theoretically robust metric for forecast verification and model training. Empirical evaluations through weather scorecards on WeatherBench 2 models demonstrate the signature kernel scoring rule's high discriminative power and unique capacity to capture path-dependent interactions. Following previous demonstration of successful adversarial-free probabilistic training, we train sliding window generative neural networks using a predictive-sequential scoring rule on ERA5 reanalysis weather data. Using a lightweight model, we demonstrate that signature kernel based training outperforms climatology for forecast paths of up to fifteen timesteps.

Submission Type: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=mxzsfDb5mg

Changes Since Last Submission:

Following reviewer suggestions, we have made considerable revisions to the content of the text:

We have cut and moved non-critical content to the appendix to produce a normal length submission.
We have fixed issues with typos, definitions, references, and the quality of tables and language choices.
We have moved the numerical instability section (4.3) to the main body following recommendations by multiple reviewers.
We have added the github repository to open-source the code and allow replication.
We have added a new simulation experiment in the appendix to better illustrate how the signature kernel score responds to different degrees of spatial and temporal correlations.
We clarified our novelty claims to better highlight our original contributions.

The second revision has addressed the remaining open issues:
We have replaced Figure 3 with a vectorised version.
We have expanded Section 4.3 and Appendix G.
The github has environment setup files and a license.
The paper has been thoroughly proofread, with remaining grammatical errors fixed.

The third revision expanded the proof of strict propriety in Appendix E.

The camera-ready version included minor formatting edits.

Code: https://github.com/SignatureKernelScoreWeatherForecasting/Signature-Kernel-Scoring-Rule-A-Spatio-Temporal-Diagnostic-for-Probabilistic-Weather-Forecasting

Assigned Action Editor: ~Wesley_Maddox1

Submission Number: 6306