ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts

Saleh Ashkboos, Langwen Huang, Nikoli Dryden, Tal Ben-Nun, Peter Dominik Dueben, Lukas Gianinazzi, Luca Nicola Kummer, Torsten Hoefler

Keywords: Ensemble Post-Processing, Ensemble Weather Forecasting, Prediction Correction

Abstract: Post-processing ensemble prediction systems can improve the reliability of weather forecasting, especially for extreme event prediction. In recent years, different machine learning models have been developed to improve the quality of weather post-processing. However, these models require a comprehensive dataset of weather simulations to produce high-accuracy results, which comes at a high computational cost to generate. This paper introduces the ENS-10 dataset, consisting of ten ensemble members spanning 20 years (1998--2017). The ensemble members are generated by perturbing numerical weather simulations to capture the chaotic behavior of the Earth. To represent the three-dimensional state of the atmosphere, ENS-10 provides the most relevant atmospheric variables at 11 distinct pressure levels and the surface at \ang{0.5} resolution for forecast lead times T=0, 24, and 48 hours (two data points per week). We propose the ENS-10 prediction correction task for improving the forecast quality at a 48-hour lead time through ensemble post-processing. We provide a set of baselines and compare their skill at correcting the predictions of three important atmospheric variables. Moreover, we measure the baselines' skill at improving predictions of extreme weather events using our dataset. The ENS-10 dataset is available under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Author Statement: Yes

TL;DR: We introduce a dataset containing ten ensemble members over 20 years for post-processing ensemble weather forecasts.

URL: https://github.com/spcl/ens10

Dataset Url:

The ENS-10 dataset can be downloaded directly from https://storage.ecmwf.europeanweather.cloud/MAELSTROM_AP4/ or through a
downloader Python script. Examining the raw data, preprocessing, normalization, and training are all provided through a set of Python APIs
in https://github.com/spcl/ens10, and examples written in PyTorch, and Jupyter notebooks are in https://github.com/spcl/climetlab-maelstrom-ens10.

License:

(C) Copyright 2021 European Centre for Medium-Range Weather Forecasts (ECMWF)

Access to this dataset is governed by the following Terms of Use.

* ECMWF retains all Intellectual Property Rights and copyright over its data.
* ECMWF must be acknowledged (attributed) as the source.
* ECMWF does not accept any liability whatsoever for any error or omission in the data, their availability, or for any loss or damage arising from their use.
* ECMWF makes no warranty as to the accuracy or completeness of its data products or the uninterrupted provision of such data products. All data products are provided on an "as is" basis. Any warranty implied by statute or otherwise is hereby excluded to the fullest extent permissible by law.
* ECMWF shall not be liable should ECMWF discontinue the provision of its data products at any time.
* ECMWF shall have no liability in contract, tort or otherwise arising out of or in connection with these Terms of Use.
* Users must remove attribution if requested by ECMWF.
* These Terms of Use shall be governed by and construed in accordance with the laws of England and Wales.
* The parties shall attempt to settle any dispute between them in relation these Terms of Use and any data licensed under these Terms of Use in an amicable manner. If the dispute cannot be so settled, it shall be finally settled under the Rules of Arbitration of the International Chamber of Commerce by one or three arbitrators appointed in accordance with the said rules; sitting in London, England. The proceedings shall be in the English language. In accordance with Sections 45 and 69 of the Arbitration Act 1996, the right of appeal by either party to the English courts on a question of law arising in the course of any arbitral proceedings or out of an award made in any arbitral proceedings is hereby agreed to be excluded.
* Nothing in these Terms of Use shall be construed as a waiver of any of the privileges and immunities conferred upon ECMWF by its Member States through its Convention and Protocol on Privileges and Immunities.
* Access to the data products and services may be unavailable, delayed or interrupted. ECMWF will make reasonable efforts to restore the access following the report of a problem, but ECMWF will not be liable for, any unavailability, delay or interruption in access.

This data product is published under a Creative Commons Attribution 4.0 International (CC BY 4.0). To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/

You are free to:

* Share - copy and redistribute the material in any medium or format
* Adapt - remix, transform, and build upon the material for any purpose, even commercially.

Under the following terms:

* You must give appropriate credit (attribution) to ECMWF as outlined below, provide a link to the licence, and indicate if changes were made.
* No additional restrictions - You may not apply legal terms or technological measures that legally restrict others from doing anything the licence permits.

The following wording shall be attached to the use of this ECMWF data product:

1. Copyright statement: Copyright "© [year] European Centre for Medium-Range Weather Forecasts (ECMWF)".
2. Source: www.ecmwf.int
3. Licence Statement: This data is published under a Creative Commons Attribution 4.0 International (CC BY 4.0). https://creativecommons.org/licenses/by/4.0/
4. Disclaimer: ECMWF does not accept any liability whatsoever for any error or omission in the data, their availability, or for any loss or damage arising from their use.
5. Where applicable, an indication if the material has been modified and an indication of previous modifications.

Contribution Process Agreement: Yes

In Person Attendance: Yes