Climate science data can be compressed efficiently by dual-stage extreme compression with a variational auto-encoder transformer

Tao Han, Zhenghao Chen, Song Guo, Wanghan Xu, Wanli Ouyang, Lei Bai

Abstract: Climate change makes accurate weather prediction and large-scale data analysis increasingly crucial, but the sheer volume of weather data strains data storage and sharing. Here we introduce Aeolus, a deep learning framework with powerful Variational Auto-Encoder transFormer (VAEFormer) modules that dramatically reduces the size of weather datasets, compressing the widely used 400-terabyte reanalysis atmospheric dataset to just 0.85 TeraBytes (TB)-a compression ratio of 470  × . Aeolus surpasses traditional image compression methods such as Joint Photographic Experts Group 2000 (JPEG2000), which typically achieve much lower compression rates on weather data. It also enables fast data processing speeds of over 1 GigaByte (GB) per second, saving computational resources. Our tests show Aeolus maintains high accuracy, with temperature errors as low as 0.17∘ Kelvin and reliable climate patterns. These advances make it easier for researchers to access and analyze massive weather datasets, supporting better weather forecasting and climate studies. Aeolus thus offers a valuable tool for climate research and efforts to address global environmental challenges. 470-fold super compression of atmospheric data has been achieved through Aeolus, a two-stage deep-learning-based variational autoencoder framework.

External IDs:doi:10.1038/s43247-025-02903-z