Compressing high-resolution data through latent representation encoding for downscaling large-scale AI weather forecast model

Download PDF
Open Webpage

Qian Liu, Bing Gong, Xiaoran Zhuang, Xiaohui Zhong, Zhiming Kang, Hao Li

Published: 22 Oct 2024, Last Modified: 12 Nov 2025CrossrefEveryoneRevisionsCC BY-SA 4.0
Abstract: Qian Liu, Bing Gong, Xiaoran Zhuang, Xiaohui Zhong, Zhiming Kang, and Hao Li Abstract. The rapid advancement of artificial intelligence (AI) in weather research has been driven by the ability to learn from large, high-dimensional datasets. However, this progress also poses significant challenges, particularly regarding the substantial costs associated with processing extensive data and the limitations of computational resources. Inspired by the Neural Image Compression (NIC) task in computer vision, this study seeks to compress weather data to address these challenges and enhance the efficiency of downstream applications. Specifically, we propose a variational autoencoder (VAE) framework tailored for compressing high-resolution datasets, specifically the High Resolution China Meteorological Administration Land Data Assimilation System (HRCLDAS) with a spatial resolution of 1 km. Our framework successfully reduced the storage size of 3 years of HRCLDAS data from 8.61 TB to just 204 GB, while preserving essential information. In addition, we demonstrated the utility of the compressed data through a downscaling task, where the model trained on the compressed dataset achieved accuracy comparable to that of the model trained on the original data. These results highlight the effectiveness and potential of the compressed data for future weather research. This preprint has been withdrawn. How to cite. Liu, Q., Gong, B., Zhuang, X., Zhong, X., Kang, Z., and Li, H.: Compressing high-resolution data through latent representation encoding for downscaling large-scale AI weather forecast model, EGUsphere [preprint], https://doi.org/10.5194/egusphere-2024-3183, 2024. Received: 15 Oct 2024 – Discussion started: 22 Oct 2024 Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. While Copernicus Publications makes every effort to include appropriate place names, the final responsibility lies with the authors. Views expressed in the text are those of the authors and do not necessarily reflect the views of the publisher. Download & links Preprint (PDF, 22140 KB) Withdrawal notice This preprint has been withdrawn. Preprint (22140 KB) Download & links This preprint has been withdrawn. Preprint (22140 KB) Metadata XML BibTeX EndNote Share document.addEventListener("DOMContentLoaded", function () { const mobileShareElement = document.querySelector(".mobile-native-share"); if (navigator.share) { // Native sharing is available if (mobileShareElement) { mobileShareElement.style.display = "block"; } } else { // Native sharing is NOT available if (mobileShareElement) { mobileShareElement.style.display = "none"; } } });