Optimizing Multi-Range based Error-Bounded Lossy Compression for Scientific Datasets

Yuanjian Liu, Sheng Di, Kai Zhao, Sian Jin, Cheng Wang, Kyle Chard, Dingwen Tao, Ian T. Foster, Franck Cappello

Abstract: Vast volumes of scientific data cannot be stored and transferred efficiently because of limited I/O bandwidth, network bandwidth, and storage capacity. Error-bounded lossy compression can be an effective method for resolving these big data issues, since not only can it significantly reduce the data size but it can also control the data distortion based on user-defined error bounds. In practice, many scientific applications have specific data fidelity requirements across different value ranges/intervals of the dataset for the lossy compression, in order to guarantee that the reconstructed data are valid for post hoc analysis. Existing state-of-the-art error-bounded lossy compressors, however, do not support multi-range based error-bounds in the lossy compression, leaving a critical gap that hampers their effective use in practice. In this work, we address this issue by proposing a multi-range based error-bounded lossy compressor based on the state-of-the-art SZ lossy compressor. Our approach allows users to set different error bounds in different value ranges for a compressoin task. We evaluate our approach on several real-world datasets and show that it can obtain a higher visual quality or data fidelity on reconstructed data with the same or even higher compression ratios achieved by SZ.