The Logarithm Trick: achieve better long term forecast via Mean Logarithm Square Loss

Tianning Zhang, Rui Su, Tao Han, kang chen, Wanli Ouyang, LEI BAI

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: autoregressive, ai for science, time series, neural scientific simulations

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A logarithmic operation following the MSE loss enhance the finetuning performance for long-term forecasts in weather prediction and time series forecasting tasks.

Abstract: Weather forecasting and time series prediction can be modeled as autoregressive prediction tasks and optimized through a pretraining-finetuning paradigm. We discovered that simply incorporating an element-wise logarithmic operation following the standard square error loss, which we term MLSE, noticeably enhances long-term forecast performance in the fine-tuning phase. Remarkably, MLSE acts as a plug-and-play, zero-cost enhancement for autoregressive tasks. In this paper, we conduct a series of comprehensive experiments that support the effectiveness of MLSE. Furthermore, we present a phenomenological theory to dive into the feasibility and limitations of MLSE, by modeling the rate of error accumulation. Our findings propose a promising direction for understanding long-term prediction based on finite history.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1119