Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series ForecastingDownload PDF

21 May 2021, 20:42 (edited 07 Jan 2022)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: Time Series Forecasting, Transformers, Deep Decomposition Model, Auto-Correlation
  • TL;DR: We renovate Transformer to a deep decomposition model Autoformer and propose a series-wise Auto-Correlation mechanism based on series periodicity to replace self-attention. Autoformer surpasses SOTA by 38% relative accuracy promotion on six datasets.
  • Abstract: Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: https://github.com/thuml/Autoformer.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/thuml/Autoformer
17 Replies

Loading