Long-term Time Series Forecasting with Vision Transformer
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: multivariate time series; forecasting; transformer;
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Transformer has been widely used for modeling sequential data in recent years. For example the Vision Transformer (ViT), which divides an image into a sequence of patches and uses Transformer to discover the underlying correlations between the patches, has become particularly popular in Computer Vision. Considering the similarity of data structure between time series data and image patches, it is reasonable to apply ViT or its variations for modeling time series data. In this work, we explore this possibility and propose the Swin4TS algorithm. It incorporates the window-based attention and hierarchical representation techniques from the Swin Transformer, a well-known ViT algorithm, and applies them to the long-term forecasting of time series data. The window-based attention enables the algorithm to achieve linear computational complexity, while the hierarchical architecture allows the representation on various scales. Furthermore, Swin4TS can flexibly adapt to channel-dependence and channel-independence strategies, in which the former can simultaneously capture correlations in both the channel and time dimensions, and the latter shows high training efficiency for large datasets. Swin4TS outperforms the latest baselines and achieves state-of-the-art performance on 8 benchmark datasets. More importantly, our results demonstrate the potential of transferring the Transformer architecture from other domains to time series analysis, which enables research on time series to leverage advancements at the forefront of other domains.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3334
Loading