Modeling Time Series as Text Sequence A Frequency-vectorization Transformer for Time Series Forecasting
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: time series prediction, frequency spectrum vectorization, self-supervised pretraining
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Time series is an essential type of sequential feature that widely exists in multiple scenarios (e.g., healthcare, weather prediction) and contains valuable information, so various studies have been conducted for modeling time series. Transformer-based models have achieved great success in modeling sequence data, especially text, but fail to understand time series sequences. The reason is that individual data points of time series are hard to utilize because they are numerical values that cannot be tokenized. To address this challenge, we design a frequency-vectorization time series forecasting method in this paper. Different from most previous studies that adopt frequency domain to extract extra features, we propose to utilize frequency spectrum as a common dictionary for tokenizing time series sequences, which converts single time series into frequency units with weights. Then, the vectorized frequency token sequence can be modeled by transformer layers directly for prediction tasks. Furthermore, to align the frequency and the time domains, we introduce two pretraining tasks: time series reconstruction task and maximum position prediction task. Experimental results on multiple datasets demonstrate that our model outperforms existing SOTA models, particularly showing significant improvements in long-term forecasting. Besides, our model exhibits remarkable transferability across various prediction tasks after pretraining.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 282
Loading