HTMformer: Hybrid Time and Multivariate Transformer for Time Series Forecasting

Tan Wang, Yunwei Dong, Tao Zhang, Qi Wang

Keywords: Time Series Forecasting, Transformer

TL;DR: This paper argues that thorough extraction of multivariate features within the embedding layer can significantly improve the performance of Transformers.

Abstract:

Transformer-based methods have achieved impressive results in time series forecasting.
However, existing Transformers still exhibit limitations in sequence modeling as they tend to overemphasize temporal dependencies. This incurs additional computational overhead without yielding corresponding performance gains.
We find that the performance of Transformers is highly dependent on the embedding method used to learn effective representations.
To address this issue, we extract multivariate features to augment the effective information captured in the embedding layer, yielding multidimensional embeddings that convey richer and more meaningful sequence representations.
These representations enable Transformer-based forecasters to better understand the series.
Specifically, we introduce Hybrid Temporal and Multivariate Embeddings (HTME).
The HTME extractor integrates a lightweight temporal feature extraction module with a carefully designed multivariate feature extraction module to provide complementary features, thereby achieving a balance between model complexity and performance.
By combining HTME with the Transformer architecture, we present HTMformer, leveraging the enhanced feature extraction capability of the HTME extractor to build a lightweight forecaster.
Experiments conducted on eight real-world datasets demonstrate that our approach outperforms existing baselines in both accuracy and efficiency.

Primary Area: learning on time series and dynamical systems

Submission Number: 1956