Temporal MLP Bridges the Gap Between Embedding and Attention for Multivariate Time Series Forecasting

Zhinan Xie, Qi Zheng, Yaying Zhang

Abstract: Multivariate time series forecasting is crucial across various applications. In recent years, numerous studies adopt embedding layer and Attention mechanism to extract the intricate spatio-temporal features of time series. This involves directly transmitting the concatenated embeddings into the Attention mechanism. However, they generally overlook the importance of sending the integrated information in the embeddings into the Attention mechanism in a more appropriate way. To address this, we propose an intuitive network model with Temporal MLP Bridging the gap between Embedding and Attention (TMBEA) to deal with the above issue. Specifically, we explore a light-weight bridge with simple Multi-Layer Perceptrons (MLPs) fusing features along the temporal dimension, processing the embeddings before feeding them into the canonical Attention networks, which help embeddings to better align with the subsequent Attention networks. Experiments on real-world datasets, traffic datasets and air pollutant concentration datasets, demonstrate the efficiency of model. Further studies also show the capacity of bridge in improving the robustness of the model.