MATE: Multimodal Time Series Forecasting via Adaptive Modality Fusion and Timestamp-Augmented Expert Modeling

Zhen Chen, Dalin Zhang, Huan Li, Zhichen Lai, Christian S. Jensen

Keywords: Time Series Forecasting, Multimodal, Mixture of Expert

Abstract: Time series forecasting is crucial for applications such as weather prediction, stock analysis, and power grid management. While early studies focus on modeling temporal patterns, recent studies explore multimodal approaches. However, most of these studies remain limited to converting time series into other modalities---such as images or text---without truly integrating external information. In this study, we propose MATE, a unified plug-in multimodal framework that enhances forecasting by integrating auxiliary modalities. It features a multimodal adaptive fusion mechanism, which dynamically selects informative modalities and routes temporal data to corresponding modality-aware experts, and a timestamp-augmented expert module that treats timestamps as an independent modality to improve temporal structure awareness. Empirically, MATE achieves up to 10.32\% and 12.54\% improvement in MAE over unimodal and multimodal models. Extensive ablation studies assess the contribution of each modality to forecasting accuracy. The implementation code is publicly available.

Primary Area: learning on time series and dynamical systems

Submission Number: 11405