Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis

Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Kamarthi, Aditya B. Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, B. Aditya Prakash

Keywords: Time Series Analysis, Multimodality, Large Language Models

Abstract:

Time series data are ubiquitous across a wide range of real-world domains. While
real-world time series analysis (TSA) requires human experts to integrate numerical series data with multimodal domain-specific knowledge, most existing TSA
models rely solely on numerical data, overlooking the significance of information beyond numerical series. This oversight is due to the untapped potential
of textual series data and the absence of a comprehensive, high-quality multimodal dataset. To overcome this obstacle, we introduce Time-MMD, the first
multi-domain, multimodal time series dataset covering 9 primary data domains.
Time-MMD ensures fine-grained modality alignment, eliminates data contamination, and provides high usability. Additionally, we develop MM-TSFlib, the
first-cut multimodal time-series forecasting (TSF) library, seamlessly pipelining
multimodal TSF evaluations based on Time-MMD for in-depth analyses. Extensive
experiments conducted on Time-MMD through MM-TSFlib demonstrate significant performance enhancements by extending unimodal TSF to multimodality,
evidenced by over 15% mean squared error reduction in general, and up to 40%
in domains with rich textual data. More importantly, our datasets and library
revolutionize broader applications, impacts, research topics to advance TSA. The
dataset is available at https://github.com/AdityaLab/Time-MMD.

Submission Number: 2011