The Language of Time: a Language Model Perspective on Time Series Foundation Models

Yi Xie, Yun Xiong, Ziyue Li, Hao Niu, Zejian Shi, Sijia Peng, Zhengfu Liu

Keywords: Time Series Foundation Models, Patch-based Representation Learning, Distributional Motif Tokens

TL;DR: This paper proposes a framework named 'The Language of Time,' demonstrating that patch-based time series foundation models are essentially a generalization of the distributional tokens language models.

Abstract:

Large language models have established a successful paradigm of training foundation models on massive datasets, extending this approach to multiple domains. Time series foundation models extend this paradigm, demonstrating exceptional cross-domain transfer and prediction capabilities in both industrial and academic scenarios. This creates a paradox: while time series from different domains reflect distinct dynamical systems that should limit transferability, empirical results demonstrate remarkable cross-domain performance.
To resolve this paradox, this paper investigates the representation learning mechanisms and generalization capabilities of patch-based time series foundation models from both theoretical and experimental perspectives. We demonstrate that these models fundamentally extend language model representations from deterministic vectors to probabilistic distributions, enabling effective cross-domain transfer. Our analysis shows that time series patches can be quantized into discrete vocabularies with statistical properties similar to natural language.
This theoretical framework explains how time series models inherit the robust representation and transfer abilities of large language models, accounting for their superior performance in temporal tasks. Our work provides a rigorous theoretical foundation for understanding, evaluating, and improving the safety and reliability of large-scale time series foundation models for time series analysis.

Primary Area: learning on time series and dynamical systems

Submission Number: 5820