Towards Long-Context Time Series Foundation Models

Nina Żukowska, Mononito Goswami, Michał Wiliński, Willa Potosnak, Artur Dubrawski

Keywords: Time Series Foundation Models, Multivariate Time Series, Forecasting, Classification, Transformers

TL;DR: Context expansion to multiple channels in Time Series Foundation Models.

Abstract:

Time series foundation models have shown impressive performance on a variety of tasks, across a wide range of domains, even in zero-shot settings. However, most of these models are designed to handle short univariate time series as an input. This limits their practical use, especially in domains such as healthcare with copious amounts of long and multivariate data with strong temporal and intra-variate dependencies. Our study bridges this gap by cataloging and systematically comparing various context expansion techniques from both language and time series domains, and introducing a novel compressive memory mechanism to allow encoder-only TSFMs to effectively model intra-variate dependencies. We demonstrate the benefits of our approach by imbuing MOMENT, a recent family of multi-task time series foundation models, with the multivariate context.

Submission Number: 63