Exploring Representations and Interventions in Time Series Foundation Models

Michał Wiliński, Mononito Goswami, Willa Potosnak, Nina Żukowska, Artur Dubrawski

TL;DR: We investigate why time series foundation models work, the kinds of concepts that these models learn, and how can these concepts be manipulated to influence their outputs?

Abstract: Time series foundation models (TSFMs) promise to be powerful tools for a wide range of applications. However, their internal representations and learned concepts are still not well understood. In this study, we investigate the structure and redundancy of representations across various TSFMs, examining the self-similarity of model layers within and across different model sizes. This analysis reveals block-like redundancy in the representations, which can be utilized for informed pruning to improve inference speed and efficiency. We also explore the concepts learned by these models, such as periodicity and trends. We demonstrate how conceptual priors can be derived from TSFM representations and leveraged to steer its outputs toward concept-informed predictions. Our work bridges representational analysis from language and vision models to TSFMs, offering new methods for building more computationally efficient and transparent TSFMs.

Lay Summary:

Everyday things like the weather, stock prices, heart-rate readings or traffic counts are all examples of time-series data, i.e. numbers that arrive one after another over time. Researchers have recently begun training very large, general-purpose AI models on many kinds of these sequences. Because a single such model can “found” (i.e., serve as the starting point for) many different forecasting tasks, scientists call them foundation models — similar to how ChatGPT is a single text model used for lots of language tasks.

In this study we opened that black box. First, we peered inside several leading time-series foundation models and found that many of their layers were doing almost identical work. By safely trimming these duplicates we cut the models’ size and made their predictions up to 50% faster, without harming accuracy.

Next, we asked what the models actually know. We discovered they naturally learn simple concepts people care about, such as whether a signal is flat, rising, falling or repeating in cycles. Finally, we showed that we can gently turn the model’s internal “knobs” to inject these concepts on demand: for example, turning a flat forecast into one that rises, or adding a seasonal ups-and-downs pattern. Crucially, this steering happens after training, so users don’t need to retrain the model or supply extra data.

Together, these findings make large time-series AI models cheaper to run, easier to understand and easier to control, paving the way for safer, more efficient tools in areas from finance to healthcare.

Link To Code: https://github.com/moment-timeseries-foundation-model/representations-in-tsfms

Primary Area: Applications->Time Series

Keywords: Time Series Foundation Models, Model Steering, Interpretability, Pruning

Submission Number: 7669