Time Series Modeling at Scale: A Universal Representation Across Tasks and Domains

Sabera J Talukder, Georgia Gkioxari

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Time Series, Tokenization, Transformers, VQVAE

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: One universal methodology for time series modeling at scale across 5 time series domains and 3 downstream tasks.

Abstract:

Time series are ubiquitous, capturing real-world phenomena ranging from human
neuronal firing and tectonic activity to atmospheric conditions. However, they are
challenging to analyze due to domain-specific timescales (e.g., sub-second for brain
activity and years for weather phenomena), complex multivariate relations, and
disparate modeling objectives. Prior works model time series by targeting specific
tasks, like forecasting, or distinct domains, like neural recordings. We introduce
a universal approach for scalable time series modeling across many tasks and
domains, which we call TOTEM: Tokenized Time Series Embeddings. We propose
a task-agnostic embedding that projects a continuous time series of any length
onto a discrete set of learned tokens. This embedding is derived by optimizing
a self-supervised objective formulated as a task-independent convolution-based
vector quantized variational autoencoder. Drawing inspiration from the recent
successes of Large Language Models, these discrete token sequences are then used
to learn downstream models with the powerful Transformer architecture. We show
that TOTEM matches or achieves SOTA performance on forecasting, classification,
and translation tasks with data drawn from a myriad of domains: neuroscience,
seismology, meteorology, power grids, and urban traffic. We further demonstrate
TOTEM’s scalability by introducing and evaluating it on new datasets, the largest
being ∼14× larger than existing benchmarks. Finally, we illustrate TOTEM’s
dominant zero-shot generalization capabilities across all of our downstream tasks.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6449