OTIS: Learning High-Quality Time Series Features With Tiny Encoders

TMLR Paper9111 Authors

Abstract: We introduce \texttt{OTIS}, an \textbf{o}pen \textbf{ti}me \textbf{s}eries encoder that yields high-quality time series features for downstream deployment on \emph{any} system, including resource-constrained wearables and industrial sensors. Currently, the development of powerful general-purpose encoders relies on the scaling laws hypothesis, using large encoder sizes to memorise the heterogeneous distributions of multi-domain training data. However, this reliance on scale creates a barrier to real-world utility, rendering deployment on resource-constrained systems infeasible due to strict memory, energy, and latency constraints. Surprisingly, we find that tailoring standard masked modelling pre-training to time series properties yields a tiny $7.1\,$M encoder that matches the state-of-the-art performance of $54\times$ larger encoders across $162$ tasks, while requiring $10\times$ less memory, $43\times$ less energy, and $37\times$ lower latency. To achieve this without the capacity tax, we introduce three novel components: (1) a \textit{domain-aware tokeniser} to resolve conflicting semantics within multi-domain training data; (2) a \textit{dual masking strategy} to capture spatiotemporal structures and temporal causality; and (3) a \textit{structure-aware objective} to decouple feature learning from modelling noise. Consequently, \texttt{OTIS} produces high-quality time series features that enable state-of-the art performance in discriminative tasks and even extend seamlessly to generative tasks at minimal additional cost. To democratise access to powerful time series features on any system, we release our code and pre-trained weights.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Adin_Ramirez_Rivera1

Submission Number: 9111