NuTime: Numerically Multi-Scaled Embedding for Large- Scale Time-Series Pretraining
Abstract: Recent research on time-series self-supervised models shows great promise in learning semantic representations. However, it has been limited to small-scale datasets, e.g., thousands of temporal sequences. In this work, we make key technical contributions that are tailored
to the numerical properties of time-series data and allow the model to scale to large datasets, e.g., millions of temporal sequences. We adopt the Transformer architecture by first partitioning the input into non-overlapping windows. Each window is then characterized by its
normalized shape and two scalar values denoting the mean and standard deviation within each window. To embed scalar values that may possess arbitrary numerical amplitudes to high-dimensional vectors, we propose a numerically multi-scaled embedding module enumerating all possible numerical scales for the scalars. The model undergoes pretraining with a simple contrastive objective on a large-scale dataset over a million sequences collected by merging existing public data. We study its transfer performance on a number of
univariate and multivariate classification tasks, few shot learning, unsupervised clustering and anomaly detection benchmarks. Our method exhibits remarkable improvement against previous pretraining approaches and establishes the new state of the art, even compared with domain-specific non-learning-based methods.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: * enhance technical contribution description
* add response curves for NME module e(x)
* add an improved baseline for instance normalization with extra scale tokens
* expand limitation discussion about hyper-parameters
* add in-depth discussion about forecasting and future works
* a number of minor fixes
Assigned Action Editor: ~Yingnian_Wu1
Submission Number: 2492
Loading