Enriching Time Series Representation: Integrating a Noise-Resilient Sampling Strategy with an Efficient Encoder Architecture

Anh Duy Nguyen, Trang H. Tran, Hieu Pham, Phi Le Nguyen, Lam M. Nguyen

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Time series analysis, Contrastive self-supervised learning, Representation learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Time series analysis has been an important research area for decades, and with the advent of foundation models, it has witnessed an explosive surge in interest. Contrastive self-supervised learning stands out as a powerful technique to learn representations capable of solving a wide range of downstream tasks. However, there have been several challenges that persist.
First, there is no previous work explicitly considering noise, which is one of the critical factors affecting the efficacy of time series tasks.
Second, there is a lack of efficient yet lightweight encoder architectures that can learn informative representations robust to various downstream tasks.
To fill in these gaps, we initiate a novel sampling strategy that promotes consistent representation learning with the presence of noise in natural time series. In addition, we propose an encoder architecture that utilizes dilated convolution within the Inception block to create a scalable and robust network architecture with a wide receptive field. Experiments demonstrate that our method consistently outperforms state-of-the-art methods in forecasting, classification, and abnormality detection tasks, e.g. ranks first over two-thirds of the classification UCR datasets, with only 40% of the parameters compared to the second-best approach.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3667