Learning to Embed Time Series Patches Independently

Seunghan Lee, Taeyoung Park, Kibok Lee

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: self-supervised learning, masked time-series modeling, contrastive learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose patch-independent strategies for self-supervised representation learning via masked time series modeling.

Abstract:

Masked time series modeling has recently gained much attention as a self-supervised representation learning strategy for time series.
Inspired by masked image modeling in computer vision, recent works first patchify and partially mask out time series, and then train Transformers to capture the dependencies between patches by predicting masked patches from unmasked patches.
However, we argue that capturing such patch dependencies might not be an optimal strategy for time series representation learning;
rather, learning to embed patches independently results in better time series representations.
Specifically, we propose to use 1) the simple patch reconstruction task, which autoencode each patch without looking at other patches, and 2) the simple patch-wise MLP that embeds each patch independently.
In addition, we introduce complementary contrastive learning to hierarchically capture adjacent time series information efficiently.
Our proposed method improves time series forecasting and classification performance compared to state-of-the-art Transformer-based models, while it is more efficient in terms of the number of parameters and training time.
Code is available at this repository: https://github.com/seunghan96/pits.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 926