Localized Linear Temporal Dynamics for Self-supervised Skeleton Action Recognition
Keywords: Skeleton-based action recognition, Self-supervised learning, Contrastive learning
Abstract: Skeleton-based action recognition has achieved remarkable performance on large-scale benchmark datasets. Nevertheless, the scarcity of annotated skeleton action data poses a significant obstacle to the advancement of this field, which inspired the recent endeavors to explore self-supervised approaches. Among them, contrastive learning based methods have gained significant attention, wherein existing approaches primarily employ a spatial-temporal backbone to extract features from the action sequence and then conduct contrastive learning within the feature space. Skeleton action sequence itself is a highly complex dynamical system, yet existing methods overlook the higher-order temporal information of the sequence. In this work, we introduce Koopman Temporal Contrastive Learning (KTCL), a Koopman theory inspired contrastive learning framework, which focuses on the localized latent dynamics of the sequence by learning discriminative linear system dynamics. Given an action sequence, we first map it into a new space where the temporal evolution becomes linear. A dynamics-oriented contrastive loss is used to enforce the dynamics of positive (or negative) samples more similar (or dissimilar). To tackle the diverse dynamics across different action phases within one sequence, we further introduce segment-level localized linear dynamics, accompanied by a cross-matching mechanism for alignment. Additionally, a cross-order contrastive loss is proposed to further amplify the effect of contrast across features of different orders. Intensive experiments on four benchmark datasets show that the proposed methods achieve superior performance than competing methods.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 1113
Loading