Temporal Repetition Counting with Dynamic Action Queries

Zishi Li, Qiuyan Shang, Xiaoxuan Ma, Wentao Zhu, Hai Ci, Yizhou Wang

Keywords: Computer Vision, Temporal Repetition Counting

Abstract: Temporal repetition counting aims to quantify the repeated action cycles within a video.The majority of existing methods rely on the similarity correlation matrix to characterize the repetitiveness of actions, but their scalability is hindered due to the quadratic computational complexity. In this work, we introduce a novel approach that employs an action query representation to localize repeated action cycles with linear complexity. Based on this representation, we further develop two key components to tackle the fundamental challenges of temporal repetition counting. Firstly, to facilitate open-set action counting, we propose the dynamic action query. Unlike static action queries, this approach dynamically embeds video features into action queries, offering a more flexible and generalizable representation. Second, to distinguish between actions of interest and background noise actions, we incorporate inter-query contrastive learning to regularize the video feature representation corresponding to different action queries. The experiments demonstrate that our method significantly outperforms the state-of-the-art methods in terms of both accuracy and efficiency. Specifically, our approach exhibits versatility in handling long video sequences, unseen actions, and actions at various speeds across two challenging benchmarks. Code and models will be publicly released.

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3031