Dual-level Prototypes Guidance for Single-frame Temporal Action Localization

Jinrong Sheng, Wenshi Li, Ao Li, Yongxin Ge

Keywords: Temporal action localization, Single-frame annotations, Memory bank, Graph matching random walk

Abstract: In recent, single-frame temporal action localization (STAL) has captured the attention of the computer vision community. Due to the sparse single-frame annotations, current STAL methods generally employ pseudo-labels strategies to bridge the gap between weakly-supervised methods and fully-supervised methods. However, these methods derive pseudo-labels from single-frame of the corresponding instances, yet the intra-class affinity from the current single-frame to other action snippets remains neglected. To capitalize on this affinity, we design a dual-level prototypes guidance (DPG) method with the graph matching random walk (Gm-Rw) algorithm to achieve instance-level and video-level prototype guidance for pseudo-labels refinement. For instance-level guidance, the Gm-Rw exploits the high affinity prototype among instances of the current video to build intra-class associations. For video-level guidance, an online memory bank is constructed to iteratively summarize more discriminative prototype. After Gm-Rw builds affinity among intra-class videos, an exponential moving average (EMA) mechanism is designed to achieve dual-level prototypes guidance for pseudo-labels refinement. Notably, the dual-level guidance is mutually reinforcing, prompting us to propose a novel adaptive collaborative strategy (ACS) for dynamic optimization. Extensive experiments on THUMOS14, GTEA, BEOID, and ActivityNet1.3 reveal that our method significantly outperforms state-of-the-art methods.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6863