A Semantic Data Augmentation driven Nested Adapter for Video Moment Retrieval

Arkaprabha Bhandari, KAJAL KANSAL, Yongkang Wong, Jianquan Liu, Mohan Kankanhalli

Keywords: Moment Retrieval, Highlight Detection, Adapter, Data Augmentation

Abstract:

Existing transformer-based video-moment retrieval models achieve sub-optimal
performance when using the pretrain-finetuning learning paradigm – a pretrained
multimodal encoder is finetuned using the target training data. While current work
has explored different model architectures and training paradigms to explore this
problem, the problem of data dilemma has been under addressed. Specifically,
there exists high diversity of how semantic is captured in textual query and the
training dataset only consist of limited moment-query pairs for the highly diverse moments. This work addresses this problem with a novel nested adaptor
and a LLM-driven semantic data generation pipeline. First, a LLM-driven data
augmentation generates queries that are semantically similar to the ground truth,
which enrich the semantic boundary captured by textual query. We empirically
analyze the effectiveness of data augmentation, and proposed a simple yet effective quality measure to retain high quality samples. Second, we propose a novel
nested adapter that utilises both augmented queries and human annotated queries
for model coarse-tuning and fine-tuning, respectively. By combining semantic
perturbation with domain adaptation, our approach addresses the variability in
video content while capturing nuanced features more effectively. Experimental
results on various baseline models show the efficacy of our proposed approach.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9631