VideoLights: A Cross-Modal Cross-Task Transformer Model for Joint Video Highlight Detection and Moment Retrieval

Dhiman Paul, Md Rizwan Parvez, Nabeel Mohammed, Shafin Rahman

Keywords: highlight detection, moment retrieval, video grounding

TL;DR: Joint moment retrieval and highlight detection from video based on natural language query.

Abstract: Video Highlight Detection and Moment Retrieval (HD/MR) are essential in video analysis. Recent joint prediction transformer models often overlook cross-task dynamics and video-text alignment. We propose VideoLights, a novel HD/MR framework addressing these limitations through: (i) Convolutional Projection and Feature Refinement modules with an intermodal alignment loss for better video-text feature alignment. (ii) Bi-Directional Cross-Modal Fusion network for strongly coupled query-aware clip representations. (iii) Uni-Directional joint-task feedback mechanism enhancing both tasks through correlation. In addition, we introduce hard positive/negative losses for adaptive error penalization and improved learning. Our approach includes intelligent pretraining and finetuning using synthetic data and features from various encoders. Comprehensive experiments on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate state-of-the-art performance.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9604