Test-Time Adaptation for Video Highlight Detection

Zahidul Islam, Sujoy Paul, Mrigank Rochan

Keywords: Video Highlight Detection, Test-Time Adaptation, Self-supervised Learning, Meta Auxiliary Learning

Abstract: Existing video highlight detection methods often struggle to generalize due to varying content, styles, and audio-visual quality in unseen test videos. We propose Highlight-TTA, a test-time adaptation framework for video highlight detection that addresses this limitation by dynamically adapting the model during inference to better align with the specific characteristics of each test video, thereby improving its generalization and highlight detection performance. Highlight-TTA is jointly optimized during training using a self-supervised auxiliary task, cross-modality hallucinations, alongside the primary task of highlight detection within a meta-auxiliary training scheme to enable effective adaptation. During testing, we adapt the trained model using the self-supervised auxiliary task on the test video to enhance its highlight detection performance. Extensive experiments on three benchmark datasets demonstrate the effectiveness of Highlight-TTA.

Submission Number: 3