Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval

Yang Liu, Qianqian Xu, Peisong Wen, Siran Dai, Qingming Huang

Abstract: The rapid growth of online video resources has significantly promoted the development of video retrieval methods. As a standard evaluation metric for video retrieval, Average Precision (AP) assesses the overall rankings of relevant videos at the top list, making the predicted scores a reliable reference for users. However, recent video retrieval methods utilize pair-wise losses that treat all sample pairs equally, leading to an evident gap between the training objective and evaluation metric. To effectively bridge this gap, in this work, we aim to address two primary challenges: a) The current similarity measure and AP-based loss are suboptimal for video retrieval; b) The noticeable noise from frame-to-frame matching introduces ambiguity in estimating the AP loss. In response to these challenges, we propose the Hierarchical learning framework for Average-Precision-oriented Video Retrieval (HAP-VR). For the former challenge, we develop the TopK-Chamfer Similarity and QuadLinear-AP loss to measure and optimize video-level similarities in terms of AP. For the latter challenge, we suggest constraining the frame-level similarities to achieve an accurate AP loss estimation. Experimental results present that HAP-VR outperforms existing methods on several benchmark datasets, providing a feasible solution for video retrieval tasks and thus offering potential benefits for the multi-media application.

Primary Subject Area: [Content] Media Interpretation

Secondary Subject Area: [Systems] Data Systems Management and Indexing

Relevance To Conference: The rapid extension of online video resources has posed tough challenges for efficient video search and analysis, highlighting the need for advanced content-based video retrieval methods, which serve as a crucial component for various multi-media applications such as recommendation systems, video processing, and online education. In this work, we propose a hierarchical learning framework for video retrieval based on Average Precision (AP) optimization, to fill the gap between training objectives and evaluation metrics that the previous works have overlooked. Specifically, we design a video-oriented similarity measure and a surrogate AP loss with proper gradients, constraining both the video-level and frame-level similarities to achieve an accurate AP loss estimation. Experimental results reveal that our framework frequently surpasses existing methods on several downstream tasks, which provides a feasible and promising solution for large-scale video understanding and management. We hope our work could bring potential benefits for broader applications and support subsequent research to further contribute to the multi-media community.

Submission Number: 2607