Global Prototype Encoding for Incremental Video Highlights Detection

Sen Pei, Shixiong Xu, Ye Yuan, Jiashi Feng, Xiaohui Shen, Xiaojie Jin

Keywords: Video highlights detection, Incremental learning, Newly released dataset

Abstract: Video highlights detection (VHD) is an active research field in computer vision, aiming to locate the most user-appealing clips given raw video inputs. However, most VHD methods are based on the closed world assumption, \emph{i.e.}, a fixed number of highlight categories is defined in advance and all training data are available beforehand. Consequently, existing methods have poor scalability with respect to increasing highlight domains and training data. To address above issues, we propose a novel video highlight detection method named \textbf{G}lobal \textbf{P}rototype \textbf{E}ncoding (GPE) to learn incrementally for adapting to new domains via parameterized prototypes. To facilitate this new research direction, we collect a finely annotated dataset termed \emph{LiveFood}, including over 5,100 live gourmet videos that consist of four domains: \emph{cooking}, \emph{eating}, \emph{ingredients} and \emph{presentation}. To the best of our knowledge, this is the first work to explore video highlight detection in the incremental learning setting, opening up new land to apply VHD for practical scenarios where both the concerned highlight domains and training data increase over time. We demonstrate the effectiveness of GPE through extensive experiments. Notably, GPE surpasses popular domain-incremental learning methods on \emph{LiveFood}, achieving significant mAP improvements on all domains. The code and dataset will be made publicly available.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2209.05166/code)