VISIOCITY: A New Benchmarking Dataset and Evaluation Framework Towards Realistic Video Summarization

Vishal Kaushal, Suraj Nandkishor Kothawade, Rishabh K Iyer, Ganesh Ramakrishnan

Keywords: video summarization, dataset, benchmark, evaluation

TL;DR: We introduce a new dataset and evaluation framework as a step towards addressing some challenges in realistic video summarization

Abstract: Automatic video summarization has attracted a lot of interest, but is still an unsolved problem due to several challenges. The currently available datasets either have very short videos or have a few long videos of only a particular type. We introduce a new benchmarking video dataset called VISIOCITY (VIdeo SummarIzatiOn based on Continuity, Intent and DiversiTY) which consists of longer videos across six different domains with dense concept annotations capable of supporting different flavors of video summarization and other vision problems. Secondly, supervised video summarization techniques require many human reference summaries as ground truth. Acquiring them is not easy, especially for long videos. We propose a strategy to automatically generate multiple reference summaries using the annotations present in VISIOCITY and show that these are at par with the human summaries. The annotations thus serve as indirect ground truth. Thirdly, due to the highly subjective nature of the task, different ideal reference summaries of long videos can be quite different from each other. Due to this, the current practice of evaluating a summary vis-a-vis a limited set of human summaries and over-dependence on a single measure has its shortcomings. Our proposed evaluation framework overcomes these and offers a better quantitative assessment of a summary's quality. Finally, based on the above observations we present insights into how a mixture model can be easily enhanced to yield better summaries and demonstrate the effectiveness of our recipe in doing so as compared to some of the representative state-of-the-art techniques when tested on VISIOCITY. We make VISIOCITY publicly available via our website (https://visiocity.github.io/).

URL: https://visiocity.github.io/