Fast Temporal Information Retrieval In Videos With Visual Memory

Jungkyoo Shin, Jinyoung Moon

Abstract: Due to recent increases in video usage, there have been many studies about processing and managing information within huge volumes of videos. Existing methods for video retrieval aim to retrieve only similar frames related to a query image and compare all frames to the query image, which is costly in run-time and memory usage. To resolve these limitations, we propose a fast retrieval method for precise temporal information with visual memory. Our model compresses an input video into a compressed visual memory and applies an attention-based layer to obtain the probability of a given query image's existence. To the best of our knowledge, we are the first to attempt video retrieval for temporal information using visual memory. To show the efficiency and effectiveness of our model, we conducted experiments for temporal information retrieval on 60-second videos from TV shows and dramas. Our model could effectively compress a video to visual memory with space-savings of 93.6% and 99.1% compared to frame features and original video, respectively. Using the compressed visual memory, our method retrieved temporal information at 250K fps, which is 28x and 4,164x faster than retrieval methods using frame features and frames, respectively.