What can we learn from Harry Potter? An Exploratory Study of Visual Representation Learning from Atypical Videos

Qiyue Sun, Qiming Huang, Yang Yang, Hongjun Wang, Jianbo Jiao

Keywords: Open-world learning, Out-of-distribution detection, Video classification

Abstract:

Humans usually show exceptional generalisation and discovery ability in the open world, when being shown uncommonly new concepts. Whereas most existing studies in the literature focus on common typical data from closed sets, and open world novel discovery is under-explored in videos. In this paper, we are interested in asking: \textit{what if atypical unusual videos are exposed in the learning process?} To this end, we collect a new video dataset consisting of various types of unusual atypical data (e.g. sci-fi, animation, etc.). To study how such atypical data may benefit representation learning in open-world discovery, we feed them into the model training process for representation learning. Taking out-of-distribution (OOD) detection as a task to evaluate the model's novel discovery capability, we found that such a simple learning approach consistently improves performance across a few different settings. Furthermore, we found that increasing the categorical diversity of the atypical samples further boosts OOD detection performance. These observations in our extensive experimental evaluations reveal the benefits of atypical videos for visual representation learning in the open world, together with the newly proposed dataset, encouraging further studies in this direction.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11238