Look Globally and Locally: Inter-Intra Contrastive Learning from Unlabeled Videos

David Fan, Deyu Yang, Xinyu Li, Vimal Bhat, Rohith MV

Keywords: Video Representation, Contrastive Learning, Action Recognition

TL;DR: Existing contrastive learning methods sample intra-video positives and we incorporate inter-video nearest-neighbors to improve performance on a variety of downstream video tasks.

Abstract: State-of-the-art video contrastive learning methods spatiotemporally augment two clips from the same video as positives. By only sampling positive clips from the same video, these methods neglect other semantically related videos that can also be useful. To address this limitation, we leverage nearest-neighbor videos from the global space as additional positives, thus improving diversity and introducing a more relaxed notion of similarity that extends beyond video and even class boundaries. Our Inter-Intra Video Contrastive Learning (IIVCL) improves performance and generalization on video classification, detection, and retrieval tasks.