Self-supervised Learning of Contextualized Local Visual Embeddings

Thalles Silva, Helio Pedrini, Adín Ramírez Rivera

Keywords: representation learning, unsupervised learning, self-supervised learning, computer vision, object detection, segmentation

TL;DR: A convolutional-based self-supervised representation learning model for dense prediction tasks.

Abstract:

We present Contextualized Local Visual Embeddings (CLoVE), a self-supervised convolutional-based method that learns representations suited for dense prediction tasks.
CLoVE deviates from current methods and optimizes a single loss function that operates at the level of contextualized local embeddings learned from output feature maps of CNN encoders.
To learn contextualized embeddings, CLoVE proposes a normalized mult-head self-attention layer that combines local features from different parts of an image based on similarity.
We extensively benchmark CLoVE's pre-trained representations on multiple datasets.
CLoVE reaches state-of-the-art performance for CNN-based architectures in 4 dense prediction downstream tasks, including object detection, instance segmentation, keypoint detection, and dense pose estimation.

Submission Number: 12