Exploring Performance Degradation in Dense Tasks for Self-supervised Learning

Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Qingming Huang

Abstract:

In this work, we observe a counterintuitive phenomenon in self-supervised learning (SSL): longer training may impair the performance of dense downstream tasks (e.g., semantic segmentation). We refer to this phenomenon as Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across ten state-of-the-art SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, measuring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge.
To tackle this issue, we introduce a Dense representation Quality Estimator (DQE), composed of a class-relevance measure and an effective dimensionality measure. The proposed DQE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and a DQE-based regularization method. Experiments on ten SSL methods across four benchmarks confirm that model selection improves mIoU by $4.0\\%$ on average with negligible computational cost. Additionally, DQE regularization consistently mitigates the effects of dense degradation. Code is provided in the supplementary material.