Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning

Huaiyuan Qin, Muli Yang, Siyuan Hu, Peng Hu, Yu Zhang, Chen Gong, Hongyuan Zhu

Abstract: Self-supervised learning (SSL) conventionally relies on the instance consistency paradigm, assuming that different views of the same image can be treated as positive pairs. However, this assumption breaks down for non-iconic data, where different views may contain distinct objects or semantic information. In this paper, we investigate the effectiveness of SSL when instance consistency is not guaranteed. Through extensive ablation studies, we demonstrate that SSL can still learn meaningful representations even when positive pairs lack strict instance consistency. Furthermore, our analysis further reveals that increasing view diversity, by enforcing zero overlapping or using smaller crop scales, can enhance downstream performance on classification and dense prediction tasks. However, excessive diversity is found to reduce effectiveness, suggesting an optimal range for view diversity. To quantify this, we adopt the Earth Mover’s Distance (EMD) as an estimator to measure mutual information between views, finding that moderate EMD values correlate with improved SSL learning, providing insights for future SSL framework design. We validate our findings across a range of settings, highlighting their robustness and applicability on diverse data sources.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We sincerely thank the reviewers and the Action Editor for their constructive feedback and positive evaluation of our work.
This feedback has led to notable improvements and helped strengthen the clarity, completeness, and broader applicability of our study.
As mentioned in our rebuttal responses, we have incorporated all remaining points in the *final camera-ready revision*.

Below is the summary of the updates:

- **Experimental Clarifications**
- **Added clearer clarification** of experimental setups and result interpretations in Sections 4.1 & 4.2.
- **Highlighted the evaluation metrics** used for downstream tasks in Appendix Section A.2.

- **Additional Experimental Results**
- **Expanded ablation studies on pseudo mask generators** in Appendix Section B.4.
- **Added validation on medical imaging domain** in Appendix Section B.6.
- **Added a time cost analysis** for EMD computation in Appendix Section B.7.

- **Additional Discussions**
- **Added a remark of the controlled experiments** to emphasize the intended research scope in Appendix 4.1.
- **Added discussion on practical strategy** for positive pair selection in SSL in Section 4.3.
- **Updated Figure 3 caption** to clarify the interpolation of line width.

- **Literature Review**
- **Expanded literature review** on foundational SSL methods in Section 2.

- **Broader Impact**
- **Added a broader impact** discussion in Conclusion.

- **Other Improvements**
- **Conducted minor language edits** for better clarity and readability.

Besides, we are currently seeking internal approval for code release.
We will try our best to release our implementation of the designed experiments.

Assigned Action Editor: ~Georgios_Leontidis1

Submission Number: 5094