Mixup Your Own Latent: Efficient and Robust Self-Supervised Learning on Small Images

Eugene Yang, Hao Chen, Seokho Kang

Abstract: Self-supervised learning has emerged as a powerful technique in computer vision, demonstrating remarkable performance in various downstream tasks by leveraging unlabeled data. Among these methods, contrastive learning has proven particularly promising by effectively learning image representations. However, its high reliance on large computational resources poses significant practical challenges. To address this issue, there is a pressing need to improve efficiency without compromising generalization performance and robustness. In this paper, we propose Mixup Your Own Latent (MYOL), a regularization method to improve the generalization performance and robustness of Bootstrap Your Own Latent (BYOL), particularly for small images under limited computational resources. MYOL achieves this using the Mixup of the representations of two input images as the target representation of the Mixup of those images. Through experiments conducted in a single GPU environment, we demonstrate that MYOL outperforms BYOL and other regularization methods across various downstream tasks on small-image datasets. The high resilience of MYOL to small batch sizes and its robustness to adversarial attacks further highlight its effectiveness in mitigating the limitations of BYOL. The source code is available at unmapped: uri https://github.com/cneyang/MYOL-MixupYourOwnLatent.