When Does Self-supervision Improve Few-shot Learning? - A Reproducibility Report

Arjun Ashok, Haswanth Aekula

Keywords: few shot learning

Abstract:

Scope of Reproducibility

The paper investigates applying self-supervised learning (SSL) as a regularizer to meta-learning based few-shot learners. The authors claim that SSL tasks reduce the relative error of few-shot learners by 4% - 27% even when the datasets are small, and the improvements are greater when the amount of supervision is lesser or the task is more challenging. Further, they observe that incorporating unlabelled images from other domains for SSL can hurt the performance, and propose a simple algorithm to select images for SSL from other domains to provide further improvements.

Methodology

We reimplement the algorithms in PyTorch, starting with the author's codebase as a reference. We had to correct several bugs in the author's codebase, and reimplement the domain selection algorithm from scratch since the codebase did not contain it. We conduct experiments involving combinations of supervised and self-supervised learning on multiple datasets, on 2 different architectures and perform extensive hyperparameter sweeps to test the claim. We used 4 GTX 1080Ti GPUs throughout, and all our experiments including the sweeps took a total compute time of 980 GPU hours.

Results

On the ResNet-18 architecture and an image size of 224 that the paper uses throughout, our results on 6 datasets overall verify the claim that SSL regularizes few-shot learners and provide higher gains with difficult tasks. Further, our results also verify that out-of-distribution images for SSL hurt the accuracy, and the domain selection algorithm that we implement from scratch also verifies the paper's claim that the algorithm can choose images from a large pool of unlabelled images from other domains, and improve the performance.

Going beyond the original paper, we also conduct SSL experiments on 5 datasets with the Conv-4-64 architecture with an image size of 84, and find that self-supervision does not help boost the accuracy of few-shot learners in this setup. Further, we also show results on a practical real-world benchmark on cross-domain few-shot learning, and show that using self-supervision when training the base models degrades performance when evaluated on these tasks.

What was easy

The paper was well written and easy to follow and provided a clear description of the experiment. The author's code implementations were relatively easy to understand and mostly reflected the experiments described in the paper.

What was difficult

Since the codebase was not fully complete, it took us a lot of time to identify and solve bugs, and reimplement the algorithms not present in the code. Further, multiple datasets needed a lot of preprocessing to be used. The number of hyperparameters being too many but each proving to be important, and evaluating all the claims of the paper on 5 datasets and 2 architectures was difficult to the number of experiment configurations, resulting in a very high computational cost of 980 GPU hours.

Communication with original authors

We maintained contact with the authors throughout the challenge to clarify several implementation details and questions regarding the domain selection algorithm. The authors were responsive and replied promptly with detailed explanations.

Paper Url: https://openreview.net/forum?id=738x44N7yUE&noteId=46xr8OHtA-