Speeding Up Online Self-Supervised Learning by Exploiting Its Limitations

Sina Mokhtarzadeh Azar, Radu Timofte

Abstract: Online Self-Supervised Learning tackles the problem of learning Self-Supervised representations from a stream of data in an online fashion. This is the more realistic scenario of Self-Supervised Learning where data becomes available continuously and must be used for training straight away. Contrary to regular Self-Supervised Learning methods where they need to go hundreds of times through a dataset to produce suitable representations, Online Self-Supervised Learning has a limited budget for training iterations for the new data points from the stream. Additionally, the training can potentially continue indefinitely without a specific end. We propose a framework for Online Self-supervised Learning with the goal of learning as much as possible from the newly arrived batch of data in a limited amount of training iterations before the next batch becomes available. To achieve this goal we use a cycle of aggressive learning rate increase for every batch of data which is combined with a memory to reduce overfitting on the current batch and forgetting the knowledge gained from previous batches. Additionally, we propose Reducible Anchor Loss Selection (RALS) to intelligently select the most useful samples from the combination of the new batch and samples from the memory. Considering the limitation of a smaller number of iterations over the data, multiple empirical results on CIFAR-100 and ImageNet-100 datasets show the effectiveness of our approach.