Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Data

Yash Kumar Sharma, Vineet Padmanabhan, Wilson Naik Bhukya

Keywords: Self Supervised Learning, Contrastive Learning, Imbalanced Data, Multi-view Learning

TL;DR: Self Supervised Learning for Imbalanced Dataset

Abstract:

Contrastive self supervised learning(CSSL) usually makes use of the "multiview" assumption which states that all relevant information must be shared between all views. The main objective of CSSL is to maximize the mutual
information(MI) between representations of different views and at the same time compress irrelevant information in each representation. Recently, as part of future work, Schwartz Ziv \& Yan LeCun pointed out that, when the multi-view assumption is violated, one of the most significant challenges in SSL is in identifying new methods to separate relevant from irrelevant information based on alternative assumptions. Taking a cue from this intuition we make the following contributions in this paper: 1) We develop a CSSL framework wherein multiple images and multiple views(MIMV) are considered as input, which is different from the traditional multiview assumption 2) We adopt a novel augmentation strategy that includes both normalized (invertible) and augmented (non-invertible) views so that complete information of one image can be preserved and hard augmentation can be chosen for the other image 3) An Information bottleneck(IB) principle is outlined for MIMV to produce optimal representations 4) We introduce a loss function that helps to learn better representations by filtering out extreme features 5) The robustness of our proposed framework is established by applying it to the imbalanced dataset problem wherein we achieve a new state-of-the-art accuracy (2\% improvement in Cifar10-LT using Resnet-18, 5\% improvement in Cifar100-LT using Resnet-18 and 3\% improvement in Imagenet-LT (1k) using Resnet-50).

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 23913