Q-Match: Self-Supervised Learning For Tabular Data by Matching Distributions Induced by a Queue

Thomas Mulc, Debidatta Dwibedi

Keywords: self-supervised learning, deep learning for tabular data

Abstract: In semi-supervised learning, student-teacher distribution matching has been successful in improving performance of models using unlabeled data in conjunction with few labeled samples. In this paper, we aim to replicate that success in the self-supervised setup where we do not have access to any labeled data during pre-training. We show it is possible to induce the student-teacher distributions without any knowledge of downstream classes by using a queue of embeddings of samples from the unlabeled dataset. We show that Q-Match outperforms previous self-supervised learning techniques on tabular datasets when measuring downstream classification performance. Furthermore, we show that our method is sample efficient, both in terms of labels required for both downstream task training and amount of unlabeled data required for pre-training.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

TL;DR: A self-supervised method to train models by minimizing the cross-entropy loss between student-teacher distributions generated using a queue of embeddings. This results in better downstream task performance with less labeled data.

Please Choose The Closest Area That Your Submission Falls Into: Unsupervised and Self-supervised learning