Logit As Auxiliary Weak-supervision for More Reliable and Accurate Prediction

Duhyeon Bang, Yunho Jeon, Jin-Hwa Kim, Jiwon Kim, Hyunjung Shim

Keywords: Inter-class correlation, Human cognitive system, Weak supervision, Calibration, Regularization

Abstract: When a person identifies objects, he or she can think by associating objects to many classes and conclude by taking inter-class relations into account. This cognitive system can make a more reliable prediction. Inspired by these observations, we propose a new network training strategy to consider inter-class relations, namely LogitMix. Specifically, we use recent data augmentation techniques (e.g., Mixup, Manifold Mixup, or Cutmix) as baselines for generating mixed samples. Then, LogitMix suggests using the mixed logit (ie., the mixture of two logits) as an auxiliary training objective. Because using logit before softmax activation preserves rich class relationships, it can serve as a weak-supervision signal concerning inter-class relations. Our experimental results demonstrate that LogitMix achieves state-of-the-art performance among recent data augmentation techniques in terms of both calibration error and prediction accuracy. The source code is attached as the supplementary material.

One-sentence Summary: We propose a new network training strategy to consider inter-class relations by utilizing Logit as an auxiliary weak-supervision.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=2CYfZH_BN