Mitigation of Over-Confidence in Scale-Adjusted Training for Early-Exit Networks

Ji-Ye Jeon, Xuan Truong Nguyen, Hyuk-Jae Lee

Abstract: Dynamic networks such as a network with multiple early exits (EEs) is a promising approach for computation reduction by effectively choosing an execution path for a given image during inference. However, an EE network may suffer from significant accuracy degradation by an overconfidence problem in which an EE classifier outputs an excessively high confidence score even with wrong prediction. In this study, we show an overconfidence case when applying scaled-adjusted training (SAT), a well-known low-bit quantization method for an EE network. More precisely, SAT originally scales down a classifier layer's weight during training only, which may result in exaggerated classification logits and overconfidence during inference. To address the problem, we propose a simple yet effective method by applying weight scaling during inference with just a one-line code modification. Experiments with Resnet-18 on the ImageNet dataset demonstrate the effectiveness of our method for EE networks in both a confidence calibration metric and a Top-1 accuracy performance.