Revisiting Internal Covariant Shift for Batch Normalization

Muhammad Awais, Md Tauhid Bin Iqbal, Sung-Ho Bae

Abstract: Despite the success of Batch Normalization (BatchNorm) and a plethora of its variants, the exact reasons for its success are still shady. The original BatchNorm paper explained it as a mechanism that reduces the Internal Covariate Shift (ICS), i.e., the distribution shifts in the input of the layers during training. Recently, some papers manifested skepticism on this hypothesis and provided \textit{alternative explanations} for the success of BatchNorm, such as the applicability of very high learning rates and the ability to smooth the landscape in optimization. In this work, we counter this \ textit {alternative arguments} by demonstrating the importance of reduction in ICS following an empirical approach. We demonstrated various ways to achieve the above-mentioned alternative properties without any performance boost. In this light, we explored the importance of different BatchNorm parameters (i.e., batch statistics, affine transformation parameters) by visualizing their effectiveness in the performance and analyzed their connections with ICS. Afterward, we showed a different normalization scheme that fulfills all the alternative explanations except reduction in ICS. Despite having all the alternative properties, we observed its poor performance, which nullifies the alternative claims rather signifies the importance of the ICS reduction. We performed comprehensive experiments on many variants of BatchNorm, finding that all of them similarly reduce ICS.