Kernel Normalized Convolutional Networks
Abstract: Existing convolutional neural network architectures frequently rely upon batch normalization (BatchNorm) to effectively train the model. BatchNorm, however, performs poorly with small batch sizes, and is inapplicable to differential privacy. To address these limitations, we propose kernel normalization and kernel normalized convolutional layers, and incorporate them into kernel normalized convolutional networks (KNConvNets) as the main building blocks. We implement KNConvNets corresponding to the state-of-the-art ResNets while forgoing BatchNorm layers. Through extensive experiments, we illustrate KNConvNets achieve higher or competitive performance compared to the BatchNorm counterparts in image classification and semantic segmentation. They also significantly outperform their batch-independent competitors including layer and group normalization in non-private and differentially private training. Given that, KNConvNets combine the batch-independence property of layer and group normalization with the performance advantage of BatchNorm.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: We thank all reviewers for their helpful comments. Here are the (main) modifications in the revised draft compared to the initial draft:
1. ResNet-50 on ImageNet case study in the same experimental setting using cosine-annealing scheduler for all normalization methods.
2. Ablation study for KernelNorm
3. Suggestions on how to reduce the computational overhead of KernelNorm and KNResNets
4. Comparison with Convolutional Normalization
5. Improving the draft by incorporating the comments from the reviewers
Assigned Action Editor: ~Yunhe_Wang1
Submission Number: 1136
Loading