Mode NormalizationDownload PDF

27 Sept 2018, 22:39 (edited 10 Feb 2022)ICLR 2019 Conference Blind SubmissionReaders: Everyone
  • Keywords: Deep Learning, Expert Models, Normalization, Computer Vision
  • TL;DR: We present a novel normalization method for deep neural networks that is robust to multi-modalities in intermediate feature distributions.
  • Abstract: Normalization methods are a central building block in the deep learning toolbox. They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules. When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent normalization method, is reduced. As a remedy, we propose a more flexible approach: by extending the normalization to more than a single mean and variance, we detect modes of data on-the-fly, jointly normalizing samples that share common features. We demonstrate that our method outperforms BN and other widely used normalization techniques in several experiments, including single and multi-task datasets.
  • Code: [![Papers with Code](/images/pwc_icon.svg) 2 community implementations](https://paperswithcode.com/paper/?openreview=HyN-M2Rctm)
  • Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [CIFAR-100](https://paperswithcode.com/dataset/cifar-100)
15 Replies

Loading