Information-Theoretic Local Minima Characterization and Regularization

Zhiwei Jia, Hao Su

Keywords: local minima, generalization, regularization, deep learning theory

Abstract: Recent advances in deep learning theory have evoked the study of generalizability across different local minima of deep neural networks (DNNs). While current work focused on either discovering properties of good local minima or developing regularization techniques to induce good local minima, no approach exists that can tackle both problems. We achieve these two goals successfully in a unified manner. Specifically, based on the Fisher information we propose a metric both strongly indicative of generalizability of local minima and effectively applied as a practical regularizer. We provide theoretical analysis including a generalization bound and empirically demonstrate the success of our approach in both capturing and improving the generalizability of DNNs. Experiments are performed on CIFAR-10 and CIFAR-100 for various network architectures.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1911.08192/code)