On the Expected Complexity of Maxout Networks

Hanna Tseran, Guido Montufar

Keywords: linear regions of neural networks, maxout units, expected complexity, decision boundary, parameter initialisation

TL;DR: The expected number of linear regions and pieces of the decision boundary of maxout networks is polynomial in the number of units.

Abstract: Learning with neural networks relies on the complexity of their representable functions, but more importantly, their particular assignment of typical parameters to functions of different complexity. Taking the number of activation regions as a complexity measure, recent works have shown that the practical complexity of deep ReLU networks is often far from the theoretical maximum. In this work, we show that this phenomenon also occurs in networks with maxout (multi-argument) activation functions and when considering the decision boundaries in classification tasks. We also show that the parameter space has a multitude of full-dimensional regions with widely different complexity, and obtain nontrivial lower bounds on the expected complexity. Finally, we investigate different parameter initialization procedures and show that they can increase the speed of convergence in training.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/hanna-tseran/maxout_complexity

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2107.00379/code)