Aligning the Number of Parameters with the Number of Linear Regions for Improved Neural Network Approximation

Podoprosvetov Aleksei, Vladimir Smolin, Sergei Sokolov

Keywords: Deep learning, neural network approximation, latent space topology, piecewise-linear regions, universal approximation theorem, signal decomposition.

TL;DR: We formalize parameter–region mismatch in ReLU nets and propose architectural constraints that control linear-region growth and improve compact function approximation.

Abstract: The paper addresses the "black box" problem of neural networks by analyzing the approximation properties of latent layers. It proposes that a key limitation preventing the practical achievement of universal approximation theorems is the mismatch between the growth rates of a network's parameters and the number of linear regions partitioning the input space. The question is examined how this imbalance is exacerbated in multidimensional cases, hindering effective learning. To resolve this, methods are suggested to align parameter counts with the number of linear regions, such as moving activations vectors to the surface of a hypercube, utilizing micro-columns, and leveraging the "blessing of dimensionality" in deep networks to decouple complex signals.

Submission Number: 87