On the Universal Approximation Properties of Deep Neural Networks using MAM Neurons

Philippe Bich, Andriy Enttsel, Luciano Prono, Alex Marchioni, Fabio Pareschi, Mauro Mangia, Gianluca Setti, Riccardo Rovatti

Keywords: universal approximation theory, multiply-and-max/min neurons

TL;DR: The paper is a first step in the theoretical characterization of the capabilities of hybrid neural networks containing both MAC and Multiply-And-Max/min neurons

Abstract:

As Deep Neural Networks (DNNs) are trained to perform tasks of increasing complexity, their size grows, presenting several challenges when it comes to deploying them on edge devices that have limited resources. To cope with this, a recently proposed approach hinges on substituting the classical Multiply-and-Accumulate (MAC) neurons in the hidden layers of a DNN with other neurons called Multiply-And-Max/min (MAM) whose selective behaviour helps identifying important interconnections and allows extremely aggressive pruning. Hybrid structures with MAC and MAM neurons promise a reduction in the number of interconnections that outperforms what can be achieved with MAC-only structures by more than an order of magnitude. However, by now, the lack of any theoretical demonstration of their ability to work as universal approximators limits their diffusion.
Here, we take a first step in the theoretical characterization of the capabilities of MAM\&MAC networks. In details, we prove two theorems that confirm that they are universal approximators providing that two hidden MAM layers are followed either by a MAC neuron without nonlinearity or by a normalized variant of the same. Approximation quality is measured either in terms of the first-order $L^p$ Sobolev norm or by the $L^\infty$ norm.

Submission Number: 13417