Deep MinMax Networks

Winfried Lohmiller, Philipp Gassert, Jean-Jacques E. Slotine

Abstract: While much progress has been achieved over the last decades in neuro-inspired machine learning, there are still fundamental problems in gradient-based learning using combinations of neurons, especially away from the infinite width limit.These problems, such as saddle points and suboptimal plateaus of the cost function, can lead in theory and practice to failures of learning. In addition, because of their global geometry, Rectified Linear Units (ReLUs) do not allow local adjustment of a pre-learned global network, making the incorporation of new data inefficient.This paper describes an alternative local learning approach for arbitrary piece-wise linear continuous functions. Neurons are not superimposed, but rather are spatially separated. Also, contrary to classical approximations based e.g. on radial basis functions or finite elements, there is no pre-fixed set of basis functions, but location and support are all adaptable.The approach yields a new multi-layer algorithm based on learning a MinMax combination of linear neurons, which is applicable directly to the high-dimensional case. Global exponential convergence of the algorithm is established using Lyapunov analysis and contraction theory. It is shown that the algorithm corresponds to a set of separate exponentially stable linear regressions, whose combination is Lyapunov or contraction stable under observability conditions.Learning is fundamentally local, which allows simple progressive adjustment of a previously learned network as new data becomes available. Unused neurons are pruned, avoiding unnecessary over-parameterization. Conversely, new neurons are created in areas where better approximation of the true function is needed. Overall stability is shown for this combination of pruning and creation mechanisms.