Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks
Keywords: theory, neural network theory, structural redundancy, functional equivalence, functional equivalence class, partial identifiability, parameter canonicalisation, parameter space, piecewise-linear, connectivity
TL;DR: By accounting for various redundant arrangements of units, we characterise all different neural net parameters that can implement any given function, and we show piecewise-linear connectivity properties of these sets (architecture: shallow tanh nets)
Abstract: Understanding the learning process of artificial neural networks requires clarifying the structure of the parameter space within which learning takes place. A neural network parameter's functional equivalence class is the set of parameters implementing the same input--output function. For many architectures, almost all parameters have a simple and well-documented functional equivalence class. However, there is also a vanishing minority of reducible parameters, with richer functional equivalence classes caused by redundancies among the network's units.
In this paper, we give an algorithmic characterisation of unit redundancies and reducible functional equivalence classes for a single-hidden-layer hyperbolic tangent architecture. We show that such functional equivalence classes are piecewise-linear path-connected sets, and that for parameters with a majority of redundant units, the sets have a diameter of at most 7 linear segments.
Submission Number: 5448
Loading