Neural Networks as Dynamical Systems of Stochastic Fields

Anonymous

Keywords: neural tangent kernel, edge of chaos, dynamical systems, stochastic processes, deep neural networks

TL;DR: First variation of covariance function determines its flow through the layers and it is related to the neural tangent kernel.

Abstract: We develop a framework for over-parametrised neural networks as dynamical systems of stochastic fields. With it, we derive a revised covariance function with squared exponential activations. More importantly, we highlight the first variation covariance function with a given set weight initialisation variances as the determinant of covariance function flow in deep neural networks for wide classes of activation functions. We explain the so-called edge-of-chaos observations and pathological amends in the literature in this framework. Lastly, we derive some conditions on the end-behaviour of activation functions for discrete flow convergence