Characterising activation functions by their backward dynamics around forward fixed points

Pieter-Jan Hoedt, Sepp Hochreiter, Günter Klambauer

Abstract: The forward dynamics in neural networks for various activation functions has beenstudied extensively in the context of initialisation and normalisation strategies, bymean field theory, edge of chaos theory, and fixed point analysis. However, thestudy of the backward dynamics appears to be largely disconnected to the insightsobtained from the forward analysis. We argue that many of the ideas from theforward analysis could and should be applied to backward dynamics. We show thatthe ideas of mean field theory and fixed point analysis apply to the backward passand allow to characterise activation functions.