Maximum and Leaky Maximum Propagation

Wolfgang Fuhl

Abstract:

In this work, we present an alternative to conven-
tional residual connections, which is inspired by maxout nets.
This means that instead of the addition in residual connections,
our approach only propagates the maximum value or, in the
leaky formulation, propagates a percentage of both. In our eval-
uation, we show on different public data sets that the presented
approaches are comparable to the residual connections and have
other interesting properties, such as better generalization with
a constant batch normalization, faster learning, and also the
possibility to generalize without additional activation functions.
In addition, the proposed approaches work very well if ensembles
together with residual networks are formed