Neural networks with trainable matrix activation functions

Yuwen Li, Zhengqi Liu, Ludmil Zikatanov

Keywords: neural networks, trainable activation function, function approximation, image classification

Abstract: The training process of neural networks usually optimize weights and bias parameters of linear transformations, while nonlinear activation functions are pre-specified and fixed. This work develops a systematic approach to constructing matrix activation functions whose entries are generalized from ReLU. The activation is based on matrix-vector multiplications using only scalar multiplications and comparisons. The proposed activation functions depend on parameters that are trained along with the weights and bias vectors. Neural networks based on this approach are simple and efficient and are shown to be robust in numerical experiments.

One-sentence Summary: We propose novel, trainable matrix activation functions (TMAF) which are competitive and robust.