Learning Polynomial Activation Functions for Deep Neural Networks

Linghao Zhang, Jiawang Nie, Tingting Tang

Track: Full Paper (8 pages)

Keywords: Activation function, polynomial, moment, relaxation, optimization

TL;DR: We give a Moment-SOS relaxation method to learn polynomial activation functions in deep neural networks.

Abstract: Activation functions are crucial for deep neural networks. This novel work frames the problem of training neural network with learnable polynomial activation functions as a polynomial optimization problem, which is solvable by the Moment-SOS hierarchy. This work represents a fundamental departure from the conventional paradigm of training deep neural networks, which relies on local optimization methods like backpropagation and gradient descent. Numerical experiments are presented to demonstrate the accuracy and robustness of optimum parameter recovery in presence of noises.

Submission Number: 7