Dynamic Activations for Neural Net Training

Chinmay Rane, Kanishka Tyagi, Tushar Chugh, NIRMALA MURALI

Keywords: Adaptive Activation functions, dynamic activation models, Neural Network optimization, Function Approximation Techniques

TL;DR: Exploring the use of dynamic activation in neural networks, this paper showcases a Neural net algorithms using adaptive activations over traditional functions, offering enhanced convergence and accuracy in complex function approximations.

Abstract: Recent advancements in deep learning have seen breakthroughs in training algorithms, benefiting speech, text, image, and video processing. While deeper architectures like ResNet have made strides, shallow Convolutional Neural Networks (CNNs) remain underexplored. Activation functions, pivotal for introducing non-linearity, drive significant progress. This paper investigates complex piece-wise linear hidden layer activations. Our experiments highlight their superiority over traditional Rectified Linear Units (ReLUs) across architectures. We introduce AdAct, an Adaptive Activation algorithm showing promising performance boosts in diverse CNN and multilayer perceptron setups, advocating for its adoption.

Submission Number: 226