Dynamic Activations for Neural Net Training

Published: 19 Mar 2024, Last Modified: 14 Aug 2024Tiny Papers @ ICLR 2024 PresentEveryoneRevisionsBibTeXCC BY 4.0
Keywords: Adaptive Activation functions, dynamic activation models, Neural Network optimization, Function Approximation Techniques
TL;DR: Exploring the use of dynamic activation in neural networks, this paper showcases a Neural net algorithms using adaptive activations over traditional functions, offering enhanced convergence and accuracy in complex function approximations.
Abstract: Recent advancements in deep learning have seen breakthroughs in training algorithms, benefiting speech, text, image, and video processing. While deeper architectures like ResNet have made strides, shallow Convolutional Neural Networks (CNNs) remain underexplored. Activation functions, pivotal for introducing non-linearity, drive significant progress. This paper investigates complex piece-wise linear hidden layer activations. Our experiments highlight their superiority over traditional Rectified Linear Units (ReLUs) across architectures. We introduce AdAct, an Adaptive Activation algorithm showing promising performance boosts in diverse CNN and multilayer perceptron setups, advocating for its adoption.
Submission Number: 226
Loading