AdAct: Learning to Optimize Activation Function Choice through Adaptive Activation Modules
Keywords: Learnable activation function, convnet, activation function
TL;DR: AdAct introduces a unified, adaptable activation function framework that aims to enhance neural network performance by adaptively selecting and combining optimal activation functions for different tasks and datasets
Abstract: This paper presents an innovative approach to enhancing neural network performance through the development and implementation of an adaptive activation function, termed adaptive activation (AdAct). AdAct amalgamates various well-established and novel activation functions into a single, learnable framework, allowing dynamic adaptation to specific network layers' needs. We explore the effectiveness of ReLU and its variants, including ELU, LReLU, PReLU, RReLU, and more recent functions like Swish and Mish, integrating them into the AdAct function. Employing ConvNet variants across FMNIST, CIFAR10, SVHN and FER datasets, our study empirically assesses each function's contribution and demonstrates AdAct's potential in optimizing neural networks, especially in selecting optimal activation functions for diverse tasks.
Submission Number: 61
Loading