L2 Norm Guided Adaptive Computation

Mani Shemiranifar, Mostafa Dehghani

Keywords: Machine learning, Adaptive Computation

TL;DR: We propose a new idea for introducing adaptivity in the computational budget for neural networks dedicated to processing different examples.

Abstract: Although the human brain can adjust the amount of time and energy it uses to solve problems of varying complexity, many standard neural networks require a fixed computation budget regardless of the problem’s complexity. This work introduces L2 Adaptive Computation (LAC), a new algorithm that adjusts the computation budget, by tracking changes in the L2 norm of a neural network’s hidden state as layers are applied to the input. Unlike previous methods, LAC does not require additional trainable modules or auxiliary loss terms to make halting decisions. LAC matches the results of best-performing methods on a complex synthetic task and improves image classification accuracy while also increasing efficiency.