A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs

Koyel Mukherjee, Alind Khare, Yogish Sabharwal, Ashish Verma

TL;DR: We propose an automated, adaptive LR tuning algorithm for training DNNs that works as well or better than SOTA for different model-dataset combinations tried for natural as well as adversarial training, with theoretical convergence analysis.

Abstract: Training neural networks on image datasets generally require extensive experimentation to find the optimal learning rate regime. Especially, for the cases of adversarial training or for training a newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for determining the learning rate trajectory, that works across datasets and models for both natural and adversarial training, without requiring any dataset/model specific tuning. It is a stand-alone, parameterless, adaptive approach with no computational overhead. We theoretically discuss the algorithm's convergence behavior. We empirically validate our algorithm extensively. Our results show that our proposed approach \emph{consistently} achieves top-level accuracy compared to SOTA baselines in the literature in natural training, as well as in adversarial training.

Keywords: adaptive LR tuning algorithm, generalization