Fast Optimizer Benchmark

Simon Blauth, Tobias Bürger, Zacharias Häringer, Jörg K.H. Franke, Frank Hutter

Keywords: benchmark, deep learning, neural network, training algorithms, optimizer

TL;DR: In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development.

Abstract:

In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development.
The benchmark supports tasks from multiple domains such as computer vision, natural language processing, and graph learning.
The focus is on convenient usage, featuring human-readable YAML configurations, SLURM integration, and plotting utilities.
FOB can be used together with existing hyperparameter optimization (HPO) tools as it handles training and resuming of runs.
The modular design enables integration into custom pipelines, using it simply as a collection of tasks.
We showcase an optimizer comparison as a usage example of our tool.
FOB can be found on GitHub: https://github.com/automl/FOB

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.

GPU Hours: 2500

Evaluation Metrics: Yes

Estimated CO2e Footprint: 0.28

Submission Number: 17