NOSBench-101: Towards Reproducible Neural Optimizer Search
Keywords: automl, nos, benchmark
TL;DR: NOSBench is a standardized and compute-efficient benchmark for evaluating Neural Optimizer Search methods.
Abstract: Recent advances in neural network architecture and hardware have revolutionized deep learning and made it a pervasive technology. Nonetheless, it is crucial to acknowledge that the achievement of training neural networks with millions and billions of parameters would not have been feasible without the advancement of effective optimization techniques. This has motivated the search for new efficient optimization algorithms that can improve the performance of deep learning networks even more. Despite the considerable manual (re)search effort, few of these methods have found their way into deep learning practice. Recently, various researchers have explored different search methods to learn/discover novel optimizers in an automated way, but the associated computational costs and lack of a standardized evaluation protocol have hindered progress in this field. Motivated by the success of Neural Architecture Search (NAS), which benefits from established and compute-efficient benchmarks like NASBench, we introduce a benchmark called NOSBench that can be used to test different Neural Optimizer Search (NOS) methods on the same tasks. We compare different NOS methods on a Prior-Data Fitted Networks (PFNs) meta-training task and show that the optimizer found transfer to other PFN training tasks (e.g., TabPFN, LC-PFN, PFNs4BO). Our experiments show that the NOSBench provides a useful way to compare and contrast different approaches in this field efficiently by caching and identifying identical optimizers, which we believe can help researchers identify promising search strategies as they search for new optimizers automatically, thereby bringing NOS into the mainstream.
Submission Checklist: Yes
Broader Impact Statement: Yes
Paper Availability And License: Yes
Code Of Conduct: Yes
Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.
Steps For Environmental Footprint Reduction During Development: One of the main contribution of our benchmark is the caching mechanism which helps us to not run unnecessary computations thus reducing the environmental footprint. We kept a common cache between our experiments.
CPU Hours: 5
GPU Hours: 331
TPU Hours: 0
Evaluation Metrics: No
Estimated CO2e Footprint: 61.49
Submission Number: 10
Loading