Abstract: Neural architecture search (NAS) has been studied extensively and has grown to become a research field with substantial impact. While classical single-objective NAS searches for the architecture with the best performance, multi-objective NAS considers multiple objectives that should be optimized simultaneously, e.g., minimizing resource usage along the validation error. Although considerable progress has been made in the field of multi-objective NAS, we argue that there is some discrepancy between the actual optimization problem of practical interest and the optimization problem that multi-objective NAS tries to solve. We resolve this discrepancy by formulating the multi-objective NAS problem as a quality diversity optimization (QDO) problem and introduce three quality diversity NAS optimizers (two of them belonging to the group of multifidelity optimizers), which search for high-performing yet diverse architectures that are optimal for application-specific niches, e.g., hardware constraints. By comparing these optimizers to their multi-objective counterparts, we demonstrate that quality diversity NAS in general outperforms multi-objective NAS with respect to quality of solutions and efficiency. We further show how applications and future NAS research can thrive on QDO.
Keywords: NAS, QDO, multifidelity, multi-objective, resource usage, efficieny, BO
One-sentence Summary: We formulate the multi-objective NAS problem as a quality diversity optimization problem and propose three quality diversity NAS optimizers that outperform their multi-objective counterparts.
Track: Main track
Reproducibility Checklist: Yes
Broader Impact Statement: Yes
Paper Availability And License: Yes
Code Of Conduct: Yes
Reviewers: Florian Pfisterer is already reviewer. Bernd Bischl is already senior area chair.
Main Paper And Supplementary Material: pdf
Steps For Environmental Footprint Reduction During Development: We used tabular and surrogate benchmarks.
CPU Hours: 939
GPU Hours: 72
TPU Hours: 0
Estimated CO2e Footprint: 72.30