OFA³: Automatic selection of the best non-dominated sub-networks for ensembles

Rafael Claro Ito, Emely Pujolli da Silva, Fernando J. Von Zuben

Keywords: Neural Architecture Search, Multi-objective Optimization, Ensemble, Once-for-All, NSGA-II

Abstract: Advancement of Neural Architecture Search (NAS) has the potential to significantly improve the efficiency and performance of machine learning systems, as well as enable the exploration of new architectures and applications across a wide range of fields, including computer vision, natural language processing, speech recognition, robotics, and more. A promising direction for developing more scalable and adaptive neural network architectures is the Once-for-All (OFA), a NAS framework that decouples the training and the search stages, meaning that one super-network is trained once, and then multiple searches can be performed according to different deployment scenarios. More recently, the OFA² strategy improved the search stage of the OFA framework by exploring the multi-objective nature of the problem: a set of non-dominated sub-networks are all obtained at once, with distinct trade-offs involving hardware constraints and accuracy. In this work, we further improve the search stage of the OFA² by fine-tuning the non-dominated solutions. Furthermore, we propose OFA³, building high-performance ensembles by solving the problem of how to automatically select the optimal subset of the already obtained non-dominated sub-networks. Particularly when components of the ensemble can run in parallel, our results dominate any other configuration of the available sub-networks, taking accuracy and latency as the conflicting objectives.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Reviewers: Yes

CPU Hours: 0

GPU Hours: 48

TPU Hours: 0