Cross-Model Operator Batching for Neural Network Architecture Search

Lingling Ye, Chi Zhang, Mingxia Li, Zhenhua Han, Haisheng Tan

Abstract: Recently, automated machine learning (AutoML) and neural architecture search (NAS), regarded as promising techniques to design deep learning (DL) models automatically, have received increasing attention from both industry and academia. NAS will generate a large number of candidate models, which typically consist of numerous common substructures, providing a vast opportunity for cross-model optimization (e.g., operator batching) to improve training efficiency. However, most of the existing AutoML frameworks do not make use of operator batching and we also lack an efficient batching strategy. In this work, we propose a heuristic scheme named DPBat to guide the operator batching among multiple models in NAS. For most models, the operator batching of DPBat can be finished in just a few seconds, which is negligible compared to the subsequent training. We adopt Microsoft’s open source AutoML framework NNI to implement DPBat to real NAS scenarios. Extensive experiments show that DPBat is highly effective in improving training efficiency and reducing the overhead of operator batching, with a throughput 3.7\(\times \) higher than the standard practice of running each job without batching.