Zero-Cost Proxy-Based Hierarchical Initialization for Evolutionary Neural Architecture Search

Nhat Minh Le, An Vo, Ngoc Hoang Luong

Abstract: Neural Architecture Search (NAS) aims to automate the process of architecture design to alleviate the burden of manual trial-and-error when seeking top-performing neural networks for certain tasks. Traditional NAS methods perform network evaluations with full network training, typically incurring hundreds of training epochs for each candidate architecture. Such expensive search costs are infeasible in real-world applications. Several zero-cost metrics, which are computed based on randomly initialized network weights, have been proposed for efficiently estimating architecture quality as a proxy for network accuracy performance. These metrics require much less computation time but may not exhibit sufficient correlation with the actual accuracy. Using a single proxy metric might mislead the search, especially in the region of good-performing architectures in the search space. In this paper, we propose a two-phase Evolutionary Neural Architecture Search with Zero-Cost Proxy-Based Hierarchical Initialization (eNAS-HI) framework. A tree search algorithm, guided by three training-free performance proxy metrics, is used to prune the search space and efficiently identify a diverse set of promising architectures in the initialization phase. These architectures then serve as the initial population for a genetic algorithm in the subsequent evolution phase. The diversity and quality of the initial population help eNAS-HI in finding top-performing architectures with reduced search costs.