Towards Distribution-Aware Active Learning for Data-Efficient Neural Architecture Predictor

Caiyang Yu, Yifan Wang, Chenwei Tang, Xianggen Liu, Jiancheng Lv

Keywords: neural architecture predictor, active learning, neural architecture search

Abstract: As the neural predictor (NP) provides a fast evaluation for neural architectures, it is highly sought after in neural architecture search (NAS). However, the high computational cost involved in generating training data results in its scarcity, which in turn limits the accuracy of the NP. Active learning (AL) has the potential to address this issue by prioritizing the most informative samples, yet existing methods struggle with selection bias when faced with imbalanced data distributions, often prioritizing diversity over representativeness. In this paper, we redefine the sample selection mechanism in AL and propose a Distribution-aware Active Learning framework for Neural Predictor (called DARE). The goal is to select samples that not only ensure diversity but also exhibit a high degree of generalizability, making them more representative of the underlying data distribution. Our approach first extracts architecture representations via a graph-based encoder enhanced with a consistency-driven objective. Then, a two-stage selection strategy identifies both globally diverse and locally reliable samples through progressive representation learning and refinement. For non-uniform data distributions, we further introduce an adaptive mechanism that anchors sampling to key regions with high similarity density, avoiding performance degradation caused by outliers. Extensive experiments have shown that the proposed distribution-aware active learning strategy samples a higher-quality training dataset for NPs, allowing the neural architecture predictor to achieve state-of-the-art results.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 24754