Crafting Zero-Cost Proxy Metrics for Neural Architecture Search via Symbolic Regression
Keywords: neural architecture search, symbolic regression, zero-cost proxy metrics, genetic programming
Abstract: Using zero-cost (ZC) metrics to estimate network performance instead of relying on expensive training processes has proven both its efficiency and efficacy in Neural Architecture Search (NAS). However, a significant limitation of most ZC proxies is their inconsistency, as reflected by the substantial variation in their performance across different problems. Additionally, the design of current ZC metrics is manual, which is a lengthy trial-and-error process and requires expert knowledge to develop ZC metrics effectively. These challenges raise two questions: (1) Can we automate the design of ZC metrics? and (2) Can we utilize the existing hand-crafted ZC metrics to synthesize a better one? In this study, we propose a framework based on Symbolic Regression to automate the design of ZC metrics. Our framework is not only highly extensible but also capable of quickly producing a ZC metric with a strong positive rank correlation to network performance across multiple problems within just a few minutes. Extensive experiments on 13 problems in NAS-Bench-Suite-Zero, covering various search spaces and tasks, demonstrate the superiority of our automatically designed proxies over hand-crafted ones. By integrating our proxy metrics into an evolutionary algorithm, we could identify a network architecture with comparable performance on the CIFAR-10 dataset within 15 minutes using a single GeForce RTX 3090 GPU.
Supplementary Material: zip
Primary Area: other topics in machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2952
Loading