BOSE-NAS: Differentiable Neural Architecture Search with Bi-Level Optimization Stable Equilibrium

Weisheng Xie, Xiangxiang Gao, Xuwei Fang, chen hang, Hui Li, Shaoyuan Li

Keywords: Neural Architecture Search, Stable Equilibrium State, Equilibrium Influential

TL;DR: This paper clarifies the ambiguities surrounding the actual role and impact of architecture parameters in DARTS and leveraging this insight proposes a more effective and robust NAS method.

Abstract: Recent research has significantly mitigated the performance collapse issue in Differentiable Architecture Search (DARTS) by either refining architecture parameters to better reflect the true strengths of operations or developing alternative metrics for evaluating operation significance. However, the actual role and impact of architecture parameters remain insufficiently explored, creating critical ambiguities in the search process. To address this gap, we conduct a rigorous theoretical analysis demonstrating that the change rate of architecture parameters reflects the sensitivity of the supernet’s validation loss in architecture space, thereby influencing the derived architecture's performance by shaping supernet training dynamics. Building on these insights, we introduce the concept of a Stable Equilibrium State to capture the stability of the bi-level optimization process and propose the Equilibrium Influential ($E_\mathcal{I}$) metric to assess operation importance. By integrating these elements, we propose BOSE-NAS, a differentiable NAS approach that leverages the Stable Equilibrium State to identify the optimal state during the search process and derives the final architecture using the $E_\mathcal{I}$ metric. Extensive experiments across diverse datasets and search spaces demonstrate that BOSE-NAS achieves competitive test accuracy compared to state-of-the-art methods while significantly reducing search costs.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6644