DARTS-: Robustly Stepping out of Performance Collapse Without IndicatorsDownload PDF

28 Sept 2020, 15:46 (edited 10 Feb 2022)ICLR 2021 PosterReaders: Everyone
  • Keywords: neural architecture search, DARTS stability
  • Abstract: Despite the fast development of differentiable architecture search (DARTS), it suffers from a standing instability issue regarding searching performance, which extremely limits its application. Existing robustifying methods draw clues from the outcome instead of finding out the causing factor. Various indicators such as Hessian eigenvalues are proposed as a signal of performance collapse, and the searching should be stopped once an indicator reaches a preset threshold. However, these methods tend to easily reject good architectures if thresholds are inappropriately set, let alone the searching is intrinsically noisy. In this paper, we undertake a more subtle and direct approach to resolve the collapse. We first demonstrate that skip connections with a learnable architectural coefficient can easily recover from a disadvantageous state and become dominant. We conjecture that skip connections profit too much from this privilege, hence causing the collapse for the derived model. Therefore, we propose to factor out this benefit with an auxiliary skip connection, ensuring a fairer competition for all operations. Extensive experiments on various datasets verify that our approach can substantially improve the robustness of DARTS. Our code is available at https://github.com/Meituan-AutoML/DARTS-
  • One-sentence Summary: Indicator-free approach to stabilize DARTS
  • Supplementary Material: zip
  • Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
  • Code: [![github](/images/github_icon.svg) Meituan-AutoML/DARTS-](https://github.com/Meituan-AutoML/DARTS-)
  • Data: [ImageNet](https://paperswithcode.com/dataset/imagenet), [NAS-Bench-201](https://paperswithcode.com/dataset/nas-bench-201)
11 Replies