Sparse Gate for Differentiable Architecture Search

Liang Fan, Handing Wang

Abstract: Differentiable architecture search is now one of mainstream methods to design the structure of neural networks. It makes the neural architecture search efficient through parameter sharing and differentiable search. However, there are still many significant challenges for designing the optimal architecture, including the phenomenon of skip connection aggregation, excessive memory usage, and large discretization errors. To address these issues, we propose a novel approach called Gate-DARTS where a sparse gating network is introduced to route each input sample to the best $k$ operators. This can reduce the memory requirements of the algorithm and break the residual structure. For the issue of discretization error, we then propose a auxiliary loss to enlarge the difference between different operators. We conduct comprehensive experiments on DARTS-like search space, and Gate-DARTS achieves 97.45% test accuracy on CIFAR10 with 0.23 GPU-days, 83.82% on CIFAR100 with 0.28 GPU-days. Our code has been made available at https://github.com/HandingWangXDGroup/Gate-DARTS.