An architecture entropy regularizer for differentiable neural architecture search

Kun Jing, Luoyu Chen, Jungang Xu

Abstract: Differentiable architecture search (DARTS) is one of the prevailing paradigms of neural architecture search (NAS) due to allowing efficient gradient-based optimization during the search phase. However, its poor stability and generalizability are intolerable. We argue that the crux is the locally optimal architecture parameter caused by a dilemma, which is that the solutions to the Matthew effect and discretization discrepancy are inconsistent. To escape from the dilemma, we propose an architecture entropy to measure the discrepancy of the architecture parameters of different candidate operations and use it as a regularizer to control the learning of architecture parameters. Extensive experiments show that an architecture entropy regularizer with a negative or positive coefficient can effectively solve one side of the contradiction respectively, and the regularizer with a variable coefficient can relieve DARTS from the dilemma. Experimental results demonstrate that our architecture entropy regularizer can significantly improve different differentiable NAS algorithms on different datasets and different search spaces. Furthermore, we also achieve more accurate and more robust results on CIFAR-10 and ImageNet. The code is publicly available at https://github.com/kunjing96/DARTS-AER.