Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variation
Abstract: Neural Architecture Search (NAS) is a powerful automatic alternative to manual design of a neural network. In the zero-shot version, we use fast ranking functions to compare architectures without training them. The outputs of the ranking functions often vary significantly due to different sources of randomness, including the evaluated architecture's weights' initialization or the batch of data used for calculations. A common approach to addressing the variation is to average a ranking function output over several evaluations. We propose taking into account the variation in a different manner, by viewing the ranking function output as a random variable representing a proxy performance metric. During the search process, we strive to construct a stochastic ordering of the performance metrics to determine the best architecture. Our experiments show that the proposed stochastic ordering can effectively boost performance of a search on standard benchmark search spaces.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Xingchen_Wan1
Submission Number: 3726
Loading