Light and Accurate: Neural Architecture Search via Two Constant Shared Weights Initialisations
Keywords: neural architecture search, zero-cost, machine learning
TL;DR: Zero-cost neural architecture search with two forward passes: no gradients, no labels, any architecture type, high accuracy.
Abstract: In recent years, zero-cost proxies are gaining grounds in the field of neural architecture search (NAS). These methods allow to find the optimal neural network for a given task faster and with lesser computational load than conventional NAS methods. Equally important is the fact that they also shed some light on the internal workings of neural architectures. In this paper we present a zero-cost metric that is highly correlated with the train accuracy across the NAS-Bench-101, NAS-Bench-201 and NAS-Bench-NLP benchmark datasets. Architectures are initialised with two distinct constant shared weights, one at a time. Then, a fixed random mini-batch of data is passed forward through each initialisation. We observe that the dispersion of the outputs between two initialisations is positively correlated with trained accuracy. The correlation further improves when the dispersion is normalised by the average output magnitude. Our metric does not require gradients computation and true labels. It thus unbinds NAS procedure from training hyperparameters, loss metric and human-labelled data. Our method is easy to integrate within existing NAS algorithms and takes a fraction of second to evaluate a single network.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Submission Guidelines: Yes
Please Choose The Closest Area That Your Submission Falls Into: General Machine Learning (ie none of the above)
5 Replies
Loading