Exploiting Network Compressibility and Topology in Zero-Cost NASDownload PDF

Published: 16 May 2023, Last Modified: 18 Sept 2023AutoML 2023 MainTrackReaders: Everyone
TL;DR: Zero-Cost Metric Attribute-conditioned Exploration of Neural Network Design Space using Conditional Continuous Normalizing Flows
Abstract: Neural Architecture Search (NAS) has been widely used to discover high-performance neural network architectures over manually designed approaches. Despite their success, current NAS approaches often require extensive evaluation of many candidate architectures in the search space or training of large super networks. To reduce the search cost, recently proposed zero-cost proxies are utilized to efficiently predict the performance of an architecture. However, while many new proxies have been proposed in recent years, relatively little attention has been dedicated to pushing our understanding of the existing ones, with their mutual effects on each other being a particularly -- but not entirely -- overlooked topic. Contrary to that trend, in our work, we argue that it is worth revisiting and analysing the existing proxies in order to further push the boundaries of zero-cost NAS. Towards that goal, we propose to view the existing proxies through a common lens of network compressibility, trainability, and expressivity, as discussed in pruning literature. Notably, doing so allows us to build a better understanding of the high-level relationship between different proxies as well as refine some of them into their more informative variants. We leverage these insights to design a novel saliency and metric aggregation method informed by compressibility, orthogonality and network topology. We show that our proposed methods are simple but powerful and yield some state-of-the-art results across popular NAS benchmarks.
Keywords: Zero-cost Proxy, Neural Network Architecture Search, Deep Learning
Submission Checklist: Yes
Broader Impact Statement: Yes
Paper Availability And License: Yes
Code Of Conduct: Yes
Reviewers: Yes
Code And Dataset Supplement: zip
CPU Hours: 320
GPU Hours: 320
TPU Hours: 0
Evaluation Metrics: No
21 Replies