Exploiting Network Compressibility and Topology in Zero-Cost NAS

Lichuan Xiang, Rosco Hunter, Minghao Xu, Łukasz Dudziak, Hongkai Wen

Keywords: Zero-cost Proxy, Neural Network Architecture Search, Deep Learning

TL;DR: Zero-Cost Metric Attribute-conditioned Exploration of Neural Network Design Space using Conditional Continuous Normalizing Flows

Abstract: Neural Architecture Search (NAS) has been widely used to discover high-performance neural network architectures over manually designed approaches. Despite their success, current NAS approaches often require extensive evaluation of many candidate architectures in the search space or training of large super networks. To reduce the search cost, recently proposed zero-cost proxies are utilized to efficiently predict the performance of an architecture. However, while many new proxies have been proposed in recent years, relatively little attention has been dedicated to pushing our understanding of the existing ones, with their mutual effects on each other being a particularly -- but not entirely -- overlooked topic. Contrary to that trend, in our work, we argue that it is worth revisiting and analysing the existing proxies in order to further push the boundaries of zero-cost NAS. Towards that goal, we propose to view the existing proxies through a common lens of network compressibility, trainability, and expressivity, as discussed in pruning literature. Notably, doing so allows us to build a better understanding of the high-level relationship between different proxies as well as refine some of them into their more informative variants. We leverage these insights to design a novel saliency and metric aggregation method informed by compressibility, orthogonality and network topology. We show that our proposed methods are simple but powerful and yield some state-of-the-art results across popular NAS benchmarks.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Reviewers: Yes

CPU Hours: 320

GPU Hours: 320

TPU Hours: 0

Evaluation Metrics: No