On Redundancy and Diversity in Cell-based Neural Architecture Search

Xingchen Wan, Binxin Ru, Pedro M Esperança, Zhenguo Li

Keywords: NAS, machine learning architectures, AutoML

Abstract:

Searching for the architecture cells is a dominant paradigm in NAS. However, little attention has been devoted to the analysis of the cell-based search spaces even though it is highly important for the continual development of NAS.
In this work, we conduct an empirical post-hoc analysis of architectures from the popular cell-based search spaces and find that the existing search spaces contain a high degree of redundancy: the architecture performance is less sensitive to changes at large parts of the cells, and universally adopted design rules, like the explicit search for a reduction cell, significantly increase the complexities but have very limited impact on the performance.
Across architectures found by a diverse set of search strategies, we consistently find that the parts of the cells that do matter for architecture performance often follow similar and simple patterns. By constraining cells to include these patterns, randomly sampled architectures can match or even outperform the state of the art.
These findings cast doubts into our ability to discover truly novel architectures in the existing cell-based search spaces and, inspire our suggestions for improvement to guide future NAS research.
Code is available at https://github.com/xingchenwan/cell-based-NAS-analysis.

One-sentence Summary: We analyse and explore the redundancies and diversity of popular cell-based search spaces in NAS.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2203.08887/code)