FaDE: Fast DARTS Estimator on Hierarchical NAS Spaces

Simon Neumeyer, Julian Stier, Michael Granitzer

Keywords: darts, hierarchical space, neural architecture search, automl, differentiable structure optimization

TL;DR: FaDe approximates ranks for a pseudo-gradient open-ended neural architecture search through aggregates from a finite but differentiable hyperarchitecture

Abstract:

Vast search spaces and expensive architecture evaluations make neural architecture search a challenging problem.
Hierarchical search spaces allow for comparatively cheap evaluations on neural network sub modules to serve as surrogate for architecture evaluations.
Yet, sometimes the hierarchy is too restrictive or the surrogate fails to generalize.
We present FaDE that uses differentiable architecture search to iteratively obtain relative performance predictions on finite regions of a hierarchical neural architecture search space.
The relative nature of these ranks calls for a memory-less, batch-wise outer search algorithm which we provide in form of an evolutionary approach that features a pseudo-gradient descent.
FaDE is especially suited on deep hierarchical respectively multi-cell search spaces which it can explore by linear instead of exponential cost and therefore eliminates the need for a proxy search space.
FaDE solely trains on the neural architecture search space, not on any space of neural architecture sub modules.
Our experiments show that firstly, FaDE-ranks on finite regions of the search space correlate with corresponding architecture performances and secondly, the ranks can empower a pseudo-gradient evolutionary search on the complete neural architecture search space.

Submission Number: 33