Runtime Analysis of Evolutionary NAS for Multiclass Classification

Zeqiong Lv, Chao Qian, Yun Liu, Jiahao Fan, Yanan Sun

Abstract: Evolutionary neural architecture search (ENAS) is a key part of evolutionary machine learning, which commonly utilizes evolutionary algorithms (EAs) to automatically design high-performing deep neural architectures. During past years, various ENAS methods have been proposed with exceptional performance. However, the theory research of ENAS is still in the infant. In this work, we step for the runtime analysis, which is an essential theory aspect of EAs, of ENAS upon multiclass classification problems. Specifically, we first propose a benchmark to lay the groundwork for the analysis. Furthermore, we design a two-level search space, making it suitable for multiclass classification problems and consistent with the common settings of ENAS. Based on both designs, we consider (1+1)-ENAS algorithms with one-bit and bit-wise mutations, and analyze their upper and lower bounds on the expected runtime. We prove that the algorithm using both mutations can find the optimum with the expected runtime upper bound of $O(rM\ln{rM})$ and lower bound of $\Omega(rM\ln{M})$. This suggests that a simple one-bit mutation may be greatly considered, given that most state-of-the-art ENAS methods are laboriously designed with the bit-wise mutation. Empirical studies also support our theoretical proof.

Lay Summary:

Building effective neural (network) architectures is a core challenge in machine learning. Evolutionary neural architecture search (ENAS) addresses this by evolving high-performing deep neural architectures with evolutionary algorithms. While ENAS works well in practice, its theoretical understanding—especially how long it takes to find the optimum—remains limited.

We study the runtime of ENAS on multiclass classification tasks. We first build a new benchmark and design a two-level search space to reflect realistic ENAS settings. Then, we analyze the runtime of simple ENAS algorithms using different mutations.

Our analysis shows that a simple one-bit mutation may be greatly considered, given that most state-of-the-art ENAS methods are laboriously designed with the bit-wise mutation. This can deepen our theoretical understanding of ENAS and offer practical guidance for designing faster and simpler ENAS algorithms.

Primary Area: Theory->Deep Learning

Keywords: Evolutionary neural architecture search; evolutionary machine learning; evolutionary algorithm

Submission Number: 5666