ES-ENAS: Blackbox Optimization over Hybrid Spaces via Combinatorial and Continuous Evolution

Xingyou Song, Krzysztof Marcin Choromanski, Jack Parker-Holder, Yunhao Tang, Daiyi Peng, Deepali Jain, Wenbo Gao, Aldo Pacchiano, Tamas Sarlos, Yuxiang Yang

Keywords: ES, ENAS, hybrid, search, space, blackbox, combinatorial, optimization, reinforcement, learning, mujoco, policies, evolutionary, computation, neuroevolution, high, dimension, supernet, one-shot, nas, neural, architecture, search, efficient

Abstract: We consider the problem of efficient blackbox optimization over a large hybrid search space, consisting of a mixture of a high dimensional continuous space and a complex combinatorial space. Such examples arise commonly in evolutionary computation, but also more recently, neuroevolution and architecture search for Reinforcement Learning (RL) policies. Unfortunately however, previous mutation-based approaches suffer in high dimensional continuous spaces both theoretically and practically. We thus instead propose ES-ENAS, a simple joint optimization procedure by combining Evolutionary Strategies (ES) and combinatorial optimization techniques in a highly scalable and intuitive way, inspired by the \textit{one-shot} or \textit{supernet} paradigm introduced in Efficient Neural Architecture Search (ENAS). Through this relatively simple marriage between two different lines of research, we are able to gain the best of both worlds, and empirically demonstrate our approach by optimizing BBOB functions over hybrid spaces as well as combinatorial neural network architectures via edge pruning and quantization on popular RL benchmarks. Due to the modularity of the algorithm, we also are able incorporate a wide variety of popular techniques ranging from use of different continuous and combinatorial optimizers, as well as constrained optimization.

One-sentence Summary: We combine ES/ARS (Gaussian smoothed gradient techniques for continuous inputs) with a variety of combinatorial optimizers (Evolutionary and Policy Gradient) in an ENAS-like fashion, producing a strong joint optimizer over hybrid search spaces.