Automated Design in Hybrid Action Spaces by Reinforcement Learning and Differential Evolution

Quirin Göttl, Haris Asif, Alexander Mattick, Robert Marzilger, Axel Plinge

Abstract: Many real world applications of artificial intelligence and machine learning require to solve a given task inside a hybrid action space. While it is possible to approach these situations with frameworks based solely on reinforcement learning (RL), it is also possible to only use RL for the discrete part of the action space and let an optimization method at hand take care of the continuous actions. In the present work, a framework suitable for hybrid action spaces is derived that uses RL for the discrete decisions alongside a genetic algorithm for the continuous decisions. The RL methodology is based on hierarchical decomposition of the action space and Gumbel AlphaZero. The genetic algorithm is based on differential evolution and can be applied inside continuous action spaces of arbitrary dimension. The flexibility of the approach is shown in two real world use-cases, namely, synthesis of chemical process flowsheets and design of optical multi-layer films. The framework shows promising results compared to solely RL-based methodologies. The versatile framework introduced here could be applied in many other domains, such as, control of energy systems or navigation tasks.