Curriculum Generation and Sequencing for Deep Reinforcement Learning in StarCraft II

Daniel Hao, Penny Sweetser, Matthew Aitchison

Abstract: Reinforcement learning has proven successful in games, but suffers from long training times when compared to other forms of machine learning. Curriculum learning, an optimisation technique that improves a model’s ability to learn by presenting training samples in a meaningful order, known as curricula, could offer a solution for reinforcement learning. Due to limitations involved with automating curriculum learning, curricula are usually manually designed. However, due to a lack of research into effective design of curricula, researchers often rely on intuition and the resulting performance can vary. In this paper, we explore different ways of manually designing curricula for reinforcement learning in real-time strategy game, StarCraft II. We propose three generalised methods of manually creating tasks for curriculum learning and verify their effectiveness through experiments. We also experiment with different curricula sequences, in addition to the most commonly used easy-to-hard order. Our results show that all three of our proposed methods can improve a reinforcement learning agent’s learning process when used correctly. We demonstrate that modifying the state space of the tasks is the most effective way to create training samples for StarCraft II and that reversed curricula can be beneficial to an agent’s convergence process under certain circumstances.