Accelerating Reinforcement Learning Through GPU Atari Emulation

Steven Dalton, Michael Garland, Iuri Frosio

Keywords: GPU, reinforcement learning

TL;DR: This paper introduces a new library to emulate Atari games on a GPU and shows its benefits in terms of acceleration and scaling to multiple GPU system, while also providing an analysis of the advantages and disadvantages of GPU emulation for RL.

Abstract: We introduce CuLE (CUDA Learning Environment), a CUDA port of the Atari Learning Environment (ALE) which is used for the development of deep reinforcement algorithms. CuLE overcomes many limitations of existing CPU-based emulators and scales naturally to multiple GPUs. It leverages GPU parallelization to run thousands of games simultaneously and it renders frames directly on the GPU, to avoid the bottleneck arising from the limited CPU-GPU communication bandwidth. CuLE generates up to 155M frames per hour on a single GPU, a finding previously achieved only through a cluster of CPUs. Beyond highlighting the differences between CPU and GPU emulators in the context of reinforcement learning, we show how to leverage the high throughput of CuLE by effective batching of the training data, and show accelerated convergence for A2C+V-trace. CuLE is available at [hidden URL].