Superior Performance with Diversified Strategic Control in FPS Games Using General Reinforcement Learning

Shuxing Li, Jiawei Xu, Chun Yuan, peng sun, Zhuobin Zheng, Zhengyou Zhang, Lei Han

Keywords: Reinforcement Learning, Hindsight Experience Replay, FPS Games

Abstract: This paper offers an overall solution for first-person shooter (FPS) games to achieve superior performance using general reinforcement learning (RL). We introduce an agent in ViZDoom that can surpass previous top agents ranked in the open ViZDoom AI Competitions by a large margin. The proposed framework consists of a number of generally applicable techniques, including hindsight experience replay (HER) based navigation, hindsight proximal policy optimization (HPPO), rule-guided policy search (RGPS), prioritized fictitious self-play (PFSP), and diversified strategic control (DSC). The proposed agent outperforms existing agents by taking advantage of diversified and human-like strategies, instead of larger neural networks, more accurate frag skills, or hand-craft tricks, etc. We provide comprehensive analysis and experiments to elaborate the effect of each component in affecting the agent performance, and demonstrate that the proposed and adopted techniques are important to achieve superior performance in general end-to-end FPS games. The proposed methods can contribute to other games and real-world tasks which also require spatial navigation and diversified behaviors.

One-sentence Summary: This paper offers an overall solution for first-person shooter (FPS) games to achieve superior performance using general reinforcement learning (RL).