Construction of Football Agents by Inverse Reinforcement Learning Using Relative Positional Information Among Players

Daiki Wakabayashi, Tomoaki Yamazaki, Kouzou Ohara

Abstract: Recent advancements in reinforcement learning have made it possible to develop football agents that autonomously emulate the behavior of human players. However, it is still challenging for existing methods to successfully replicate realistic player behaviors. In fact, agents exhibit behaviors like clustering around the ball or shooting prematurely. One cause of this problem lies in reward functions that always assign large rewards to certain actions, such as scoring a goal, regardless of the situation, which bias agents towards high-reward actions. In this study, we incorporate the relative positional reward and the positional weight for shooting into the reward function used for reinforcement learning. The relative positional reward, derived from the positions of players, the ball, and the goal, is estimated using inverse reinforcement learning on a dataset of real football games. The positional weight for shooting is similarly based on actual shooting positions observed in these ga