Human-Level Competitive Pokémon via Scalable Offline Reinforcement Learning with Transformers
Keywords: Offline RL, Meta-RL, Transformers
Abstract: Competitive Pokémon Singles (CPS) is a popular strategy game where players learn to exploit their opponent based on imperfect information in battles that can last more than one hundred stochastic turns. AI research in CPS is led by heuristic tree search and online self-play, but the game may create a platform to study adaptive policies trained offline on large datasets. We develop a pipeline to reconstruct the first-person perspective of an agent from logs saved from the third-person perspective of a spectator, thereby unlocking a dataset of real human battles spanning more than a decade that grows larger every day. This dataset enables a black-box approach where we train large sequence models to adapt to their opponent based solely on their input trajectory while selecting moves without explicit search of any kind. We study a progression from imitation learning to offline RL and offline fine-tuning on self-play data in the hardcore competitive setting of Pokémon’s four oldest (and most partially observed) game generations. The resulting agents outperform recent LLM approaches and rival or exceed the best heuristic search engines. Playing anonymously in online battles against humans, our agents surpass a 50% estimated win rate in all four generations and climb inside the top ranked players in the game’s longest-horizon rulesets.
Submission Number: 340
Loading