On the Statistical Efficiency of Mean Field RL with General Function Approximation

Jiawei Huang, Batuhan Yardim, Niao He

Keywords: Reinforcement Learning Theory, Mean-Field Game, Mean-Field Control, Multi-Agent RL

TL;DR: We provide algorithms and sample complexity guarantees for RL in MFC and MFG with general function approximation, and provide evidence for the exponential separation between MFC and MFG from sample complexity perspective.

Abstract: In this paper, we study the statistical efficiency of Reinforcement Learning in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general function approximation. We introduce a new concept called Mean-Field Model-Based Eluder Dimension (MBED), which subsumes a rich family of Mean-Field RL problems. Additionally, we propose algorithms based on Optimistic Maximal Likelihood Estimation, which can return an $\epsilon$-optimal policy for MFC or an $\epsilon$-Nash Equilibrium policy for MFG, with sample complexity polynomial w.r.t. relevant parameters and independent of the number of states, actions and the number of agents. Notably, our results only require a mild assumption of Lipschitz continuity on transition dynamics and avoid strong structural assumptions in previous work. Finally, in the tabular setting, given the access to a generative model, we establish an exponential lower bound for MFC setting, while providing a novel sample-efficient model elimination algorithm to approximate equilibrium in MFG setting. Our results reveal a fundamental separation between RL for single-agent, MFC, and MFG from the sample efficiency perspective.