Deep Generalized Schrödinger Bridge

Guan-Horng Liu, Tianrong Chen, Oswin So, Evangelos Theodorou

Keywords: Mean-Field Game, Schrödinger Bridge, Deep Reinforcement Learning, Forward-Backward Stochastic Differential Equations

TL;DR: A novel numerical method connecting Schrödinger Bridge with Deep Reinforcement Learning for solving a challenging class of Mean-Field Games with hard distributional boundary constraints.

Abstract: Mean-Field Game (MFG) serves as a crucial mathematical framework in modeling the collective behavior of individual agents interacting stochastically with a large population. In this work, we aim at solving a challenging class of MFGs in which the differentiability of these interacting preferences may not be available to the solver, and the population is urged to converge exactly to some desired distribution. These setups are, despite being well-motivated for practical purposes, complicated enough to paralyze most (deep) numerical solvers. Nevertheless, we show that Schrödinger Bridge — as an entropy-regularized optimal transport model — can be generalized to accepting mean-field structures, hence solving these MFGs. This is achieved via the application of Forward-Backward Stochastic Differential Equations theory, which, intriguingly, leads to a computational framework with a similar structure to Temporal Difference learning. As such, it opens up novel algorithmic connections to Deep Reinforcement Learning that we leverage to facilitate practical training. We show that our proposed objective function provides necessary and sufficient conditions to the mean-field problem. Our method, named Deep Generalized Schrödinger Bridge (DeepGSB), not only outperforms prior methods in solving classical population navigation MFGs, but is also capable of solving 1000-dimensional opinion depolarization, setting a new state-of-the-art numerical solver for high-dimensional MFGs. Our code will be made available at https://github.com/ghliu/DeepGSB.

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/arxiv:2209.09893/code)