Dynamic Mean-Field Control for Network MDPs with Exogenous Demand

Botao Ye, Hengquan Guo, Weichang Wang, Xin Liu

Keywords: mean-field control; demand matching; reinforcement learning

Abstract:

This paper studies the network control problems with exogenous demand, where network controller must dynamically allocate resources to satisfy exogenous demands with unknown distributions.
We formalize the problem using Networked Markov Decision Processes with Exogenous Demands (Exo-NMDPs), where the system states are decoupled into endogenous states and stochastic exogenous demands.
However, Exo-NMDPs pose three main challenges: scalability in large-scale networks; stochasticity from fluctuating exogenous demands; and delayed feedback of scheduling actions.
To address these issues, we propose the Dynamic Mean-Field Control (DMFC) algorithm, a scalable and computationally efficient approach for matching exogenous demands. Specifically, DMFC transforms the high-dimensional actual states of the Exo-NMDP into low-dimensional mean-field states, and dynamically optimizes the policy by solving a mean-field control problem at each time step.
This enables DMFC to capture spatiotemporal correlations between demand and system state, while remaining robust against demand fluctuations and action execution delay.
We validate DMFC on two representative scenarios: supply-chain inventory management and vehicle routing. Our experimental results show that DMFC adapts well to various demand patterns and outperforms state-of-the-art baselines in both scenarios.

Submission Number: 26