Federated Model-Based Offline Multi-Agent Reinforcement Learning for Wireless Networks

Dohyeok Kwon, YeonseoJeong, Sungweon Hong, Songnam Hong

Keywords: Offline Multi-Agent Reinforcement Learning, Federated Learning, Model-based Reinforcement Learing, Distributed Channel Access

Abstract: Wireless networks are naturally modeled as multi-agent reinforcement learning (MARL) problems: distributed entities act on partial observations under interference and non-stationary traffic while pursuing common network objectives. Online exploration is risky and sample-inefficient in live systems, yet large operational logs are available, motivating an offline MARL approach. We introduce FedMORL, a federated model-based offline framework that shares an environment model rather than policy parameters. Each client learns dynamics and reward predictors from its logs and periodically aggregates them into a shared world model, which is then used locally to improve policies without environment interaction through (i) a planner-based policy learning scheme that improves training stability, and (ii) a rollout-based data augmentation mechanism that enhances local data coverage. The design preserves decentralized execution and limits privacy exposure by avoiding raw-data and policy sharing. Applied to the canonical Distributed Channel Access (DCA) task, FedMORL significantly reduces collisions and mean delay compared with rule-based baselines. We also empirically demonstrate that model federation is most beneficial, particularly under conditions of high data heterogeneity and limited local coverage. This supports model federation as a practical, privacy-preserving, offline-first path for multi-agent wireless control.

Submission Number: 83