ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value FactorizationDownload PDF

Published: 31 Oct 2022, Last Modified: 20 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: Multi-Agent Reinforcement Learning, Value Factorization, residual Q
TL;DR: ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization
Abstract: The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, we propose, ResQ, a MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function. ResQ can be used with mean-value and stochastic-value RL. We theoretically show that ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations. Through experiments on matrix games, the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.
Supplementary Material: zip
19 Replies

Loading