Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques

Natalia Zhang, Xinqi Wang, Qiwen Cui, Runlong Zhou, Sham M. Kakade, Simon Shaolei Du

Keywords: multi-agent reinforcement learning, reinforcement learning with human feedback, dataset coverage

TL;DR: We study Multi-Agent Reinforcement Learning from Human Feedback (MARLHF) with preference-only offline data.

Abstract:

We initiate the study of Multi-Agent Reinforcement Learning from Human Feedback (MARLHF), exploring both theoretical foundations and empirical validations. We define the task as identifying Nash equilibrium from a preference-only offline dataset in general-sum games, a problem marked by the challenge of sparse feedback signals. Our theory establishes the upper complexity bounds for Nash Equilibrium in effective MARLHF, demonstrating that single-policy coverage is inadequate and highlighting the importance of unilateral dataset coverage. These theoretical insights are verified through comprehensive experiments. To enhance the practical performance, we further introduce two algorithmic techniques.
(1) We propose a Mean Squared Error (MSE) regularization along the time axis to achieve a more uniform reward distribution and improve reward learning outcomes.
(2) We propose an extra penalty based on dataset distribution to incorporate pessimism, enhancing stability and effectiveness during training.
Our findings underscore the multifaceted approach required for MARLHF, paving the way for effective preference-based multi-agent systems.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7373