Learning Homophilic Incentives in Sequential Social Dilemmas

Heng Dong, Tonghan Wang, Jiayuan Liu, Chi Han, Chongjie Zhang

Keywords: Multi-Agent Reinforcement Learning, Sequential Social Dilemma, Cooperation Emergence

Abstract: Promoting cooperation among self-interested agents is a long-standing and interdisciplinary problem, but receives less attention in multi-agent reinforcement learning (MARL). Game-theoretical studies reveal that altruistic incentives are critical to the emergence of cooperation but their analyses are limited to non-sequential social dilemmas. Recent works using deep MARL also show that learning to incentivize other agents has the potential to promote cooperation in more realistic sequential social dilemmas (SSDs). However, we find that, with these incentivizing mechanisms, the team cooperation level does not converge and regularly oscillates between cooperation and defection during learning. We show that a second-order social dilemma resulting from the incentive mechanisms is the main reason for such fragile cooperation. We analyze the dynamics of second-order social dilemmas and find that a typical tendency of humans, called homophily, provides a promising solution. We propose a novel learning framework to encourage homophilic incentives and show that it achieves stable cooperation in both SSDs of public goods and tragedy of the commons.

One-sentence Summary: We pinpoint the problem of second-order social dilemmas, analyze their dynamics, and propose a novel learning framework to encourage homophilic incentives to solve this problem.