Human attention guided multiagent hierarchical reinforcement learning for heterogeneous agents

Dingbang Liu, Fenghui Ren, Jun Yan, Guoxin Su, Shohei Kato, Wen Gu, Minjie Zhang

Abstract: The imperative for coordination among intelligent machines has popularized cooperative multiagent reinforcement learning (MARL) in AI research. However, compared to the well-explored homogeneous agent cooperation, heterogeneous agents with different attributes or behaviors are more prevalent in practical scenarios yet they have received relatively little attention. Due to the heterogeneity of agents and the diversity of relationships, learning efficient coordination among heterogeneous agents is particularly challenging, suffering from the curse of dimensionality and the start-up problem. To tackle these challenges, we propose a novel method that connects humans and agents under a hierarchical structure to guide the learning of MARL agents. Drawing inspiration from knowledge transfer among diverse human individuals, we consider human attention as a general pattern that can be applied to heterogeneous agents. Instead of relying on comprehensive step-by-step demonstrations, we utilize fuzzy logic to capture the abstraction and vagueness within suboptimal human guidance. To avoid negative knowledge transfer, we leverage human attention as an auxiliary source through hyper-networks, allowing agents to selectively adapt to the proposed human prior knowledge. The proposed method is agnostic to specific MARL methods and can be flexibly integrated with diverse algorithms. We conduct experiments on challenging tasks within the StarCraft Multiagent Challenge (SMAC) and SMACv2 environments, and the empirical results demonstrate that our method can improve existing methods in several heterogeneous scenarios.