Hierarchical RNNs-Based transformers MADDPG for mixed cooperative-competitive environments

Xiaolong Wei, Xianglin Huang, Lifang Yang, Gang Cao, Zhulin Tao, Bing Wang, Jing An

Abstract: Structural models based on Attention can not only record the relationships between features’ position, but also can measure the importance of different features based on their weights. By establishing dynamically weighted parameters for choosing relevant and irrelevant features, the key information can be strengthened, and the irrelevant information can be weakened. Therefore, the efficiency of Deep Learning algorithms can be significantly elevated and improved. Although Transformer have been performed very well in many fields including Reinforcement Learning (RL). We tried to integrate Transformers into RL, however there are some challenge in this task. Especially, MARL (known as Multi-Agent Reinforcement Learning), which can be recognized as a set of independent agents trying to adapt and learn through their way to reach the goal. In order to emphasize the relationship between each MDP decision in a certain time period, we applied the hierarchical coding method and validated the effectiveness of this method. This paper proposed a Hierarchical Transformer MADDPG based on recurrent neural network(RNN) which we call it Hierarchical RNNs-Based Transformers MADDPG(HRTMADDPG). It consists of a lower level encoder based on RNNs that encodes multiple step sizes in each time sequence, and it also consists of an upper sequence level encoder based on Transformer for learning the correlations between multiple sequences. Then we can capture the causal relationship between sub-time sequences and make HRTMADDPG more efficient.