Towards Efficient Multi-Agent Learning Systems

Kailash Gogineni, Peng Wei, Tian Lan, Guru Prasadh Venkataramani

Keywords: Multi-Agent Systems, Performance Analysis, Reinforcement Learning, Performance Optimization

Abstract: Multi-Agent Reinforcement Learning (MARL) is an increasingly popular domain for modeling and controlling multiple large-scale autonomous systems. Existing multi-agent learning implementations typically involve intensive computations regarding training time and power requirements arising from large observation-action space and a huge number of training steps. Therefore, a key challenge is understanding and characterizing the computationally intensive functions in several popular classes of MARL algorithms during their training phases. Our preliminary experiments reveal new insights into the key modules of MARL algorithms that limit their adoption in real-world systems. We explore neighbor sampling strategy to improve the cache locality and observe performance improvement ranging from $26.66\%$ ($3$ agents) to $27.39\%$ ($12$ agents) for the computationally intensive mini-batch sampling phase. Additionally, we demonstrate that improving the cache locality leads to an end-to-end training time reduction of $10.2\%$~(for $12$ agents) compared to existing multi-agent algorithms without significant degradation in the mean reward.

Workshop Track: MLArchSys

TL;DR: This research conducts workload characterization of MARL algorithms and addresses the computational challenges through a neighbor sampling strategy, resulting in considerable performance improvements.

Presentation: In-Person

Presenter Full Name: Kailash Gogineni

Presenter Email: [e-mail]

Presenter Bio: Kailash Gogineni is a Ph.D. student at George Washington University under the guidance of Dr. Guru Venkataramani. His research focuses on efficient machine learning in systems and hardware/software security.