EReLELA: Exploration in Reinforcement Learning via Emergent Language Abstractions
Keywords: Emergent Communication, Exploration, Reinforcement Learning, Abstraction, Emergent Languages, Natural Languages
TL;DR: This paper proposes to use abstractions guided by cheap Emergent Languages instead of expensive Natural ones to improve Exploration in Reinforcement Learning, showing similar performance despite differences in the kind of abstractions performed.
Abstract: The ability of AI agents to follow natural language (NL) instructions is important for Human-AI collaboration.
Training Embodied AI agents for instruction-following can be done with Reinforcement Learning (RL), yet it poses many challenges.
Among which is the exploitation versus exploration trade-off in RL.
Previous works have shown that NL-based state abstractions can help address this challenge.
However, NLs descriptions have limitations in that they are not always readily available and are expensive to collect.
In order to address these limitations, we propose to use the Emergent Communication paradigm, where artificial agents learn an emergent language (EL) in an unsupervised fashion, via referential games.
Thus, ELs constitute cheap and readily-available abstractions.
In this paper, we investigate (i) how EL-based state abstractions compare to NL-based ones for RL in hard-exploration, procedurally-generated environments, and (ii) how properties of the referential games used to learn ELs impact the quality of the RL exploration and learning.
We provide insights about the kind of state abstractions performed by NLs and ELs over RL state spaces, using our proposed Compactness Ambiguity Metric.
Our results indicate that our proposed EL-guided agent, entitled EReLELA, achieves similar performance as its NL-based counterparts without its limitations.
Our work shows that RL agents can leverage unsupervised EL abstractions to greatly improve their exploration skills in sparse reward settings, thus opening new research avenues between Embodied AI and Emergent Communication.
Supplementary Material: zip
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10736
Loading