DISCO-DANCE: Learning to Discover Skills with Guidance

Hyunseung Kim, Byungkun Lee, Sejik Park, Hojoon Lee, Dongyoon Hwang, Kyushik Min, Jaegul Choo

Keywords: Unsupervised skill discovery, Reinforcement Learning

TL;DR: This paper proposes a novel unsupervised skill learning algorithm GSD, which attempts to provide direct guidance in order to accelerate the learning process of diverse skills by encouraging further exploration.

Abstract:

Unsupervised skill discovery (USD) allows agents to learn diverse and discriminable skills without access to pre-defined rewards,
by maximizing the mutual information (MI) between skills and states reached by each skill.
The most common problem of MI-based skill discovery is insufficient exploration, because each skill is heavily penalized when it deviates from its initial settlement. Recent works introduced an auxiliary reward to encourage the exploration of the agent via maximizing the state's epistemic uncertainty or entropy.
However, we have discovered that the performance of these auxiliary rewards decreases as the environment becomes more challenging. Therefore, we introduce a new unsupervised skill discovery algorithm, skill discovery with guidance (DISCO-DANCE), which (1) selects the guide skill which has the highest potential to reach the unexplored states, (2) guide other skills to follow the guide skill, then (3) the guided skills are diffused to maximize their discriminability in the unexplored states. Empirically, DISCO-DANCE substantially outperforms other USD baselines on challenging environments including two navigation benchmarks and a continuous control benchmark.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Reinforcement Learning (eg, decision and control, planning, hierarchical RL, robotics)