Skill Discovery for Exploration and Planning using Deep Skill Graphs

Akhil Bagaria, Jason Crowley, Jing Wei Nicholas Lim, George Konidaris

Student First Author: Yes

Keywords: Deep reinforcement learning, hierarchical reinforcement learning, planning, exploration

Abstract: Temporal abstraction provides an opportunity to drastically lower the decision making burden facing reinforcement learning agents in rich sensorimotor spaces. Well constructed hierarchies induce state and action abstractions that can reduce large continuous MDPs into small discrete ones, in which planning with a learned model is feasible. We propose a novel algorithm, Deep Skill Graphs, for acquiring such a minimal representation of an environment. Our algorithm seamlessly interleaves discovering skills and planning using them to gain unsupervised mastery over ever increasing portions of the state-space. The constructed skill graph can be used to drive the agent to novel goals at test time, requiring little-to-no additional learning. We test our algorithm on a series of continuous control tasks where it outperforms baseline flat and hierarchical RL methods alike.

TL;DR: We introduce a skill discovery algorithm that can be used for exploration and planning in continuous RL domains