Guiding reinforcement learning using constrained uncertainty-aware movement primitives

Abhishek Padalkar, Freek Stulp, Gerhard Neumann, João Silvério

Keywords: Learning from Demonstrations, Reinforcement Learning, Constrained Learning, Guided Learning

TL;DR: In this paper, we propose a constrained, uncertainty-aware movement primitive representation that leverages both demonstrations and hard constraints for sample efficient, safe reinforcement learning..

Abstract:

Guided reinforcement learning (RL) presents an effective approach for robots to acquire skills efficiently, directly in the real world environments.
Recent works indicate that incorporating hard constraints into RL can expedite the learning of manipulation tasks, enhance safety, and reduce the complexity of the reward function.
In parallel, learning from demonstration (LfD) using movement primitives is a well-established method for initializing RL policies.
In this paper, we propose a constrained, uncertainty-aware movement primitive representation that leverages both demonstrations and hard constraints to guide RL.
By integrating hard constraints, our approach aims to facilitate safer and sample-efficient learning, as the robot is not required to violate these constraints during the learning process.
At the same time, demonstrations are employed to offer a baseline policy that supports exploration.
Our method enhances state-of-the-art techniques by introducing a projector that enables state-dependent noise derived from demonstrations while ensuring that the constraints are respected throughout training.
Collectively, these elements contribute to safe and efficient learning alongside streamlined reward function design. We validate our framework through an insertion task involving a torque-controlled, 7-DoF robotic manipulator.

Submission Number: 4