Action Space Design in Reinforcement Learning for Robot Motor Skills

Julian Eßer, Gabriel B. Margolis, Oliver Urbann, Sören Kerner, Pulkit Agrawal

Keywords: Reinforcement Learning, Action Spaces, Sim-to-Real

TL;DR: We examine action space selection for robots trough the lense of exploration, expressivity, and timing.

Abstract:

Practitioners often rely on intuition to select action spaces for learning. The choice can substantially impact final performance even when choosing among configuration-space representations such as joint position, velocity, and torque commands. We examine action space selection considering a wheeled-legged robot, a quadruped robot, and a simulated suite of locomotion, manipulation, and control tasks.
We analyze the mechanisms by which action space can improve performance and conclude that the action space can influence learning performance substantially in a task-dependent way. Moreover, we find that much of the practical impact of action space selection on learning dynamics can be explained by improved policy initialization and behavior between timesteps.

Student Paper: yes

Submission Number: 656