AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversification

ICLR 2026 Conference Submission16033 Authors

Keywords: unsupervised reinforcement learning, skill discovery, self-supervised learning, multi-joint robot locomotion

TL;DR: AMPED proposes a novel framework for skill‐based reinforcement learning that simultaneously maximizes state coverage and skill diversity through several carefully designed components.

Abstract: Skill-based reinforcement learning (SBRL) enables rapid adaptation in environments with sparse rewards by pretraining a skill-conditioned policy. Effective skill learning requires jointly maximizing both exploration and skill diversity. However, existing methods often face challenges in simultaneously optimizing for these two conflicting objectives. In this work, we propose a new method, Adaptive Multi-objective Projection for balancing Exploration and skill Diversification (AMPED), which explicitly addresses both: during pre-training, a gradient-surgery projection balances the exploration and diversity gradients, and during fine-tuning, a skill selector exploits the learned diversity by choosing skills suited to downstream tasks. Our approach achieves performance that surpasses SBRL baselines across various benchmarks. Through an extensive ablation study, we identify the role of each component and demonstrate that each element in AMPED is contributing to performance. We further provide theoretical and empirical evidence that, with a greedy skill selector, greater skill diversity reduces fine-tuning sample complexity. These results highlight the importance of explicitly harmonizing exploration and diversity and demonstrate the effectiveness of AMPED in enabling robust and generalizable skill learning.

Primary Area: reinforcement learning

Submission Number: 16033