Prioritizing safety via curriculum learning

Cevahir Koprulu, Thiago D. Simão, Nils Jansen, ufuk topcu

Confirmation: Yes

Keywords: Curriculum learning, constrained reinforcement learning

TL;DR: We propose a safe curriculum generation method that reduces safety constraint violations during training while boosting the learning speed of constrained RL agents.

Abstract: Curriculum learning for reinforcement learning (RL) aims to accelerate learning by generating sequences of tasks of increasing difficulty. Besides its sample-efficiency benefits, curriculum learning has the potential to address safety-critical settings where an RL agent must adhere to safety constraints. However, existing curriculum generation approaches still overlook such constraints and thus propose tasks that cause RL agents to violate safety constraints during training and behave suboptimally after. We propose a safe curriculum generation approach (SCG) that aligns the objectives of constrained RL and curriculum learning: improving safety during training and boosting learning speed. SCG generates sequences of tasks where the RL agent can be both safe and performant by initially preferring tasks with minimum safety violations over high-reward ones. In constrained RL environments, we empirically show that compared to the state-of-the-art curriculum learning approaches and their naively modified safe versions, SCG achieves optimal performance and the lowest amount of constraint violations during training.

Submission Number: 3