Meta-Reinforcement Learning With Evolving Gradient Regularization

Jiaxing Chen, Ao Ma, Shaofei Chen, Weilin Yuan, Zhenzhen Hu, Peng Li

Abstract: Deep reinforcement learning (DRL) typically requires reinitializing training for new tasks, limiting its generalization due to isolated knowledge transfer. Meta-reinforcement learning (Meta-RL) addresses this by enabling rapid adaptation through prior task experiences, yet existing gradient-based methods like MAML suffer from poor out-of-distribution performance due to overfitting narrow task distributions. To overcome this limitation, we propose Evolving Gradient Regularization MAML (ER-MAML). By integrating evolving gradient regularization into the MAML framework, ER-MAML optimizes meta-gradients while constraining adaptation directions via a regularization policy. This dual mechanism prevents overparameterization and enhances robustness across diverse task distributions. Experiments demonstrate ER-MAML outperforms state-of-the-art baselines by 14.6% in out-of-distribution success rates. It also achieves strong online adaptation performance in the MetaWorld benchmark. These results validate ER-MAML's effectiveness in improving meta-RL generalization under distribution shifts.