A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control

Zilin Kang, Chenyuan Hu, Yu Luo, Zhecheng Yuan, Ruijie Zheng, Huazhe Xu

TL;DR: We find a forget-and-grow approach for deep reinforcement learning scaling in continuous control

Abstract: Deep reinforcement learning for continuous control has recently achieved impressive progress. However, existing methods often suffer from primacy bias—a tendency to overfit early experiences stored in the replay buffer—which limits an RL agent’s sample efficiency and generalizability. A common existing approach to mitigate this issue is periodically resetting the agent during training. Yet, even after multiple resets, RL agents could still be impacted by early experiences. In contrast, humans are less susceptible to such bias, partly due to *infantile amnesia*, where the formation of new neurons disrupts early memory traces, leading to the forgetting of initial experiences. Inspired by this dual processes of forgetting and growing in neuroscience, in this paper, we propose *Forget and Grow* (**FoG**), a new deep RL algorithm with two mechanisms introduced. First, *Experience Replay Decay (ER Decay)*—"forgetting early experience''—which balances memory by gradually reducing the influence of early experiences. Second, *Network Expansion*—"growing neural capacity''—which enhances agents' capability to exploit the patterns of existing data by dynamically adding new parameters during training. Empirical results on four major continuous control benchmarks with more than 40 tasks demonstrate the superior performance of **FoG** against SoTA existing deep RL algorithms, including BRO, SimBa and TD-MPC2.

Lay Summary:

AI systems learning to control robots, like those for movement or grasping objects, often get stuck on their initial experiences. These early "memories" are replayed so much they can prevent the AI from fully using newer lessons, trapping it in old habits.

To tackle this, we developed "Forget and Grow" (FoG), a method inspired by human learning. FoG has two key features: first, it gradually lessens the influence of early experiences, much like humans forget their infant memories, preventing them from becoming overly dominant. Second, it dynamically increases the AI's learning capacity – its "brain" makes new connections during training, enabling it to better handle new tasks.

Our tests on various robotic control tasks show that FoG significantly improves learning efficiency and adaptability, helping AIs learn better and faster.

Link To Code: https://github.com/nothingbutbut/FoG.git

Primary Area: Reinforcement Learning->Deep RL

Keywords: Deep Continuous Control, Infantile Amnesia, Primacy Bias, Experience Replay

Flagged For Ethics Review: true

Submission Number: 8755