Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning

Edoardo Urettini, Antonio Carta

Abstract:

Online Continual Learning (OCL) models continuously adapt to nonstationary data streams, usually without task information. These settings are complex and many traditional CL methods fail, while online methods (mainly replay-based) suffer from instabilities after the task shift. To address this issue, we formalize replay-based OCL as a second-order online joint optimization with explicit KL-divergence constraints on replay data. We propose Online Curvature-Aware Replay (OCAR) to solve the problem: a method that leverages second-order information of the loss using a K-FAC approximation of the Fisher Information Matrix (FIM) to precondition the gradient. The FIM acts as a stabilizer to prevent forgetting while also accelerating the optimization in non-interfering directions. We show how to adapt the estimation of the FIM to a continual setting, stabilizing second-order optimization for non-iid data, uncovering the role of the Tikhonov damping in the stability-plasticity tradeoff.
Empirical results show that OCAR outperforms state-of-the-art methods in continual metrics, achieving higher average accuracy throughout the training process in three different benchmarks.

Lay Summary:

As neural networks are used in real-world settings, it's important that they can learn new information in real-time without forgetting what they already know. This is especially challenging when new data comes in continuously and without clear breaks: a situation known as online continual learning.
Many traditional learning methods struggle in these conditions, especially when the type of task or data changes suddenly. One common approach, called replay, helps by mixing in examples from past data, but it often becomes unstable when the data shifts.
To tackle this, we developed a new method called Online Curvature-Aware Replay (OCAR). It helps neural networks remember old knowledge while learning new things more effectively. From a mathematical point of view, as the objective is to find the minimum of a function, OCAR, instead of following only the direction with the highest slope, also considers the curvature of the function. Doing this with replay allows us to avoid changing the parameters that are important for past knowledge.
Our experiments showed that OCAR consistently performs better than existing methods, helping AI models stay accurate and stable over time across different types of learning tasks. This brings us closer to creating AI that learns more like humans do — continuously and reliably.

Link To Code: https://github.com/edo-urettini/CL_stability

Primary Area: Deep Learning

Keywords: Online Continual Learning, Second-order optimization, Deep Learning

Submission Number: 10994