From Continual Learning to SGD and Back: Better Rates for Continual Linear Models

Itay Evron, Ran Levinstein, Matan Schliserman, Uri Sherman, Tomer Koren, Daniel Soudry, Nathan Srebro

Keywords: Continual learning, Lifelong learning, Last iterate, SGD, Forgetting, Task ordering, Single pass, Kaczmarz

TL;DR: We prove a reduction from continual learning to stepwise-optimal SGD, and derive novel last-iterate SGD analysis to yield tighter forgetting rates for continual linear models

Abstract:

We study the common continual learning setup where an overparameterized model is sequentially fitted to a set of jointly realizable tasks. We analyze forgetting, defined as the loss on previously seen tasks, after $k$ iterations. For continual linear models, we prove that fitting a task is equivalent to a single stochastic gradient descent (SGD) step on a modified objective. We develop novel last-iterate SGD upper bounds in the realizable least squares setup and leverage them to derive new results for continual learning. Focusing on random orderings over $T$ tasks, we establish universal forgetting rates, whereas existing rates depend on problem dimensionality or complexity and become prohibitive in highly overparameterized regimes.
In continual regression with replacement, we improve the best existing rate from $\mathcal{O}((d-\bar{r})/k)$ to
$\mathcal{O}(\min(1/\sqrt[4]{k}, \sqrt {d-\bar{r}}/k, \sqrt {T\bar{r}}/k))$, where $d$ is the dimensionality and $\bar{r}$ the average task rank.
Furthermore, we establish the first rate for random task orderings without replacement.
The resulting rate of $\mathcal{O}(\min(1/\sqrt[4]{T}, (d-\bar{r})/T))$ shows that randomization alone, without task repetition, prevents catastrophic forgetting in sufficiently long task sequences.
Finally, we prove a matching $\mathcal{O}(1/\sqrt[4]{k})$ forgetting rate for continual linear classification on separable data.
Our universal rates extend to broader methods, such as block Kaczmarz and POCS, illuminating their loss convergence under i.i.d. and single-pass orderings.

Submission Number: 56