Task Similarity Matters: Greedy Orderings in Continual Linear Regression

Ran Levinstein, Matan Tsipory, Itay Evron, Mark Kong, Deanna Needell, Daniel Soudry

Keywords: continual learning, lifelong learning, task ordering, curriculum learning, forgetting, kaczmarz

TL;DR: We study similarity-guided task orderings in continual learning; formalize greedy orderings, and develop a framework to analyze and understand them.

Abstract:

We analyze task ordering strategies in continual learning for realizable linear regression.
We focus on task orderings that greedily maximize dissimilarity between consecutive tasks, a concept briefly explored in prior work but still surrounded by open questions.
Using tools from the Kaczmarz method literature, we formalize these orderings and develop both geometric and algebraic intuitions around them.
We show empirically that, under random data, greedy orderings lead to faster convergence of the loss compared to random orderings.
In a simplified setting, we prove bounds on the loss and establish optimality guarantees for greedy orderings.
However, we also construct an adversarial task sequence that exploits high dimensionality to induce maximal forgetting under greedy orderings—an effect to which random orderings are notably more robust.
Altogether, our findings advance the theoretical understanding of task orderings in continual learning, offer new insights into Kaczmarz methods, and provide a foundation for future research.

Submission Number: 16