Inter-Task Learning Dynamics in Deep Linear Multi-Task Networks

Emilie Gregoire, Hannah Pinson, Sam Verboven

Keywords: Multi-Task Learning; Deep Learning; Learning Dynamics

Abstract:

Despite significant empirical progress in Multi-Task Learning (MTL), the theoretical understanding of task interactions and their dynamics remains limited. We present a theoretical analysis of how task alignment shapes learning dynamics in linear MTL, providing a theoretical justification for why task importance is inherently dynamic and why loss weighting schemes should adapt during training.
Leveraging the Riccati formulation of gradient flow, we analytically characterize the evolution and interaction of shared and task-specific components in deep linear neural networks. For a broad class of initializations, we show how task alignment and magnitude differences govern the trajectories of task outputs, losses, and neural representations throughout training, as well as the representations at convergence. Our analysis reveals that task alignment impacts learning speed and modulates the relative importance of tasks throughout training, with magnitude differences further amplifying these effects. We further show that these factors determine how the structural relationships of the tasks are encoded at convergence in deep linear networks.
Our framework provides a principled comparison between single-task and multi-task settings, grounded solely in data and task alignment. These results establish a theoretical foundation for understanding task interactions and pave the way toward principled approaches to adaptive loss weighting and task grouping.

Primary Area: learning theory

Submission Number: 8784