Reactivation: Empirical NTK Dynamics Under Task Shifts

Yuzhi LIU, Zixuan Chen, Zirui zhang, Yufei Liu, Giulia Lanzillotta

Keywords: Continual Learning, Neural Tangent Kernels, Deep learning, Learning Dynamics

TL;DR: This work provides a systematic, empirical investigation of Neural Tangent Kernel (NTK) dynamics in the context of continual learning—a setting that challenges the conventional assumption of stationary data distributions.

Abstract:

The Neural Tangent Kernel (NTK) offers a powerful tool to study the functional dynamics of neural networks. In the so-called lazy, or kernel regime, the NTK remains static during training and the network function is linear in the static neural tangents feature space.
The evolution of the NTK during training is necessary for feature learning, a key ingredient of the success of deep learning. The study of the dynamics of the NTK has led to several critical discoveries in recent years, in generalization and scaling behaviours. However, this body of work has been limited to the single task setting, where the data distribution is assumed constant over time. In this work, we present a comprehensive empirical analysis of NTK dynamics in continual learning, where the data distribution shifts over time. Our findings highlight continual learning as a rich and underutilized testbed for probing the dynamics of neural training. At the same time, they challenge the validity of static-kernel approximations in theoretical treatments of continual learning, even at large scale.

Student Paper: Yes

Submission Number: 102