On Transfer Learning via Linearized Neural Networks

Wesley J. Maddox, Shuai Tang, Pablo Moreno, Andrew Gordon Wilson, Andreas Damianou

Abstract: We propose to linearize neural networks for transfer learning via a first order Taylor approximation. Making neural networks linear in this way allows the optimization to become convex (or even closed form) across several tasks. Not only does this vastly simplify the problem, but it allows us to rephrase transfer learning as sharing hyper-parameters across Gaussian processes, which can be solved using standard numerical linear algebra methods. Probabilistically, the framework is interpreted as a Gaussian process model with finite Neural Tangent Kernels. Our approach is fast not only thanks to the linearization, but also because we leverage numerical results from relating the Fisher Information Matrix to the NTK.