An Adaptive Tangent Feature Perspective of Neural Networks
Keywords: adaptive, kernel learning, tangent kernel, neural networks, low rank
TL;DR: Adaptive feature models with neural network structure impose approximate low rank regularization on weight matrices
Abstract: In order to better understand feature learning in neural networks, we propose and study linear models in tangent feature space where the features are allowed to be transformed during training. We consider linear feature transformations, resulting in a joint optimization over parameters and transformations with a bilinear interpolation constraint. We show that this relaxed optimization problem has an equivalent linearly constrained optimization with structured regularization that encourages approximately low rank solutions. Specializing to structures arising in neural networks, we gain insights into how the features and thus the kernel function change, providing additional nuance to the phenomenon of kernel alignment when the target function is poorly represented by tangent features. In addition to verifying our theoretical observations in real neural networks on a simple regression problem, we empirically show that an adaptive feature implementation of tangent feature classification has an order of magnitude lower sample complexity than the fixed tangent feature model on MNIST and CIFAR-10.
Track Confirmation: Yes, I am submitting to the proceeding track.
Submission Number: 32
Loading