Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel’s Spectrum

Amnon Geifman, Daniel Barzilai, Ronen Basri, Meirav Galun

Abstract: Wide neural networks are biased towards learning certain functions, influencing both the rate of convergence of gradient descent (GD) and the functions that are reachable with GD in finite training time. As such, there is a great need for methods that can modify this bias according to the task at hand. To that end, we introduce Modified Spectrum Kernels (MSKs), a novel family of constructed kernels that can be used to approximate kernels with desired eigenvalues for which no closed form is known. We leverage the duality between wide neural networks and Neural Tangent Kernels and propose a preconditioned gradient descent method, which alters the trajectory of GD. As a result, this allows for a polynomial and, in some cases, exponential training speedup without changing the final solution. Our method is both computationally efficient and simple to implement.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Russell_Tsuchida1

Submission Number: 1479