Generalized Tangent Kernel: A Unified Geometric Foundation for Natural Gradient and Standard Gradient
Abstract: Natural gradients have been widely studied from both theoretical and empirical perspectives, and it is commonly believed that natural gradients have advantages over standard (Euclidean) gradients in capturing the intrinsic geometric structure of the underlying functional space and being invariant under reparameterization. However, under function approximation, a fundamental theoretical issue regarding the existence of natural gradients on the functional space remains underexplored. We address this issue by providing a geometric perspective and mathematical framework for studying both natural gradient and standard gradient that is more complete than existing studies. The key tool that unifies natural gradient and standard gradient is a generalized form of the Neural Tangent Kernel (NTK), which we name the Generalized Tangent Kernel (GTK). Using a novel orthonormality property of GTK, we show that for a fixed parameterization, GTK determines a Riemannian metric on the entire functional space which makes the standard gradient as “natural" as the natural gradient in capturing the intrinsic structure of the parameterized functional space. Many aspects of this approach relate to RKHS theory. For the practical side of this theory paper, we showcase that our framework motivates new solutions to the non-immersion/degenerate case of natural gradient and leads to new families of natural/standard gradient descent methods.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Bamdev_Mishra1
Submission Number: 3449
Loading