A Kernel Perspective for the Decision Boundary of Deep Neural Networks

Yifan Zhang

Abstract: Deep learning has achieved great success in many fields, but they still lack theoretical understandings. Although some recent theoretical and experimental results have investigated the representation power of deep learning, little effort has been devoted to analyzing the generalization ability of deep learning. In this paper, we analyze deep neural networks from a kernel perspective and use kernel methods to investigate the effect of the implicit regularization introduced by gradient descent on the generalization ability. Firstly, we argue that the multi-layer nonlinear feature transformation in deep neural networks is equivalent to a kernel feature mapping and analyze our point from the perspective of the unique mathematical advantages of kernel methods and the method of constructing multi-layer kernel machines, respectively. Secondly, using the representer theorem, we analyze the decision boundary of deep neural networks and prove that the last hidden layers of deep neural networks converge to nonlinear SVMs. Systematical experiments demonstrate that the decision boundaries of neural networks converge to those of nonlinear SVMs.