Model-Efficient Deep Learning with Kernelized Classification

Sadeep Jayasumana, Srikumar Ramalingam, Sanjiv Kumar

Keywords: deep networks, kernels on the sphere, nonlinear classification

Abstract: We investigate the possibility of using the embeddings produced by a lightweight network more effectively with a nonlinear classification layer. Although conventional deep networks use an abundance of nonlinearity for representation (embedding) learning, they almost universally use a linear classifier on the learned embeddings. This is suboptimal since better nonlinear classifiers could exist in the same embedding vector space. We advocate a nonlinear kernelized classification layer for deep networks to tackle this problem. We theoretically show that our classification layer optimizes over all possible kernel functions on the space of embeddings to learn an optimal nonlinear classifier. We then demonstrate the usefulness of this layer in learning more model-efficient classifiers in a number of computer vision and natural language processing tasks.

One-sentence Summary: We propose a kernelized classification layer for deep networks to obtain the best possible classifier with embeddings learned in the network.