Leveraging Pre-trained Models for Kernel Machines

Yawen Chen, Zeyi Wen, Jian Chen, Jin Huang

Abstract: Pre-training techniques have successfully promoted the training of neural networks. Since neural networks and kernel machines share similar properties, such as both learning the problems by the non-linear projection on features and both being capable of handling complex tasks, the idea of pre-training may also help kernel machines achieve promising training speed. However, existing pre-training-based kernel machine solvers show limited improvements on efficiency when the hyper-parameter varies. To effectively reduce the training cost, we propose a novel method that can make efficient use of pre-trained models to infer kernel machine models with different hyper-parameters. Our pre-training-based method is built on top of theoretical foundations. The difference between the model inferred based on pre-training and the optimal model is theoretically bounded by a constant. Experimental results show that our method can save an order of magnitude of training time compared with the existing approach while producing competitive accuracy.