Sparsification of core set models in non-metric supervised learning

Frank-Michael Schleif

Abstract:

Supervised learning employing positive semi definite kernels has gained wide attraction and lead to a
variety of successful machine learning approaches. The restriction to positive semi definite kernels and
a hilbert space is common to simplify the mathematical derivations of the respective learning methods,
but is also limiting because more recent research indicates that non-metric, and therefore non positive
semi definite, data representations are often more effective. This challenge is addressed by multiple approaches and recently dedicated algorithms for so called indefinite learning have been proposed. Along
this line, the Krein˘ space Support Vector Machine (KSVM) and variants are very efficient classifiers for
indefinite learning problems, but with a non-sparse decision function. This very dense decision function
prevents practical applications due to a costly out of sample extension. We focus on this problem and
provide two post processing techniques to sparsify models as obtained by a Krein˘ space SVM approach.
In particular we consider the indefinite Core Vector Machine and indefinite Core Vector Regression Machine which are both efficient for psd kernels, but suffer from the same dense decision function, if the
Krein˘ space approach is used. We evaluate the influence of different levels of sparsity and employ a Nyström approach to address large scale problems. Experiments show that our algorithm is similar efficient
as the non-sparse Krein˘ space Support Vector Machine but with substantially lower costs, such that also
problems of larger scale can be processed.