Large-Scale Maximum Margin Discriminant Analysis Using Core Vector Machines

Ivor Wai-Hung Tsang, András Kocsor, James Tin-Yau Kwok

Abstract: para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> Large-margin methods, such as support vector machines (SVMs), have been very successful in classification problems. Recently, maximum margin discriminant analysis (MMDA) was proposed that extends the large-margin idea to feature extraction. It often outperforms traditional methods such as kernel principal component analysis (KPCA) and kernel Fisher discriminant analysis (KFD). However, as in the SVM, its time complexity is cubic in the number of training points <formula formulatype="inline"><tex>$m$</tex></formula>, and is thus computationally inefficient on massive data sets. In this paper, we propose an <formula formulatype="inline"> <tex>$(1+\epsilon)^2$</tex></formula>-approximation algorithm for obtaining the MMDA features by extending the core vector machine. The resultant time complexity is only linear in <formula formulatype="inline"><tex>$m$</tex> </formula>, while its space complexity is independent of <formula formulatype="inline"> <tex>$m$</tex></formula>. Extensive comparisons with the original MMDA, KPCA, and KFD on a number of large data sets show that the proposed feature extractor can improve classification accuracy, and is also faster than these kernel-based methods by over an order of magnitude. </para>