Nonlinear regularization path for the modified Huber loss Support Vector Machines

Masayuki Karasuyama, Ichiro Takeuchi

Abstract: Regularization path algorithms have been proposed to deal with model selection problem in several machine learning approaches. These algorithms allow to compute the entire path of solutions for every value of regularization parameter using the fact that their solution paths have piecewise linear form. In this paper, we propose nonlinear regularization path for the Support Vector Machine (SVM) with a modified Huber loss. We first show that the solution path of the modified Huber loss SVM is represented as piecewise nonlinear function. Since the solutions between two breakpoints are characterized by a rational function, the breakpoint itself can be identified solving the rational equations. Then we develop an efficient iterative algorithm to solve these rational equations with quadratic convergence rate. Note that our algorithm is NOT a predictor-corrector type method that can only follow nonlinear regularization path with rough approximation. We show the algorithm performance on some artificial and real data sets