Revised Optimizer of SVR Hyperparameters Minimizing Cross-Validation Error

Masayuki Karasuyama, Daisuke Kitakoshi, Ryohei Nakano

Abstract: The performance of support vector regression (SVR) deeply depends on its hyperparameters such as an insensitive zone thickness epsiv, a penalty factor C, and RBF kernel parameter sigma. A method called MCV-SVR was once proposed, which optimizes SVR hyperparameters so that a cross-validation error is minimized. However, as pointed out in this paper, the MCV-SVR (or its variants) has numerical instability in gradient calculation, which may cause bad influence on performance. Thus, this paper introduces a new method of computing the gradient of parameters with respect to hyperparameters. The revised optimizers incorporating the new method is shown to be free from the instability problem. Our experiments using three data sets showed that the revised optimizers considerably improved generalization performance of the MCV-SVR or its variant, and outperformed other methods such as multi-layer perceptrons or SVR with practical setting of hyperparameters.