Analytic Optimization of Shrinkage Parameters Based on Regularized Subspace Information Criterion

Masashi Sugiyama, Keisuke Sakurai

Abstract: For obtaining a higher level of generalization capability in supervised learning, model parameters should be optimized, i.e., they should be determined in such a way that the generalization error is minimized. However, since the generalization error is inaccessible in practice, model parameters are usually determined in such a way that an estimate of the generalization error is minimized. A standard procedure for model parameter optimization is to first prepare a finite set of candidates of model parameter values, estimate the generalization error for each candidate, and then choose the best one from the candidates. If the number of candidates is increased in this procedure, the optimization quality may be improved. However, this in turn increases the computational cost. In this paper, we give methods for <I>analytically</I> finding the optimal model parameter value from a set of <I>infinitely</I> many candidates. This maximally enhances the optimization quality while the computational cost is kept reasonable.