Tuning regularization via scenario optimization

Simone Formentin, Simone Garatti, Marco C. Campi, Sergio M. Savaresi

Abstract: In learning problems, avoiding to overfit the training data is of fundamental importance in order to achieve good predictive capabilities. Regularization networks have shown to be an effective tool to find reliable models, however their tuning is all but straightforward. In this paper, we consider learning problems that can be formulated as random convex minimization programs, and leverage on recent results established within the Wait & Judge theory for scenario optimization. Our main result is that, within this framework, generalization is deeply connected to the number of so-called support points found in optimization. By suitably selecting the regularization parameter, one can adjust the support points set and thereby can tune the trade-off between performance and generalization of the solution on the ground of a rigorous and quantitative theory.