Huber-Norm Regularization for Linear Prediction Models

Oleksandr Zadorozhnyi, Gunthard Benecke, Stephan Mandt, Tobias Scheffer, Marius Kloft

Abstract: In order to avoid overfitting, it is common practice to regularize linear prediction models using squared or absolute-value norms of the model parameters. In our article we consider a new method of regularization: Huber-norm regularization imposes a combination of $$\ell _{1}$$ and $$\ell _{2}$$ -norm regularization on the model parameters. We derive the dual optimization problem, prove an upper bound on the statistical risk of the model class by means of the Rademacher complexity and establish a simple type of oracle inequality on the optimality of the decision rule. Empirically, we observe that logistic regression with Huber-norm regularizer outperforms $$\ell _{1}$$ -norm, $$\ell _{2}$$ -norm, and elastic-net regularization for a wide range of benchmark data sets.