Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization

Antonio H. Ribeiro, David Vävinggren, Dave Zachariah, Thomas B. Schön, Francis Bach

Keywords: Kernel learning; adversarial training; generalization bounds

TL;DR: We introduce a computationally efficient kernel learning method that employs adversarial feature perturbations instead of input perturbations, achieving better optimization, adaptive regularization, and strong theoretical and empirical results.

Abstract: Adversarial training has emerged as a key technique to enhance model robustness against adversarial input perturbations. Many of the existing methods rely on computationally expensive min-max problems that limit their application in practice. We propose a novel formulation of adversarial training in reproducing kernel Hilbert spaces, shifting from input to feature-space perturbations. This reformulation enables the exact solution of inner maximization and efficient optimization. It also provides a regularized estimator that naturally adapts to the noise level and the smoothness of the underlying function. We establish conditions under which the feature-perturbed formulation is a relaxation of the original problem and propose an efficient optimization algorithm based on iterative kernel ridge regression. We provide generalization bounds that help to understand the properties of the method. We also extend the formulation to multiple kernel learning. Empirical evaluation shows good performance in both clean and adversarial settings.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 11708