Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces

Published: 09 May 2023, Last Modified: 09 May 2023Accepted by TMLREveryoneRevisionsBibTeX
Abstract: Data-driven machine learning models are being increasingly employed in several important inference problems in biology, chemistry, and physics, which require learning over combinatorial spaces. Recent empirical evidence (see, e.g., ~\cite{tseng2020fourier,aghazadeh2021epistatic,ha2021adaptive}) suggests that regularizing the spectral representation of such models improves their generalization power when labeled data is scarce. However, despite these empirical studies, the theoretical underpinning of when and how spectral regularization enables improved generalization is poorly understood. In this paper, we focus on learning pseudo-Boolean functions and demonstrate that regularizing the empirical mean squared error by the $L_1$ norm of the spectral transform of the learned function reshapes the loss landscape and allows for data-frugal learning under a restricted secant condition on the learner's empirical error measured against the ground truth function. Under a weaker quadratic growth condition, we show that stationary points, which also approximately interpolate the training data points achieve statistically optimal generalization performance. Complementing our theory, we empirically demonstrate that running gradient descent on the regularized loss results in a better generalization performance compared to baseline algorithms in several data-scarce real-world problems.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: N/A
Video: https://youtu.be/ER12pwvxTSU
Code: https://github.com/tonytu16/Walsh-Hadamard-Transform
Supplementary Material: zip
Assigned Action Editor: ~Joan_Bruna1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 471