Co-Regularization Enhances Knowledge Transfer in High Dimensions

Shuo Shuo Liu, Haotian Lin, Matthew Reimherr, Runze Li

Keywords: High Dimensional Statistics, Transfer Learning, Generalized Linear Models, Statistical Learning

Abstract: Most existing transfer learning algorithms for high-dimensional models employ a two-step regularization framework, whose success heavily hinges on the assumption that the pre-trained model closely resembles the target. To relax this assumption, we propose a co-regularization process to directly exploit beneficial knowledge from the source domain for high-dimensional generalized linear models. The proposed method learns the target parameter by constraining the source parameters to be close to the target one, thereby preventing fine-tuning failures caused by significantly deviated pre-trained parameters. Our theoretical analysis demonstrates that the proposed method accommodates a broader range of sources than existing two-step frameworks, thus being more robust to less similar sources. Its effectiveness is validated through extensive empirical studies.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 23292