PAC-Bayes Bounds for Multivariate Linear Regression and Linear Autoencoders

Ruixin Guo, Ruoming Jin, Xinyu Li, Yang Zhou

Abstract: Linear Autoencoders (LAEs) have shown strong performance in state-of-the-art recommender systems. However, these impressive results are mainly based on experiments, with little theoretical support. This paper investigates the generalizability -- a theoretical measure of model performance in statistical machine learning -- of multivariate linear regression and LAEs. We first propose a PAC-Bayes bound for multivariate linear regression, which is generalized from an earlier PAC-Bayes bound for multiple linear regression by (Shalaeva et al., 2020), and outline sufficient conditions that ensure its theoretical convergence. We then apply this bound to LAEs by showing that LAEs can be viewed as constrained multivariate linear regression on bounded data, and develop practical methods for minimizing the bound, addressing the calculation challenges posed by the constraints. Experimental results demonstrates the non-vacuousness of our bound for LAEs.

Primary Area: Theory->Learning Theory

Keywords: PAC-Bayes bound, multivariate linear regression, linear autoencoder, recommender system

Submission Number: 7694