On PAC-Bayes Bounds for Linear Autoencoders

Ruixin Guo, Ruoming Jin, Xinyu Li, Yang Zhou

Keywords: PAC-Bayes bound, linear regression, linear autoencoder, recommender system

Abstract: Linear Autoencoders (LAEs) have shown strong performance in state-of-the-art recommender systems. Some LAE models, like EASE, can be viewed as multivariate (multiple-output) linear regression models with a zero-diagonal constraint. However, these impressive results are mainly based on experiments, with little theoretical support. This paper investigates the generalizability -- a theoretical measure of model performance in statistical machine learning -- of multivariate linear regression and LAEs. We first propose a PAC-Bayes bound for multivariate linear regression, which is generalized from an earlier PAC-Bayes bound for single-output linear regression by Shalaeva et al., and outline sufficient conditions that ensure its theoretical convergence. We then apply this bound to EASE, a classic LAE model in recommender systems, and develop a practical method for minimizing the bound, addressing the calculation challenges posed by the zero-diagonal constraint. Experimental results show that our bound for EASE is non-vacuous on real-world datasets, demonstrating its practical utility.

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8609