Private Overparameterized Linear Regression without Suffering in High Dimensions
Primary Area: societal considerations including fairness, safety, privacy
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Differential Privacy, Overparameterization, Linear Regression, Optimization
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: This study focuses on differentially private linear regression in the over-parameterized regime. We propose a new variant of the differentially private Follow-The-Regularized-Leader (DP-FTRL) algorithm that uses a random noise with a general covariance matrix for differential privacy. This leads to improved privacy and utility (excess risk) trade-offs. Firstly, even when reduced to an existing DP-FTRL algorithm that uses an isotropic noise, our excess risk bound is sharper as a function of the eigenspectrum of the data covariance matrix and the ground truth model parameter. Furthermore, when unlabeled public data is available, we can design a better noise covariance matrix structure to improve the utility. For example, when the ground truth has a bounded $\ell_2$-norm, and the eigenspectrum decays polynomially (i.e., $\lambda_i=i^{-r}$ for $r>1$), our method achieves $\mathcal{\tilde O}(N^{-\frac{r}{1+2r}})$ and $\mathcal{\tilde O}(N^{-\frac{r}{3+r}\wedge\frac{2r}{1+3r}})$ excess error for identity and specially designed covariance matrices, respectively. Notably, our method with a specially designed covariance matrix outperforms the one with an identity matrix when the eigenspectrum decays at least quadratically fast, i.e., $r\geq 2$. Our proposed method significantly improves upon existing differentially private methods for linear regression, which tend to scale with the problem dimension, leading to a vacuous guarantee in the over-parameterized regime.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7718
Loading