On Empirical Risk Minimization with Dependent and Heavy-Tailed Data

Abhishek Roy, Krishna Balasubramanian, Murat A Erdogdu

Keywords: non-iid learning, risk bounds, empirical risk minimization, concentration inequalities, small-ball method

TL;DR: This paper develops risk bounds for empirical risk minimization with (polynomially) heavy-tailed and strictly stationary exponentially $\beta$-mixing data generating model.

Abstract: In this work, we establish risk bounds for Empirical Risk Minimization (ERM) with both dependent and heavy-tailed data-generating processes. We do so by extending the seminal works~\cite{pmlr-v35-mendelson14, mendelson2018learning} on the analysis of ERM with heavy-tailed but independent and identically distributed observations, to the strictly stationary exponentially $\beta$-mixing case. We allow for the interaction between the noise and inputs to be even polynomially heavy-tailed, which covers a significantly large class of heavy-tailed models beyond what is analyzed in the learning theory literature. We illustrate our theoretical results by obtaining rates of convergence for high-dimensional linear regression with dependent and heavy-tailed data.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.