A spectral least-squares-type method for heavy-tailed corrupted regression with unknown covariance \& heterogeneous noise

Roberto Imbuzeiro Oliveira, Zoraida R. Rico, Philip Thompson

Abstract:

We revisit heavy-tailed corrupted least-squares linear regression assuming to have a corrupted $n$-sized label-feature sample of at most $\epsilon n$ arbitrary outliers. We wish to estimate $\bb^*\in\re^p$ given such sample of a label-feature pair $(y,\bx)\in\re\times\re^p$ satisfying
$
y=\langle\bx,\bb^*\rangle+\xi,
$
with heavy-tailed $(\bx,\xi)$. We only assume $\bx$ is $L^4-L^2$ hypercontractive with constant $L>0$ and has covariance matrix $\bfSigma$ with minimum eigenvalue
$\nicefrac{1}{\mu^2(\mbB_2)}>0$ and bounded condition number $\kappa>0$. The noise $\xi\in\re$ can be arbitrarily dependent on $\bx$ and nonsymmetric as long as $\xi\bx$ has finite covariance matrix $\bfXi$. We propose a near-optimal computationally tractable estimator, based on the power method, assuming no knowledge on
$(\bfSigma,\bfXi)$ nor the operator norm of $\bfXi$. With probability at least $1-\delta$, our proposed estimator attains the statistical rate
$
\mu^2(\mbB_2)\Vert\bfXi\Vert^{1/2}(\frac{p}{n}+\frac{\log(1/\delta)}{n}+\epsilon)^{1/2}
$
and breakdown-point
$
\epsilon\lesssim\frac{1}{L^4\kappa^2},
$
both optimal in the $\ell_2$-norm, assuming the near-optimal minimum sample size
$
L^4\kappa^2(p\log p + \log(1/\delta))\lesssim n
$, up to a log factor. To the best of our knowledge, this is the first computationally tractable algorithm satisfying simultaneously all the mentioned properties. Our estimator is based on a two-stage Multiplicative Weight Update algorithm. The first stage estimates a descent direction $\hat\bv$ with respect to the (unknown) pre-conditioned inner product $\langle\bfSigma(\cdot),\cdot\rangle$. The second stage estimate the descent direction $\bfSigma\hat\bv$ with respect to the (known) inner product
$\langle\cdot,\cdot\rangle$, without knowing nor estimating $\bfSigma$.