Batch List-Decodable Linear Regression via Higher Moments

Ilias Diakonikolas, Daniel Kane, Sushrut Karmalkar, Sihan Liu, Thanasis Pittas

TL;DR: This paper designs an efficient algorithm based on the sum-of-square algorithm for list-decodable linear regression with batched samples.

Abstract:

We study the task of list-decodable linear regression using batches, recently introduced by Das et al. 2023..
In this setting, we are given $m$ batches with each batch containing
$n$ points in $\mathbb R^d$. A batch is called clean if the points it contains are i.i.d. samples from an unknown linear regression distribution.
For a parameter $\alpha \in (0, 1/2)$, an unknown $\alpha$-fraction
of the batches are clean and no assumptions are made on the remaining batches.
The goal is to output a small list of vectors at least one of which is close to the true regressor vector in $\ell_2$-norm. Das et al. 2023 gave an efficient algorithm for this task, under natural distributional assumptions,
with the following guarantee.
Under the assumption that the batch size satisfies
$n \geq \tilde{\Omega}(\alpha^{-1})$ and the total number of batches
is $m = \text{poly}(d, n, 1/\alpha)$,
their algorithm runs in polynomial time and
outputs a list of $O(1/\alpha^2)$ vectors at least one of which
is $\tilde{O}(\alpha^{-1/2}/\sqrt{n})$ close to the target regressor.
Here we design a new polynomial-time algorithm
for this task with significantly stronger guarantees under the assumption that the low-degree moments of the covariates distribution are
Sum-of-Squares (SoS) certifiably bounded.
Specifically, for any constant $\delta>0$, as long as the batch size is
$n \geq \Omega_{\delta}(\alpha^{-\delta})$
and the degree-$\Theta(1/\delta)$ moments of the covariates are SoS certifiably bounded,
our algorithm uses $m = \text{poly}((dn)^{1/\delta}, 1/\alpha)$ batches,
runs in polynomial-time, and outputs an $O(1/\alpha)$-sized list of vectors one of which is
$O(\alpha^{-\delta/2}/\sqrt{n})$ close to the target. That is, our algorithm substantially
improves both the minimum batch size and the final error guarantee, while achieving the optimal list size.
Our approach leverages higher-order moment information by carefully combining the SoS paradigm interleaved with an iterative method and a novel list pruning procedure for this setting.
In the process, we give an SoS proof of the Marcinkiewicz-Zygmund inequality that may be of broader applicability.

Lay Summary: This paper extends the algorithm toolbox for statistical tasks in crowdsource settings.

Primary Area: Theory->Learning Theory

Keywords: Robust Statistics;Linear Regression;Sum of Square

Submission Number: 12676