Agnostic Membership Query Learning with Nontrivial Savings: New Results and Techniques

Ari Karchmer

Abstract: Designing computationally efficient algorithms in the agnostic learning model (Haussler, 1992; Kearns et al., 1994) is notoriously difficult. In this work, we consider agnostic learning with membership queries for touchstone classes at the frontier of agnostic learning, with a focus on how much computation can be saved over the trivial run-time of $2^n$. This approach is inspired by and continues the study of “learning with nontrivial savings” (Servedio and Tan, 2017). To this end, we establish multiple agnostic learning algorithms, highlighted by: An agnostic learning algorithm for circuits consisting of a sublinear number of gates, which can each be any function computable by a sublogarithmic degree $k$ polynomial threshold function (the depth of the circuit is bounded only by size). This algorithm runs in time $2^{n -s(n)}$ for $s(n) \approx n/(k+1)$, and learns over the uniform distribution over unlabelled examples on $\{0,1\}^n$. An agnostic learning algorithm for circuits consisting of a sublinear number of gates, where each can be any function computable by a $\sym^+$ circuit of subexponential size and sublogarithmic degree $k$. This algorithm runs in time $2^{n-s(n)}$ for $s(n) \approx n/(k+1)$, and learns over distributions of unlabelled examples that are products of $k+1$ \textit{arbitrary and unknown} distributions, each over $\{0,1\}^{n/(k+1)}$ (assume without loss of generality that $k+1$ divides $n$). Furthermore, we apply our new agnostic learning algorithms for these classes to also obtain algorithms for randomized compression, exact learning with membership and equivalence queries, and distribution-independent PAC-learning with membership queries. Our core technique, which may be of independent interest, remixes the learning from natural proofs paradigm (Carmosino et al. 2016, 2017), so that we can tolerate concept classes fundamentally different than $\AC^0[p]$, and achieve fully agnostic learning. We make use of communication-complexity-based natural proofs (Nisan, 1993), rather than natural proofs of Razborov (1987) and Smolensky (1987) for $\AC^0[p]$.