Universal Ordering for Efficient PAC Learning

ICLR 2026 Conference Submission20981 Authors

Keywords: learning theory, data ordering, PAC learning

Abstract:

We initiate the study of the \emph{universal ordering} problem within the PAC learning framework: given a set of $n$ samples independently drawn from an unknown distribution $\mathcal{D}$, can we order these samples such that every prefix of length $k \le n$ yields a near-optimal subset for training a PAC learner?
This question is fundamentally motivated by practical scenarios involving incremental learning and adaptive computation, where guarantees must hold uniformly across varying data budgets.
We formalize this requirement as achieving anytime-valid PAC guarantees.
As a warm-up, we analyze the simple random ordering baseline using classical concentration inequalities.
Through a careful union bound over a geometric partitioning of prefixes, we establish that it provides a surprisingly strong universal guarantee, incurring at most an $O(\log\log n)$ overhead compared to a random subset of size $k$.
We then present a more powerful analysis based on the theory of test martingales and Ville's inequality, demonstrating that a random permutation achieves PAC guarantees for all prefixes that match the statistical rate of a random subset of size $k$, without the logarithmic overhead incurred by naive union-bound techniques.
Our work establishes a conceptual bridge between universal learning on fixed datasets and the broader field of sequential analysis, revealing that random permutations are efficient and provably robust anytime-valid learners but opening the door to further improvements.

Primary Area: learning theory

Submission Number: 20981