On the Sparsity of the Strong Lottery Ticket Hypothesis

Emanuele Natale, Davide Ferre', Giordano Giambartolomei, Frédéric Giroire, Frederik Mallmann-Trenn

Keywords: strong lottery ticket hypothesis, random subset sum, neural network pruning, random neural network

TL;DR: We provide the first proof of the SLTH in classical settings with guarantees on the sparsity of the subnetworks.

Abstract:

Considerable research efforts have recently been made to show that a random neural network $N$ contains subnetworks capable of accurately approximating any given neural network that is sufficiently smaller than $N$, without any training.
This line of research, known as the Strong Lottery Ticket Hypothesis (SLTH), was originally motivated by the weaker Lottery Ticket Hypothesis, which states that a sufficiently large random neural network $N$ contains sparse subnetworks that can be trained efficiently to achieve performance comparable to that of training the entire network $N$.
Despite its original motivation, results on the SLTH have so far not provided any guarantee on the size of subnetworks.
Such limitation is due to the nature of the main technical tool leveraged by these results, the Random Subset Sum (RSS) Problem.
Informally, the RSS Problem asks how large a random i.i.d. sample $\Omega$ should be so that we are able to approximate any number in $[-1,1]$, up to an error of $ \epsilon$, as the sum of a suitable subset of $\Omega$.

We provide the first proof of the SLTH in classical settings, such as dense and equivariant networks, with guarantees on the sparsity of the subnetworks. Central to our results, is the proof of an essentially tight bound on the Random Fixed-Size Subset Sum Problem (RFSS), a variant of the RSS Problem in which we only ask for subsets of a given size, which is of independent interest.

Primary Area: Learning theory

Submission Number: 19255