One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

Varun Gohil, S. Deepak Narayanan, Atishay Jain

Abstract: The lottery ticket hypothesis states that smaller subnetworks within a larger deep network can be trained in isolation to achieve accuracy similar to that of original network, as long as they are initialized appropriately. However, whether these subnetworks or winning tickets are transferable across datasets and optimizers remains unclear. The paper "One ticket to win them all:generalizing lottery ticket initializations across datasets and optimizers" empirically shows that these winning tickets are transferable. We reproduce the results in the paper from scratch by implementing all the experiments. Our results support the original paper's claim of the winning ticket initializations being transferable. While the paper is replicable, we find that reproducing the paper requires access to large amount of computing resources for generating the winning tickets. Hence, along with the code base, we also open-source the winning tickets we find, so others can avoid the compute-intensive procedure of generating them.

Track: Replicability

NeurIPS Paper Id: https://openreview.net/forum?id=HJeUKEBlLS