Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, Kentaro Inui

Keywords: ensemble, pretrained model, lottery ticket hypothesis, subnetwork

TL;DR: Ensembling subnetworks can outperform the dense ensemble even using only a single pretrained model if we can find winning-ticket subnetworks.

Abstract: Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks and their ensemble outperformed the standard ensemble in some tasks when accurate lottery tickets are found on the tasks.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2205.11833/code)