Transferring Lottery Tickets in Computer Vision Models: a Dynamic Pruning Approach

Qisheng He, Ming Dong

Abstract: Deep neural networks can achieve state-of-the-art results on small size datasets by transferring the backbone from a network pre-trained on large datasets. Recent work has shown that pruned networks can also be used as pre-trained models in transfer learning. In this paper, we proposed a novel framework, Transferring Lottery Ticket (TLT), to adapt both masks and weights of a pre-trained and pruned network dynamically during the knowledge transfer to downstream tasks. We show that the lottery tickets of downstream tasks are dramatically different from each other and from the one obtained from the pre-trained network. Thus, both masks and weights need to be learned to better adapt a pre-trained model to the target domain. Our extensive experiments on multiple computer vision tasks, such as image classification and segmentation, show that the transferred networks with adapted masks outperform the ones with original masks at various pruning ratios.