Ranking Neural Checkpoints

YANDONG LI, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, Boqing Gong

Keywords: checkpoint selection, transfer learning, task transferability, network generalization prediction

Abstract: This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task. Thanks to the broad use of DNNs, we may easily collect hundreds of checkpoints from various sources. Which of them transfers the best to our downstream task of interest? Striving to answer this question thoroughly, we establish a neural checkpoint ranking benchmark (\benchmark) and study some intuitive ranking measures. These measures are generic, applying to the checkpoints of different output types without knowing how the checkpoints are pre-trained on which dataset. They also incur low computation costs, making them practically meaningful. Our results suggest that the linear separability of the features extracted by the checkpoints is a strong indicator of transferability. We also arrive at a new ranking measure, $\mathcal{N}$LEEP, which gives rise to the best performance in the experiments.

One-sentence Summary: A comprehensive study on a novel research topic: "ranking neural checkpoints"

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2011.11200/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=dOktJmTNdX