Scalable Diverse Model Selection for Accessible Transfer Learning

Daniel Bolya, Rohit Mittapalli, Judy Hoffman

Keywords: Computer Vision, Transfer Learning, Model Selection, Transferability Estimation, Benchmark, Accessibility

TL;DR: We design a benchmark that tests diverse model selection and show that current methods fail on this benchmark. We then design a method that works well in this setting.

Abstract: With the preponderance of pretrained deep learning models available off-the-shelf from model banks today, finding the best weights to fine-tune to your use-case can be a daunting task. Several methods have recently been proposed to find good models for transfer learning, but they either don't scale well to large model banks or don't perform well on the diversity of off-the-shelf models. Ideally the question we want to answer is, "given some data and a source model, can you quickly predict the model's accuracy after fine-tuning?" In this paper, we formalize this setting as "Scalable Diverse Model Selection" and propose several benchmarks for evaluating on this task. We find that existing model selection and transferability estimation methods perform poorly here and analyze why this is the case. We then introduce simple techniques to improve the performance and speed of these algorithms. Finally, we iterate on existing methods to create PARC, which outperforms all other methods on diverse model selection. We have released the benchmarks and method code in hope to inspire future work in model selection for accessible transfer learning.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/dbolya/parc

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2111.06977/code)