Model Zoos for Benchmarking Phase Transitions in Neural Networks

Konstantin Schürholt, Léo Meynent, Yefan Zhou, Yaoqing Yang, Damian Borth

Keywords: Phase Transition, Model Zoo, Population, Evaluation

TL;DR: We propose to evalute ML methods systematically for phase transitions, and present a dataset of pre-trained models to faciliate the identification of phase distributions

Abstract:

Understanding the complex dynamics of neural network training remains a central challenge in deep learning research. Work rooted in statistical physics has identified phases and phase transitions in neural network (NN) models, where models within the same phase exhibit similar characteristics but qualitatively differ across phases. A prominent example is the double-descent phenomenon. Recognizing these transitions is essential for building a deeper understanding of model behavior and the underlying mechanics. So far, these phases are typically studied in isolation or in specific applications. In this paper, we show that phase transitions are a widespread phenomenon. However, identifying phase transitions across different methods requires populations that cover different phases. For that reason, we introduce Phase Transition Model Zoos, a structured collection of neural networks trained on diverse datasets and architectures. These model zoos are carefully designed to help researchers systematically identify and study phase transitions in their methods. We demonstrate the relevance of phase transitions across multiple applications, including fine-tuning, transfer learning, out-of-distribution generalization, pruning, ensembling, and weight averaging. The diversity of applications underscores the universal nature of phase transitions and their impact on different tasks. By providing the first structured dataset specifically designed to capture phase transitions in NNs, we offer a valuable tool for the community to systematically evaluate machine learning methods and improve their understanding of phase behavior across a wide range of applications and architectures.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11531