everyone
since 04 Oct 2024">EveryoneRevisionsBibTeXCC BY 4.0
This paper delves into the interplay between vision backbones and optimizers, revealing an inter-dependent phenomenon termed backbone-optimizer coupling bias} (BOCB). Notably, canonical CNNs, such as VGG and ResNet, exhibit a marked co-dependency with SGD, while recent architectures, including ViTs and ConvNeXt, share a strong coupling with adaptive learning rate optimizers. We further show that strong BOCB may result in extra tuning efforts and poor generalization ability for pre-trained neural networks, substantially limiting their real-world applications. Through in-depth analysis and apples-to-apples comparisons, however, we surprisingly observed that certain types of network architecture could significantly mitigate BOCB, which might serve as practical takeaways for backbone design. We hope this work can inspire the community to rethink the long-held assumptions on backbones and optimizers, consider their interplay in future studies, and contribute to more robust vision systems. The source code and models are publicly available.