How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation

Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang, Tailin Wu, Yilong Yin, Salman Khan, Lina Yao, Tongliang Liu, Kun Zhang

Keywords: Multimodal Foundation Models, GPT-4V Performance Evaluation, Distribution Shifts, In-context Learning

TL;DR: This paper examines GPT-4V's zero-shot generalization and in-context learning prowess across domain shifts, highlighting its adaptability and the strategic use of in-context examples for enhanced performance.

Abstract: In machine learning, generalization against distribution shifts is crucial, particularly in fields like climate modeling, biomedicine, and autonomous driving. The emergence of foundation models has led to an increased interest in their adaptability to distribution shifts. GPT-4V(ision) acts as one of the most advanced publicly accessible multimodal foundation models, with extensive applications across various domains. However, its robustness against data distributions remains largely underexplored. Addressing this gap, this study rigorously evaluates GPT-4V's adaptability and generalization capabilities in dynamic environments, benchmarking against prominent models like CLIP, LLaVA, and Gemini. We delve into GPT-4V's zero-shot generalization across 13 diverse datasets spanning natural, medical, and molecular domains. We further investigate its adaptability to controlled data perturbations and examine the efficacy of in-context learning as a tool to enhance its adaptation. Our findings delineate GPT-4V's capability boundaries in distribution shifts, shedding light on its strengths and limitations across various scenarios. Importantly, this investigation contributes to our understanding of how AI foundation models generalize to distribution shifts, offering pivotal insights into their adaptability and robustness. Code is publicly available at \url{https://github.com/jameszhou-gl/gpt-4v-distribution-shift}.

Submission Number: 33