African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification
Abstract: Recent Large Vision-Language Models (LVLMs) demonstrate impressive abilities on numerous image understanding and reasoning tasks.
The task of fine-grained object classification (e.g., distinction between \textit{animal species}), however, has been probed insufficiently, despite its downstream importance.
We fill this evaluation gap by creating FOCI (\textbf{F}ine-grained \textbf{O}bject \textbf{C}lass\textbf{I}fication),
a difficult multiple-choice benchmark for fine-grained object classification, from existing object classification datasets:
(1) multiple-choice avoids ambiguous answers associated with casting classification as open-ended QA task;
(2) we retain classification difficulty by mining negative labels with a CLIP model.
FOCI complements five popular classification datasets with four domain-specific subsets from ImageNet-21k.
We benchmark 12 public LVLMs on \bench and show that it tests for a \textit{complementary skill} to established image understanding and reasoning benchmarks.
Crucially, CLIP models exhibit dramatically better performance than LVLMs. Since the image encoders of LVLMs come from these CLIP models, this points to inadequate alignment for fine-grained object distinction between the encoder and the LLM
and warrants (pre)training data with more fine-grained annotation.
We release our code at \url{ANONYMIZED}.
Paper Type: Long
Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond
Research Area Keywords: vision question answering,cross-modal pretraining
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources
Languages Studied: English
Submission Number: 663
Loading