Color Names in Vision-Language Models

ICLR 2026 Conference Submission348 Authors

Keywords: color, color naming, visual language models

TL;DR: Vision-language models excel at naming obvious colors but stumble on subtle hues, while showing severe bias toward English and Chinese and changing their color descriptions based on what object they're looking at.

Abstract: Color serves as a fundamental dimension of human visual perception and a primary means of communicating about objects and scenes. As vision-language models (VLMs) become increasingly prevalent, understanding whether they name colors like humans is crucial for effective human-AI interaction. We present the first systematic evaluation of color naming capabilities across VLMs, replicating classic color naming methodologies using 957 color samples across five representative models. Our results show that while VLMs achieve high accuracy on prototypical colors from classical studies, performance drops significantly on expanded, non-prototypical color sets. We identify 21 common color terms that consistently emerge across all models, revealing two distinct approaches: constrained models using predominantly basic terms versus expansive models employing systematic lightness modifiers. Cross-linguistic analysis across nine languages demonstrates severe training imbalances favoring English and Chinese, with hue serving as the primary driver of color naming decisions. Finally, ablation studies reveal that language model architecture significantly influences color naming independent of visual processing capabilities.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 348