Perceptual Structure in the absence of grounding: the impact of abstractedness and subjectivity in color language for LLMs
Submission Type: Regular Short Paper
Submission Track: Language Grounding to Vision, Robotics and Beyond
Submission Track 2: Language Modeling and Analysis of Language Models
Keywords: Color language, grounding, language models
TL;DR: Exploring alignment between color and language spaces in the presence of abstractedness and subjectivity
Abstract: The need for grounding in language understanding is an active research topic. Previous work has suggested that color perception and color language appear as a suitable test bed to empirically study the problem, given its cognitive significance and showing that there is considerable alignment between a defined color space and the feature space defined by a language model. To further study this issue, we collect a large scale source of colors and their descriptions, containing almost a 1 million examples
, and perform an empirical analysis to compare two kinds of alignments: (i) inter-space, by learning a mapping between embedding space and color space, and (ii) intra-space, by means of prompting comparatives between color descriptions. Our results show that while color space alignment holds for monolexemic, highly pragmatic color descriptions, this alignment drops considerably in the presence of examples that exhibit elements of real linguistic usage such as subjectivity and abstractedness, suggesting that grounding may be required in such cases.
Submission Number: 5199
Loading