Correspondences between word learning in children and captioning models

Sunayana Rane, Mira L Nencheva, Zeyu Wang, Casey Lew-Williams, Olga Russakovsky, Thomas L. Griffiths

Keywords: cognitive science, child development, language, image captioning, computer vision

TL;DR: We show that image captioning systems' performance correlates with the age at which children acquire words from a variety of word categories.

Abstract: For human children as well as machine learning systems, a key challenge in learning a word is linking the word to the visual phenomena it describes. By organizing model output into word categories used to analyze child language learning data, we show a correspondence between word learning in children and the performance of image captioning models. Although captioning models are trained only on standard machine learning data, we find that their performance in producing words from a variety of word categories correlates with the age at which children acquire words from each of those categories. To explain why this correspondence exists, we show that the performance of captioning models is correlated with human judgments of the concreteness of words, suggesting that these models are capturing the complex real-world association between words and visual phenomena.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Neuroscience and Cognitive Science (e.g., neural coding, brain-computer interfaces)