Can Language Models Understand Physical Concepts?

Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Xu Sun, Lingpeng Kong, Qi Liu

Submission Type: Regular Long Paper

Submission Track: Language Grounding to Vision, Robotics and Beyond

Submission Track 2: Interpretability, Interactivity, and Analysis of Models for NLP

Keywords: large language models, embodied concept understanding

Abstract:

Language models (LMs) gradually become general-purpose interfaces in the interactive and embodied world, where the understanding of physical concepts is an essential prerequisite.
However, it is unclear whether LMs can understand physical concepts in the human world.
To investigate this, we design a benchmark VEC that covers the tasks of (i) Visual concepts, such as the shape and material of objects, and (ii) Embodied Concepts, learned from the interaction with the world such as the temperature of objects.
Our zero (few)-shot prompting results show that
the understanding of certain visual concepts
emerges as scaling up LMs, but there are still
basic concepts to which the scaling law does not apply.
For example, OPT-175B performs close to humans with a zero-shot accuracy of $85$\% on the material concept, yet behaves like random guessing on the mass concept.
Instead, vision-augmented LMs such as CLIP and BLIP achieve a human-level understanding of embodied concepts.
Analysis indicates that the rich semantics in visual representation can serve as a valuable source of embodied knowledge. Inspired by this, we propose a distillation method to transfer embodied knowledge from VLMs to LMs, achieving performance gain comparable with that by scaling up parameters of LMs $134\times$. Our dataset is available at https://github.com/TobiasLee/VEC.

Submission Number: 671