From Sensory Signals to Modality-Independent Conceptual Representations: A Probabilistic Language of Thought Approach

Goker Erdogan, Ilker Yildirim, Robert A. Jacobs

Abstract: Author Summary When viewing an object, people perceive the object’s shape. Similarly, when grasping the same object, they also perceive its shape. In general, the perceived shape is identical in these two scenarios, illustrating modality invariance, an important type of perceptual constancy. Modality invariance suggests that people infer a modality-independent, conceptual representation that is the same regardless of the modality used to sense the environment. If so, how do people infer modality-independent representations from modality-specific sensory signals? We present a hypothesis about the components that any system will include if it infers modality-independent representations from sensory signals. This hypothesis is instantiated in a computational model that infers object shape representations from visual or haptic (i.e., active touch) signals. The model shows perfect modality invariance—it infers the same shape representations regardless of the sensory modality used to sense objects. The model also provides a highly accurate account of data collected in an experiment in which people judged the similarity of pairs of objects that were viewed, grasped, or both. Conceptually, our research contributes to our understanding of modality invariance. Methodologically, it contributes to cognitive modeling by showing how symbolic and statistical approaches can be combined in order to understand aspects of human perception.