Words and Worlds Both: Dynamic Effects of Distributional and Sensorimotor Information in Semantic Processing

Harshada Vinaya, Sean Trott, Diane Pecher, René Zeelenberg, Seana Coulson

Abstract: An important issue in the semantic memory literature concerns the relative importance of experience-based sensorimotor versus language corpus-based distributional information in conceptual representations. To explore how each contributes to behavioral and neural responses on a conceptual task, EEG and RTs were recorded as healthy young adults viewed terms for concepts (e.g., “APPLE”) followed by properties (e.g., “red”) and pressed a button to indicate whether the property is true or false for the concept. Next, we constructed a series of mixed effects models of response times (RTs) and single-trial electroencephalogram (EEG) responses to the property words. Distributional models predicted data using semantic distance measures (e.g., between “APPLE” and “red”) derived from language corpus-based measures developed by computational linguists. Sensorimotor models predicted data using sensorimotor distance, a measure based on comparisons of each word’s experiential strength on the perceptual and action-effector dimensions from the crowd-sourced Lancaster Sensorimotor Norms. Statistical model comparison was used to determine whether the data was best fit by Distributional, Sensorimotor, or both sorts of information. In keeping with hybrid accounts of semantic memory, we find that both measures of semantic distance explained unique variance for behavioral and neural measures. Modelling EEG across seven successive 100-ms intervals revealed that the predictors’ temporal dynamics varies between true (APPLE – red) and false (APPLE – black) trials, but showed early sensorimotor activation for both. Results show how linguistic context and task demands modulate the recruitment of different information sources, supporting dynamic hybrid accounts of semantic memory.

External IDs:doi:10.1162/opmi.a.316