Shaped by meaning, weighted by reliability: New insights into multisensory integration

Elizaveta Sycheva, Léa St-Gelais, Karim Jerbi, Shahab Bakhtiari, Franco Lepore, Vanessa Hadid

Abstract: In everyday perception, sensory information is rarely interpreted in isolation. Instead, observers generate expectations about what is likely to occur next, using contextual information to guide decisions, particularly when sensory input is weak, ambiguous, or uncertain. These expectations can facilitate behaviour when they align with incoming evidence, but impose costs when they conflict, shaping how information from different senses contributes to behaviour. However, it remains unclear what kind of multisensory process is engaged when meaning guides decisions, and whether multisensory facilitation and non-independence reflect a single integrative mechanism or distinct routes shaped by context. Here, we examined how semantic context, visual clarity, and the relationship between auditory and visual signals jointly shape behaviour in a semantic correspondence task using dynamic, naturalistic events. Participants judged whether a target matched a preceding concept prime, establishing an expectation at the level of meaning. Targets were auditory-only, visual-only (clear or blurry), or audiovisual (AV), with AV components conveying either the same or different meanings across modalities. AV facilitation was not a general consequence of stimulus redundancy. When the prime accurately predicted the target and auditory and visual signals conveyed the same meaning, degrading visual input increased the contribution of auditory information, leading to faster responses and violations of the race-model inequality. By contrast, when the prime indicated the target but auditory and visual signals conveyed competing meanings, AV stimulation produced robust behavioural costs relative to vision. When the prime did not match the target, congruent and incongruent audiovisual events produced comparable effects, and race-model violations were observed even when auditory and visual signals encoded different concepts, indicating convergence at the level of the response rather than shared perceptual content. Finally, exploratory clustering revealed two participant subgroups with comparable overall response speed but systematically different multisensory profiles when semantic expectations did not allow reliable anticipation of the upcoming event. Together, these findings show that multisensory effects in semantic decisions are shaped by meaning and arise through two functionally distinct routes: one in which reduced sensory reliability promotes joint use of congruent audiovisual information, and another in which perceptually different signals converge on the same response when semantic expectations are not met.