MoMCE: Mixture of Modality and Cue Experts for Multimodal Deception Detection

Dongliang Zhu, Ruimin Hu, Zitong YU, Mei Wang, shuo Ye, Fei Ma, Xiaochun Cao

Keywords: multimodal deception detection; audio-visual

Abstract: Multimodal audio-visual deception detection aims to predict whether a person is lying by integrating visual and acoustic modalities, which has two main challenges: 1) the modality conflict problem and 2) heterogeneous cue representation difficulty. However, existing approaches 1) often overlook the differences across modalities for different individuals; and 2) typically rely on a single encoder to handle diverse and individual-specific cues, which limits models' representation capacity for heterogeneous cues. To address these challenges, we propose MoMCE, a novel model with mixture of modality and cue experts for deception detection. It consists of two key components: 1) Prompt-aware Mixture of Modality Experts, which employs a learnable prompt routing mechanism to generate adaptive instance-aware modality weight distributions for dynamic modality adjustment. In addition, we propose a consistency-aware expert weighting loss. For samples with high cross-modal consistency, it encourages balanced contributions across modalities. In contrast, for samples with strong conflicts, it reduces the entropy of the modality weight distribution to focus on more reliable modalities. 2) Prompt-aware Mixture of Cue Experts, which captures heterogeneous and diverse deceptive cues within each modality. This module introduces multiple experts with distinct semantic biases on top of a shared backbone to model different deceptive patterns. Additionally, we introduce a cue expert diversity loss to balance learning across multiple cue experts, promoting effective representation of diverse deceptive cues. Extensive experiments demonstrate that MoMCE adapts to variations in both cross-modal contributions and cue heterogeneity, achieving substantial improvements in deception detection performance.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 4325