Mixed Signals: Understanding Model Disagreement in Multimodal Empathy Detection

ACL ARR 2025 July Submission1000 Authors

Abstract: Multimodal models play a key role in empathy detection, but their performance can suffer when modalities provide conflicting cues. To understand these failures, we examine cases where unimodal and multimodal predictions diverge. Using fine-tuned models for text, audio, and video, along with a gated fusion model, we find that such disagreements often reflect underlying ambiguity, as evidenced by annotator uncertainty. Our analysis shows that dominant signals in one modality can mislead fusion when unsupported by others. We also observe that humans, like models, do not consistently benefit from multimodal input. These insights position disagreement as a useful diagnostic signal for identifying challenging examples and improving empathy system robustness.

Paper Type: Short

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Research Area Keywords: empathy detection, multimodality, model interpretability

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English

Previous URL: https://openreview.net/forum?id=dzXdX93sGq

Reassignment Request Area Chair: No, I want the same area chair from our previous submission (subject to their availability).

Reassignment Request Reviewers: Yes, I want a different set of reviewers

Justification For Not Keeping Action Editor Or Reviewers: Certain reviewers were unresponsive in the author response period. We hope to get more engagement in the upcoming round.

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: All previous work, models, and tools we reference and use have been properly cited in all sections and listed in the References section.

B2 Discuss The License For Artifacts: Yes

B2 Elaboration: All previous work, models, and tools we reference and use have been properly cited in all sections and listed in the References section.

B3 Artifact Use Consistent With Intended Use: Yes

B3 Elaboration: We fine-tune publicly available, open-source models (e.g., RoBERTa, HuBERT, VideoMAE) strictly for analysis purposes, as described in Section 3.

B4 Data Contains Personally Identifying Info Or Offensive Content: N/A

B4 Elaboration: We are reusing a publicly available dataset (published dataset) for which those authors already moderated content.

B5 Documentation Of Artifacts: Yes

B5 Elaboration: We will release all code and experimental resources at https://anonymous.4open.science/r/multimodal-empathy-disagreement-F48B to support reproducibility.

B6 Statistics For Data: Yes

B6 Elaboration: Relevant statistics are included in appendix sections A and B.

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: Details are included in section 3 and appendix section B.

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: Details are included in section 3 and appendix section B.

C3 Descriptive Statistics: Yes

C3 Elaboration: Details are included in Section 3.1 and 4.

C4 Parameters For Packages: Yes

C4 Elaboration: Details are included in Section 3.1.

D Human Subjects Including Annotators: Yes

D1 Instructions Given To Participants: Yes

D1 Elaboration: Details are included in appendix section C.

D2 Recruitment And Payment: Yes

D2 Elaboration: See ethics statement and appendix section C.

D3 Data Consent: Yes

D3 Elaboration: We provide detailed information on what we ask the annotators to annotate and how we plan to use the data. The annotators willingly agreed to participate with full knowledge of the task. We provide the full annotation instructions in appendix section C.

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: Yes

D5 Elaboration: Details are provided in appendix section C.

E Ai Assistants In Research Or Writing: No

E1 Information About Use Of Ai Assistants: No

Author Submission Checklist: yes

Submission Number: 1000