Hierarchical Fusion for Online Multimodal Dialog Act Classification

Md Messal Monem Miah, Adarsh Pyarelal, Ruihong Huang

Submission Type: Regular Long Paper

Submission Track: Dialogue and Interactive Systems

Submission Track 2: Speech and Multimodality

Keywords: Dialog Act Classification, Multimodality, Early Fusion, Online Inference

TL;DR: We propose a hierarchical fusion network that predicts dialog acts based on audio and ASR generated transcripts.

Abstract: We propose a framework for online multimodal dialog act (DA) classification based on raw audio and ASR-generated transcriptions of current and past utterances. Existing multimodal DA classification approaches are limited by ineffective audio modeling and late-stage fusion. We showcase significant improvements in multimodal DA classification by integrating modalities at a more granular level and incorporating recent advancements in large language and audio models for audio feature extraction. We further investigate the effectiveness of self-attention and cross-attention mechanisms in modeling utterances and dialogs for DA classification. We achieve a substantial increase of 3 percentage points in the F1 score relative to current state-of-the-art models on two prominent DA classification datasets, MRDA and EMOTyDA.

Submission Number: 4291