Multi-Modality Multi-Loss Fusion Network

Anonymous

Abstract: In this work we investigate the optimal selection and fusion of features across multiple modalities and combine these in a neural network to improve emotion detection. We compare different fusion methods and examine the impact of multi-loss training within the multi-modality fusion network, identifying surprising findings relating to subnet performance. Our best model achieves state-of-the-art performance for three datasets (CMU-MOSI, CMU-MOSEI and CH-SIMS), and outperforms the other methods in most metrics. We have found that training on multimodal features improves single modality testing and designing fusion methods based on dataset annotation schema enhances model performance. These results suggest a roadmap toward an optimized feature selection and fusion approach for enhancing emotion detection in neural networks.

Paper Type: long

Research Area: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English, Mandarin

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.