MultiSEAO-Mix: A Multimodal Multitask Framework for Sentiment, Emotion, Support, and Offensive Analysis in Code-Mixed Setting

Gopendra Vikram Singh, Mamta, Atul Verma, Asif Ekbal

Abstract: Social media platforms have become an open door for users to share their views, resulting in a growing trend of offensive content being shared on social media. Detecting and addressing offensive content is crucial due to its significant impact on society. Although there has been extensive research on the detection of offensive content in the English language, there is a notable gap in detecting offensive content in multimodal settings involving code-mixed languages. In this article, we propose a large scale multimodal code-mixed dataset for Hinglish (Hindi+English) MultiSEAO-Mix focusing on women and children. The MultiSEAO-Mix is annotated with offensiveness, sentiment, emotion, and their respective intensities. Additionally, it is also annotated with author support. A multimodal, multitask framework is proposed that considers offensive detection, intensity prediction, and author support as the primary tasks and improves their performance using sentiment, emotion, and corresponding intensities as the auxiliary tasks. Further, we propose a fusion technique that captures the enhanced multimodal representation to improve the performance of our model. Experimental results demonstrate that the proposed multitask framework improves the model performance by more than 4.5 points compared to multitask system without sentiment and emotion as the auxiliary tasks.