Unveiling Misogyny Memes: A Multimodal Analysis of Modality Effects on Identification

Shijing Chen, Usman Naseem, Imran Razzak, Flora D. Salim

Abstract: In today's digital era, memes have become a popular means of communication that often reflect societal attitudes as well as prejudices. Misogyny memes are a form of memes that explicitly discriminate against women in various aspects, such as shaming or stereotyping. This research aims to identify misogynous memes through deep learning multimodal analysis and determine which modality, text or image, plays a more significant role in fairness considerations. To achieve this, we utilized the dataset GOAT-benchmarks, which comprises over 6,000 diverse memes covering topics like implicit hate speech, sexism, and cyberbullying. Furthermore, we evaluated the fairness of these models by assessing their performance across different demographic groups. Our findings revealed that while both text and image modalities contribute to identifying misogynous memes, text plays a significant role in misogyny identification, while image contributes further in terms of fairness. This study emphasizes the importance of multimodal analysis in recognizing and mitigating biases in online content.Disclaimer: This paper contains content that may be disturbing to some readers.