Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection

ACL ARR 2025 May Submission3665 Authors

Abstract:

Hateful memes have become a significant concern on the Internet, necessitating robust automated detection systems. While LMMs have shown promise in hateful meme detection, they face notable challenges like sub-optimal performance and limited out-of-domain generalization capabilities. Recent studies further reveal the limitations of both SFT and in-context learning when applied to LMMs in this setting. To address these issues, we propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs. Experiments on six meme classification datasets show that our approach achieves state-of-the-art performance, outperforming larger agentic systems.
Moreover, our method generates higher-quality rationales for explaining hateful content compared to standard SFT, enhancing model interpretability.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: hate speech detection, multimodal applications

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 3665