MemOracle: Symbolic Reasoning with Associative Retrieval for Harmful Content Detection

Hanling Wang, Chenlong Wei, Yingjuan Li, Di Wu, Yuchao Zhang, Xiaohui Zhu, Yao Zhu

Keywords: Symbolic Reasoning, Neuro-Symbolic AI, Multimodal Representation Learning, Harmful Content Detection

TL;DR: We introduce MemOracle, a retrieval-augmented symbolic reasoning system for multimodal hateful meme detection with improved interpretability and accuracy.

Abstract: Memes, as a prevalent form of online communication, often combine text and imagery to convey complex and sometimes harmful messages. Detecting hateful content in memes poses significant challenges due to their multimodal nature and the need for contextual reasoning. we propose a novel framework built upon vision-language models, empowered by multimodal retrieval and symbolic reasoning, to assess the harmfulness of memes. Specifically, Our system first parses the input meme using a vision-language model to extract image-text elements and semantic descriptions. These are embedded into a joint representation and stored in a vector database. For any given query meme, we retrieves similar examples from the database. A large language model is then employed to reason over the query meme based on the retrieved examples, guided by a predefined definition of hateful memes and a symbolic Chain-of-Thought prompt. The reasoning proceeds in three stages: translation, planning, and solver, producing both a decision and an explanatory rationale. Our approach enables a more transparent and context-aware assessment of online multimodal content. Comprehensive experiments on public FHM, HarM and MultiOff datasets outperforms state-of-the-art hateful meme detection frameworks on acc, balanced acc and mcc, demonstrate the effectiveness of MemOracle in interpreting and identifying harmful content. Comprehensive experiments on the public FHM, HarM, and MultiOff datasets demonstrate that MemOracle consistently surpasses state-of-the-art hateful meme detection models across accuracy, balanced accuracy, and MCC, highlighting its effectiveness in interpreting and identifying harmful content.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 8583