Beyond Multi-modality: Evaluating AI’s Meme Literacy in Conversational Contexts on Bluesky

Juyoung An, Shirui Zhou, Haewoon Kwak, Jisun An

Keywords: Internet Memes, Meme Literacy, Multimodal AI, Conversational AI, Bluesky

Abstract:

Internet memes function as multimodal speech acts embedded in social context. While modern vision–language models (VLMs) perform strongly on standard benchmarks, their ability to interpret and communicate through memes in multi-turn conversations remains underexplored. We define meme literacy as the capacity to access, evaluate, synthesize, and communicate memes within context, and investigate two research questions: (RQ1) Can VLMs accurately evaluate contextual meme meaning? and (RQ2) Does conversational context improve meme generation?
For RQ1, we construct a multiple-choice benchmark using real conversational memes as gold answers. The Easy condition employs randomly sampled distractors, whereas the Hard condition introduces systematically constructed lexical, visual, and subtly semantic distractors to probe reliance on surface cues versus multimodal reasoning. GPT-4.1 remains stable across conditions (71.2% → 70.0%), while Qwen3-VL-Plus drops sharply (76.2% → 30.0%), and Phi-4-multimodal performs near random. Error analysis reveals a consistent lexical shortcut pattern: models frequently select distractors that reuse conversational keywords while failing to integrate visual nuance and pragmatic alignment. For RQ2, we evaluate meme generation using in-context learning with and without conversational context. Adding context often leads models to insert surface-level keywords directly into meme text rather than synthesize tone or situational meaning.
Overall, current VLMs appear to rely heavily on shallow lexical matching rather than robust multimodal reasoning. Advancing AI meme literacy therefore requires benchmark designs that explicitly discourage shortcut strategies and promote contextual multimodal integration.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 57