Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques

Lang Xiong, Raina Gao, Alyssa Jeong, Yicheng Fu, Kevin Zhu, Sean O'Brien, Vasu Sharma

Keywords: sarcasm classification, sarcasm generation, large language models (LLMs), emotion-based prompting, sarcasm benchmark, sarcasm types, AI safety, bias, transparency, interpretability, robustness

TL;DR: Sarc7 introduces a new benchmark built by hand-annotating conversations with seven sarcasm types, exploring emotion-based prompting to improve large language models’ understanding of multi-class sarcasm classification and generation.

Abstract: In interactive systems, misreading sarcasm can undermine safety and robustness by causing models to interpret ironic remarks literally or generate unintended hostility. Sarc7 addresses this with a fine-grained, pragmatically grounded benchmark—built on MUStARD and annotated with seven distinct subtypes (self-deprecating, brooding, deadpan, polite, obnoxious, raging, manic)—that measures both classification and controlled generation performance. For classification, we compare zero-shot, few-shot, chain-of-thought (CoT), and a novel emotion-based prompting across five LLMs and find that emotion prompts boost macro-F1 compared to CoT prompting, reaching a highest of 0.3664 (Gemini 2.5).On generation, structured prompts defined by incongruity, shock value, context dependency, and emotion improve subtype alignment by 38.5% over zero-shot (Claude 3.5 Sonnet), enhancing interpretability and alignment with user intent. A human baseline (Cohen’s κ = 0.6694, macro-F1 = 0.6663) further highlights persistent error modes in brooding, deadpan, and polite sarcasm. By quantifying model versus human performance and exposing alignment failures—bias toward “not sarcasm” or “deadpan”—Sarc7 advances transparency, explainability, and the safe deployment of LLMs where pragmatic understanding is critical.

Submission Number: 33