"I Never Said That": A dataset, taxonomy and baselines on response clarity classification.

Anonymous

TL;DR: A dataset and taxonomy on evaluating response clarity on questions from political interviews

Abstract: Equivocation and ambiguity in public speech is a well-studied discourse phenomenon, especially in political science for the analysis of political interviews. Inspired by the well-grounded theory on equivocation, we aim to resolve the closely related problem of response clarity in questions extracted from political interviews, leveraging the capabilities of Large Language Models (LLMs) and human expertise. To this end, we introduce a novel taxonomy that frames the task of detecting and classifying response clarity and a clarity classification dataset which consists of question-answer pairs drawn from political interviews and annotated accordingly. Our proposed two-level taxonomy addresses the clarity of a response in terms of the information provided with respect to a given question (high-level), and also provides a fine-grained taxonomy of evasion techniques that relate to unclear, ambiguous responses (lower-level). Our annotation process leverages ChatGPT towards decomposing political dialogues into discrete question-answer pairs, each of which belongs to a specific response clarity and evasion category. Consequently, human annotators decide upon the correctness of this decomposition, while assigning an evasion label for each question-answer pair. We provide a detailed analysis of the dataset and we conduct several experiments using a range of LLMs to establish new baselines over the proposed dataset.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources, Data analysis

Languages Studied: English