A LLM-based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation
Abstract: The proliferation of misinformation and harmful narratives in online discourse has underscored the critical need for effective Counter Narrative (CN) generation techniques. However, existing automatic evaluation methods often lack interpretability and fail to capture the nuanced relationship between generated CNs and human perceptions. Aiming to achieve a higher correlation with human judgments, this paper proposes a novel approach to asses generated CNs that consists on the use of a Large Language Model (LLM) as a evaluator. By comparing generated CNs pairwise in a tournament-style format, we establish a model ranking pipeline that achieves a correlation of 0.88 with human preference. As an additional contribution, we leverage LLMs as zero-shot CN generators and conduct a comparative analysis of chat, instruct, and base models, exploring their respective strengths and limitations. Through meticulous evaluation, including fine-tuning experiments, we elucidate the differences in performance and responsiveness to domain-specific data. We conclude that chat-aligned models in zero-shot are the best option for carrying out the task, provided they do not refuse to generate an answer due to security concerns.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: NLG Evaluation, Counter Narrative Generation, Large Language Models
Languages Studied: English
Submission Number: 5406
Loading