Large Language Models for Logical Fallacy Detection

Nicole Teo, Donghao Huang, Erik Cambria, Zhaoxia Wang

Abstract: Identifying logical fallacies is essential for maintaining logical reasoning and reducing false information in a variety of domains, such as the media, law, and education. We present an extensive study on the use of large language models (LLMs) for logical fallacy detection and provide a comparative overview of model performance across various fallacy classes. We evaluate the logical fallacy detection capabilities of multiple state-of-the-art models (LLaMA, Qwen, Gemma, Phi) utilizing accuracy, precision, recall, and F1-score as assessment measures. According to our findings, our models do well on simple fallacies like “circular reasoning,” but they have trouble with more interpretive reasoning when it comes to more complex categories like “equivocation” and “intentional”. These results highlight the potential of LLMs in fallacy detection tasks but also indicate a need for improved prompt engineering, fine-tuning, and context-rich datasets to enhance interpretive accuracy. This research offers insights into advancing LLMs for critical reasoning applications, contributing to improved information integrity across domains.

External IDs:dblp:conf/pakdd/TeoHCW25