What’s Your Argument? A Detailed Investigation on Argument Detection and Understanding with LLMs

ACL ARR 2024 December Submission1743 Authors

Abstract: Automated large-scale analysis of online argumentation around contested issues like abortion requires detecting and understanding recurring arguments. Despite extensive research in computational argumentation analysis, these tasks have not been tested with large language models (LLMs). We address this gap using a dataset of over 2,000 opinion comments on polarizing topics and define three tasks: argument detection, extraction, and identifying whether an argument is supported or attacked in a comment. We compare the performance of four state-of-the-art LLMs and a fine-tuned RoBERTa baseline. Our findings show that while LLMs excel at binary support/attack decisions, they struggle to extract argument spans in text, and performance does not consistently improve with in-context learning. We conclude by discussing the implications and limitations of current LLMs in argument-based opinion mining.

Paper Type: Long

Research Area: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Research Area Keywords: Argument Mining, Argument-Based Opinion Mining, LLMs, Argumentation

Languages Studied: Python

Submission Number: 1743