LLM Persuasiveness Evaluation: A Structured Review of Automated Methods

Kamile Dementaviciute, Guillaume Bied, Tijl De Bie

Keywords: large language models, persuasion, manipulation, evaluation

Abstract: As Large Language Models (LLMs) become increasingly persuasive, their ability to shape beliefs and behaviour at scale has raised concerns, prompting regulatory attention and calls for robust evaluation frameworks. Human-participant studies provide ecological validity but are costly, slow, and constrained by ethical challenges, making them impractical for systematic assessment of rapidly evolving systems. As an alternative, fully automated evaluation methods requiring no human involvement enable reproducible, fast, and ethically unconstrained large-scale testing. To organise this rapidly growing literature and inform future research and development, we provide the first systematic taxonomy of 30 automated methods across 27 papers, examining their designs, human validation results, limitations, and associated risks.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 245