Go to ACL ARR 2025 May homepagePsyScam: A Benchmark for Psychological Techniques in Real-World Scams
Abstract: Online scams have become increasingly prevalent, with scammers using psychological techniques (PTs) to manipulate victims.
While existing research has developed benchmarks to study scammer behaviors, these benchmarks do not adequately reflect the PTs observed in real-world scams.
To fill this gap, we introduce PsyScam, a benchmark designed to systematically capture and evaluate PTs embedded in real-world scam reports. In particular, PsyScam bridges psychology and real-world cyber security analysis through collecting a wide range of scam reports from six public platforms and grounding its annotations in well-established cognitive and psychological theories.
We further demonstrate PsyScam's utility through three downstream tasks: PT classification, scam completion, and scam augmentation.
Experimental results show that PsyScam presents significant challenges to existing models in both detecting and generating scam content based on the PTs used by real-world scammers.
Our code and dataset are available at: https://anonymous.4open.science/r/PsyScam-66E4.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: Interpretability and Analysis of Models for NLP, Language Modeling, Linguistic Theories, Cognitive Modeling, and Psycholinguistics, NLP Applications
Contribution Types: Data resources, Data analysis
Languages Studied: English
Submission Number: 2316
Loading