Do Prompted Strategic Personas Influence Decision Making in Large Language Models? A Chess-Based Experimental Study

AADIT SHAH, Yash Sinha

Keywords: large language models, persona prompting, chess, decision making, prompt steering, behavioral evaluation, move quality, structured reasoning

TL;DR: A chess-based evaluation shows that persona prompts measurably change LLM move choices, but the resulting behavior only partially aligns with the intended strategic style.

Abstract: Persona prompting is widely used to steer large language models (LLMs), but it remains unclear whether a persona changes only explanation style or also concrete decisions in structured tasks. We study this question using chess, where board states are exactly represented by FEN strings, legal actions are enumerable, and selected moves can be evaluated against a search baseline. Instead of comparing full games, where early mistakes change all later positions, we use a controlled position-based evaluation over 800 model calls across 199 unique FEN positions. Each position is evaluated under four prompts: Neutral Baseline, Aggressive, Defensive, and Beginner Materialist. For every call, the system provides the board state and legal moves, parses a SAN/UCI move, checks legality, compares the move against a depth-4 negamax alpha-beta baseline, and computes persona-aligned behavioral metrics for aggression, defense, and material-based accuracy. Results show that persona prompts do alter decisions: Aggressive, Defensive, and Materialist match the neutral move only 26.0%, 17.0%, and 29.5% of the time, respectively. However, style adherence is uneven. The Defensive persona achieves the strongest objective quality, with 98.5% legal moves and 70.5% Best/Good moves, while the Aggressive persona obtains the highest attack metric but lower accuracy than Neutral. These findings suggest that persona prompting can measurably influence LLM decision making, but the induced behavior is not always reliably aligned with the named strategic role.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Paper Type: Short paper

Submission Number: 20