Framing the Game: How Context Shapes LLM Decision-Making

Isaac Robinson, John Burden

Keywords: AI Evaluation, Context Framing, Game Theory, Frontier Model Evaluation

Abstract: Large Language Models (LLMs) are increasingly deployed across diverse contexts to support decision-making. While existing evaluations effectively probe latent model capabilities, they often overlook the impact of context framing on perceived rational decision-making. In this study, we introduce a novel evaluation framework that systematically varies evaluation instances across key features and procedurally generates vignettes to create highly varied scenarios. By analyzing decision-making patterns across different contexts with the same underlying game structure, we uncover significant and specific contextual influence on LLM decision-making. Our findings demonstrate this variability is largely predictable, yet acutely sensitive to framing effects. These results underscore the urgent need for dynamic context-aware evaluation methodologies to ensure reliable LLM deployment in real-world applications, and provides initial directions for their construction.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 9454