Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations

Aryan Shrivastava, Jessica Hullman, Max Lamparth

Keywords: Language Models, AI Safety, Natural Language Processing, Inconsistency, Transparency, Military

TL;DR: Using a metric that we test, we quantitatively measure free-form response inconsistency of LMs in a military setting and find they are prone to giving semantically inconsistent responses.

Abstract: There is an increasing interest in using language models (LMs) for automated decision-making, with multiple countries actively testing LMs to aid in military crisis decision-making. To scrutinize relying on LM decision-making in high-stakes settings, we examine the inconsistency of responses in a crisis simulation (``wargame"), similar to reported tests conducted by the US military. Prior work illustrated escalatory tendencies and varying levels of aggression among LMs but were constrained to simulations with pre-defined actions. This was due to the challenges associated with quantitatively measuring semantic differences and evaluating natural language decision-making without relying on pre-defined actions. In this work, we query LMs for free-form responses and use a metric based on BERTScore to quantitatively measure response inconsistency. We show that the inconsistency metric is robust to linguistic variations that preserve semantic meaning in a question-answering setting across text lengths. We first study the impact of different prompt sensitivity variations on wargame decision-making inconsistency at temperature $T = 0$. We find that all models exhibit levels of inconsistency indicative of semantic differences, even if answering to semantically identical prompts. We also study models at $T > 0$ under fixed prompts. We find that all studied models still exhibit high levels of inconsistency, even when adjusting the wargame setting, anonymizing involved conflict countries, or adjusting the sampling temperature parameter $T$. Further qualitative evaluation shows that models recommend courses of action that share few to no similarities. We find that inconsistency due to semantically equivalent prompt variations can exceed inconsistency from temperature sampling for most studied models across different levels of ablations. Given the high-stakes nature of military deployment, we recommend further caution be taken before using LMs to inform military decisions or other cases of high-stakes decision-making.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11361