WHEN STABILITY FAILS: HIDDEN FAILURE MODES OF LLMS IN DATA-CONSTRAINED SCIENTIFIC DECISION-MAKING

Nazia Riasat

Keywords: large language models, scientific decision-making, negative results, reproducibility, prompt sensitivity, differential expression analysis, evaluation metrics, hallucination, stability vs correctness

TL;DR: Stable and reproducible LLM outputs can still be wrong: we identify prompt-sensitive, systematic failure modes in data-constrained scientific decision-making tasks.

Abstract: Large language models (LLMs) are increasingly used as decision-support tools in data-constrained scientific workflows, where correctness and validity are critical. However, evaluation practices often emphasize stability or reproducibility across repeated runs. While these properties are desirable, stability alone does not guarantee agreement with statistical ground truth when such references are available. We introduce a controlled behavioral evaluation framework that explicitly separates four dimensions of LLM decision-making: stability, correctness, prompt sensitivity, and output validity under fixed statistical inputs. We evaluate multiple LLMs using a statistical gene prioritization task derived from differential expression analysis across prompt regimes involving strict and relaxed significance thresholds, borderline ranking scenarios, and minor wording variations. Our experiments show that LLMs can exhibit near-perfect run-to-run stability while systematically diverging from statistical ground truth, over-selecting under relaxed thresholds, responding sharply to minor prompt wording changes, or producing syntactically plausible gene identifiers absent from the input table. Although stability reflects robustness across repeated runs, it does not guarantee agreement with statistical ground truth in structured scientific decision tasks. These findings highlight the importance of explicit ground-truth validation and output validity checks when deploying LLMs in automated or semi-automated scientific workflows.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 10