Position: Correct Answer, Wrong Mechanism - When AI Scientists Defend General Claims Their Own Data Contradicts

Steven Eulig

Track: Track 1: Original Research/Position/Education/Attention Track

TL;DR: Agents using a physics simulator reach correct outcomes through wrong mechanisms and defend them with claims that contradict their data or overreach. AI scientist evaluation must measure outcome, mechanism fidelity, and epistemic honesty separately

Abstract: AI scientist systems are described as tools, co-authors, or founders, but we evaluate them as if only the final answer matters. This position paper argues that outcome-only evaluation is insufficient, and that task outcome, mechanism fidelity, and epistemic honesty must be measured separately. Our evidence comes from 28 episodes of a coding agent attempting to rediscover a known particle identification observable in a Geant4 simulation, including an 8-episode probe across two additional frontier models. In 4/20 primary-model and 4/8 cross-model episodes, agents reach right-looking results through incorrect reasoning that breaks when conditions change, which we call Correct Answer, Wrong Mechanism. Honesty and mechanism fidelity dissociate within a single agent trajectory. When given a partially-misleading prior, all five agents reject the false component on evidence, yet one defends its chosen observable with physics inconsistent with its own data. Current coding agents are reliable tools but unreliable scientific co-authors for open-ended claim-making, where co-author trust requires mechanism-fidelity verification they do not reliably self-apply. We propose a lightweight test any simulation-based benchmark can adopt by presenting the agent with a setting where a standard observable is known to fail and scoring whether it catches the failure on its own.

Keywords: AI scientists, Agent evaluation, Scientific reasoning, Mechanism fidelity, Epistemic honesty

Submission Number: 26