Go to MIDL 2026 Short Papers homepageEpistemic Reliability of Frontier Multimodal LLMs in Subspecialist-Ophthalmology: A Factorial Evaluation of Image-Reliant Reasoning
Keywords: large language models, ophthalmology, clinical reasoning, performance metric, confidence calibration, reasoning verbosity
TL;DR: While frontier multimodal LLMs show near-ceiling diagnostic accuracy, they exhibit systematic overconfidence and a 'verbosity paradox' in complex ophthalmic management decisions requiring image-reliant reasoning.
Registration Requirement: Yes
Abstract: Existing evaluations of large language models (LLMs) in ophthalmology measure accuracy under idealized,
single-condition settings, leaving uncharacterized three epistemic properties—confidence calibration, output self-consistency, and cross-model failure convergence—directly relevant to clinical reliability. We evaluated four frontier multimodal LLMs (GPT-5.4, Claude Opus 4.6, Gemini 2.5 Pro, Gemini 3.1 Pro Preview) on 20 real-world subspecialist-level retinal and uveitic cases using a fully factorial 2×2×3 design (case language × prompt language × input modality) with five repeated runs (4,800 total responses). Overall accuracy was 89.2%–90.3% with no significant between-model differences (P ≥.109), substantially outperforming senior residents (50.0%; P ≤.008). Diagnostic accuracy was near-ceiling (97.5%–100.0%) while management accuracy was substantially lower (63.9%–72.8%; P <.001). Three systematic epistemic patterns emerged invisible to accuracy alone: systematic overconfidence amplified in management tasks (+14.4 to +32.1 pp); a verbosity paradox in which incorrect responses were significantly more elaborate (P ≤.007, three of four models); and cross-model convergence on identical errors. These findings reframe LLM evaluation in medical imaging toward epistemic reliability.
Visa & Travel: Yes
Read CFP & Author Instructions: Yes
Originality Policy: Yes
Single-blind & Not Under Review Elsewhere: Yes
LLM Policy: Yes
Submission Number: 93
Loading