Model-Free Assessment of Simulator Fidelity via Quantile Curves

Yu-Shiou Willy Lin, Garud Iyengar, Kaizheng Wang

Keywords: Simulation, Quantile function estimation, human-AI alignment, distribution-free

TL;DR: We propose a computationally tractable method to estimate the quantile function of the discrepancy between the simulator outputs and the ground-truth distributions.

Abstract: Simulation is now pervasive, arising from manufacturing to LLM-driven applications in research, education, and consumer surveys. Yet, fully characterizing the discrepancy between simulators and ground truth remains challenging. We propose a computationally tractable method to estimate the quantile function of the discrepancy between the simulated and ground-truth distributions. The approach does not impose any modeling assumptions on the simulator and it applies broadly across many parameter families: from Bernoulli and multinomial to continuous, vector-valued settings. The resulting quantile curve supports risk-aware summaries (e.g., VaR/CVaR) and comparison of simulators or prompts performance. We illustrate our framework through an application assessing LLM simulation fidelity on the OpinionQA dataset, augmented with simulations spanning seven LLMs.

Submission Number: 193