Generalizing Coverage Plots for Simulation-based Inference

TMLR Paper6257 Authors

Abstract:

Simulation-based inference (SBI) aims to find the probabilistic inverse of a non-linear function
by fitting the posterior with a generative model on samples. Applications demand accurate
uncertainty quantification, which can be difficult to achieve and verify. Since the ground
truth model is implicitly defined in SBI, we cannot compute likelihood values nor draw
samples from the posterior. This renders two-sample testing against the posterior impossible
for any practical use and calls for proxy verification methods such as expected coverage
testing. We introduce a differentiable objective that encourages coverage in the generative
model by parameterizing the dual form of the total variation norm with neural networks.
However, we find that coverage tests can easily report a good fit when the approximant
deviates significantly from the target distribution and give strong empirical evidence and
theoretical arguments why the expected coverage plot is, in general, not a reliable indicator
of posterior fit. To address this matter, we introduce a new ratio coverage plot as a better
alternative to coverage, which is not susceptible to the same blind spots. It comes at the
price of estimating a ratio between our model and the ground truth posterior, which can be
done using standard algorithms. We provide experimental results that back up this claim,
and provide multiple algorithms for estimating ratio coverage.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Michael_U._Gutmann1

Submission Number: 6257