Neural Ratio Estimators Meet Distributional Shift and Mode Misspecification: A Cautionary Tale from Strong Gravitational Lensing
Keywords: Machine Learning, ICML, Physics, Cosmology, strong gravitational lensing, NREs, dark matter, distribution shifts
TL;DR: We investigate the behaviour of Neural Ratio Estimators when evaluating data originating from different distributions than the training distribution and found high biases for even slight changes in the parameter distributions.
Abstract: In recent years, there has been increasing interest in the field of astrophysics in applying Neural Ratio Estimators (NREs) to large-scale inference problems where both amortization and marginalization over a large number of nuisance parameters are needed.
Here, in order to assess the true potential of this method to produce unbiased inference on real data, we investigate the robustness of NREs to distribution shifts and model misspecification in the specific scientific application of the measurement of dark matter population-level parameters using strong gravitational lensing. We investigate the behaviour of a trained NRE for test data presenting distributional shifts inside the bounds of training, as well as out of distribution, both in the linear and non-linear parameters of this problem. While our results show that NREs perform when tested perfectly in distribution, we find that they exhibit significant biases and drawbacks when confronted with slight deviations from the examples seen in the training distribution. This indicates the necessity for caution when applying NREs to real astrophysical data, where underlying distributions are not perfectly known and models do not perfectly reconstruct the true underlying distributions.
Submission Number: 140
Loading