The LU-Mirage - An independent evaluation of the zero-shot claims in the LUMIR challenge

Rohit Jena, Pratik Chaudhari, James Gee

Keywords: neuroimaging, deep learning, image registration, foundation models

TL;DR: Independent, transparent, and fair evaluation of registration methods on LUMIR show that deep learning methods do not magically generalize to OOD contrasts and settings

Abstract:

The LUMIR challenge represents an important benchmark for evaluating deformable image registration methods on large-scale neuroimaging data.
While the challenge demonstrates that modern deep learning methods achieve competitive accuracy on T1-weighted MRI, it also claims exceptional zero-shot generalization to unseen contrasts and resolutions---assertions that contradict established understanding of domain shift in deep learning.
In this paper, we perform an independent re-evaluation of these zero-shot claims using rigorous evaluation protocols while addressing potential sources of instrumentation bias.
Our findings reveal a more nuanced picture: (1) deep learning methods perform comparably to iterative optimization on in-distribution T1w images and even on human-adjacent species (macaque), demonstrating improved task understanding; (2) however, performance degrades significantly on out-of-distribution contrasts (T2, T2*, FLAIR), with Cohen's d scores ranging from 0.7--1.5, indicating substantial practical impact on downstream clinical workflows; (3) deep learning methods face scalability limitations on high-resolution data, failing to run on 0.6mm isotropic images, while iterative methods benefit from increased resolution; and (4) deep methods exhibit high sensitivity to preprocessing choices.
These results align with the well-established literature on domain shift and suggest that claims of universal zero-shot superiority require careful scrutiny.
We advocate for evaluation protocols that reflect practical clinical and research workflows rather than conditions that may inadvertently favor particular method classes.

Primary Subject Area: Image Registration

Secondary Subject Area: Application: Neuroimaging

Registration Requirement: Yes

Reproducibility: https://github.com/rohitrango/lumirage-evals

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 9