A robust and interpretable deep learning framework for multi-modal registration via keypoints

Alan Q. Wang, Evan M. Yu, Adrian V. Dalca, Mert R. Sabuncu

Abstract: Highlights•We present a deep learning-based image registration framework, called KeyMorph, that relies on automatically detecting corresponding keypoints, and which uses those keypoints to obtain the optimal transformation which aligns a pair of images via a differentiable, closed-form expression.•We use this observation to drive the end-to-end learning of keypoints tailored for the registration task, and without knowledge of ground-truth keypoints.•This framework allows for substantially more robust registrations, allows for better interpretability, can be designed to be equivariant under image translations and/or symmetric with respect to the input image ordering, and can generate multiple deformation fields corresponding to different transformation variants.•We demonstrate KeyMorph in solving 3D affine and spline-based registration of multi-modal brain MRI scans. In particular, we show registration accuracy that surpasses current state-of-the-art methods, especially in the context of large displacements.