Detecting Beta-Amyloid via Transformer-Based Multimodal Integration of MRI and PET Imaging

Francesco Chiumento, Julia Dietlmeier, Ronan P. Killeen, Kathleen M Curran, Noel O'Connor, Mingming Liu

Keywords: Contrastive Learning, Cross-modal Attention, MRI, PET, Transformer

TL;DR: PET-guided cross-attention approach to enhance MRI embeddings through contrastive learning for beta-amyloid detection.

Abstract: In this paper, we propose a multimodal contrastive learning framework that integrates AV45 PET and 3T MRI data from 511 baseline participants in the OASIS-3 cohort. Built on BiomedCLIP, our model incorporates cross-modal attention and a soft triplet loss with adaptive margin to align PET–MRI embeddings. After contrastive pretraining, a lightweight MLP predicts amyloid positivity using PET-guided MRI representations. Results show our approach learns robust MRI features that capture PET-derived signals for reliable beta-amyloid prediction.

Submission Number: 50