TGV: Tabular Data-Guided Learning of Visual Cardiac Representations

Marta Hasny, Maxime Di Folco, Keno Bressem, Julia Schnabel

Keywords: Visual Representations, Contrastive Learning, Cardiac MRI

TL;DR: Using tabular data to define clinically meaningful pairs in contrastive learning

Abstract: Contrastive learning methods in computer vision typically rely on different views of the same image to form pairs. However, in medical imaging, we often seek to compare entire patients with different phenotypes rather than just multiple augmentations of one scan. We propose harnessing clinically relevant tabular data to identify distinct patient phenotypes and form more meaningful pairs in a contrastive learning framework. Our method uses tabular attributes to guide the training of visual representations, without requiring a joint embedding space. We demonstrate its strength using short-axis cardiac MR images and clinical attributes from the UK Biobank, where tabular data helps to more effectively distinguish between patient subgroups. Evaluation on downstream tasks, including fine-tuning and zero-shot prediction of cardiovascular artery diseases and cardiac phenotypes, shows that incorporating tabular data yields stronger visual representations than conventional methods that rely solely on image augmentations or combined image-tabular embeddings. Our results show that tabular-guided training produces strong unimodal image encoders, highlighting the potential of our approach for medical foundation model development.

Submission Number: 11