X2CT-CLIP: Enable Multi-Abnormality Detection in Computed Tomography from Chest Radiography via Tri-Modal Contrastive Learning

Jianzhong You, Yuan Gao, Sangwook Kim, Chris Mcintosh

Abstract:

Computed tomography (CT) is a key imaging modality for
diagnosis, yet its clinical utility is marred by high radiation exposure
and long turnaround times, restricting its use for larger-scale screening.
Although chest radiography (CXR) is more accessible and safer, existing CXR foundation models focus primarily on detecting diseases that
are readily visible on the CXR. Recently, works have explored training
disease classification models on simulated CXRs, but they remain limited to recognizing a single disease type from CT. CT foundation models
have also emerged with significantly improved detection of pathologies in
CT. However, the generalized application of CT-derived labels on CXR
has remained illusive. In this study, we propose X2CT-CLIP, a tri-modal
knowledge transfer learning framework that bridges the modality gap between CT and CXR while reducing the computational burden of model
training. Our approach is the first work to enable multi-abnormality classification in CT, using CXR, by transferring knowledge from 3D CT volumes and associated radiology reports to a CXR encoder via a carefully
designed tri-modal alignment mechanism in latent space. Extensive evaluations on three multi-label CT datasets demonstrate that our method
outperforms state-of-the-art baselines in cross-modal retrieval, few-shot
adaptation, and external validation. These results highlight the potential
of CXR, enriched with knowledge derived from CT, as a viable efficient
alternative for disease detection in resource-limited settings.