Multi-modality Contrastive Learning for Sarcopenia Screening from Hip X-rays and Clinical Information

Qiangguo Jin, Changjiang Zou, Hui Cui, Changming Sun, Shu-Wei Huang, Yi-Jie Kuo, Ping Xuan, Leilei Cao, Ran Su, Leyi Wei, Henry B. L. Duh, Yu-Pin Chen

Abstract: Sarcopenia is a condition of age-associated muscle degeneration that shortens the life expectancy in those it affects, compared to individuals with normal muscle strength. Accurate screening for sarcopenia is a key process of clinical diagnosis and therapy. In this work, we propose a novel multi-modality contrastive learning (MM-CL) based method that combines hip X-ray images and clinical parameters for sarcopenia screening. Our method captures the long-range information with Non-local CAM Enhancement, explores the correlations in visual-text features via Visual-text Feature Fusion, and improves the model’s feature representation ability through Auxiliary contrastive representation. Furthermore, we establish a large in-house dataset with 1,176 patients to validate the effectiveness of multi-modality based methods. Significant performances with an AUC of 84.64%, ACC of 79.93%, F1 of 74.88%, SEN of 72.06%, SPC of 86.06%, and PRE of 78.44%, show that our method outperforms other single-modality and multi-modality based methods.