Rethinking masked image modelling for medical image representation

Yutong Xie, Lin Gu, Tatsuya Harada, Jianpeng Zhang, Yong Xia, Qi Wu

Abstract: Highlights•Introduce MedIM, a pioneering approach using radiology reports as a guidance for masked medical image modelling.•Develop knowledge-driven masking, leveraging Medical Subject Headings for targeted masking.•Develop sentence-driven masking, leveraging sentence-level information to identify symptom regions for masking.•Demonstrated superior performance of MedIM in medical image and image–text analysis tasks over leading methods.