MMLN: Leveraging Domain Knowledge for Multimodal Diagnosis

Haodi Zhang, Chenyu Xu, Peirou Liang, Ke Duan, Hao Ren, Weibin Cheng, Kaishun Wu

Abstract: Deep learning models have been widely studied and have achieved expert-level performance in medical imaging tasks such as diagnosis. Recent research also considers integrating data from various sources, for instance, chest X-rays (CXR) radiographs and electronic medical records (EMR), to further improve the performance. However, most existing methods ignore the intrinsic relations among different sources of data, thereby lack interpretability. In this paper, we propose a framework for pulmonary disease diagnosis that combines deep learning and domain-knowledge reasoning. We first formalize the standard medical guidelines into formal-logic rules, and then learn the weights of the rules from medical data, integrating multimodal data for pulmonary disease diagnosis. We verify our method on a real dataset collected from a hospital, and the experimental results show that the proposed method outperforms the previous state-of-the-art multi-modal baselines.