Divide and Conquer: Isolating Normal-Abnormal Attributes in Knowledge Graph-Enhanced Radiology Report Generation

Xiao Liang, Yanlei Zhang, Di Wang, Haodi Zhong, Ronghan Li, Quan Wang

Abstract: Radiology report generation aims to automatically generate clinical descriptions for radiology images, reducing the workload of radiologists. Compared to general image captioning tasks, the subtle differences in medical images and the specialized, complex nature of medical terminology limit the performance of data-driven radiology report generation. Previous research has attempted to leverage prior knowledge, such as organ-disease graphs, to enhance models' abilities to identify specific diseases and generate corresponding medical terminology. However, these methods cover only a limited number of disease types, focusing solely on disease terms mentioned in reports but ignoring their normal or abnormal attributes, which are critical to generating accurate reports. To address this issue, we propose a Divide-and-Conquer approach, named DCG, which separately constructs disease-free and disease-specific nodes within the knowledge graphs. Specifically, we extracted more comprehensive organ-disease entities from reports than previous methods and constructed disease-free and disease-specific nodes by rigorously distinguishing between normal conditions and specific diseases. This enables our model to consciously focus on abnormal information and mitigate the impact of excessively common diseases on report generation. Subsequently, the constructed graph is utilized to enhance the correlation between visual representations and disease terminology, thereby guiding the decoder in report generation. Extensive experiments conducted on benchmark datasets IU-Xray and MIMIC-CXR demonstrate the superiority of our proposed method. Code is available at the anonymous repository {https://anonymous.4open.science/r/DCG_Enhanced_distilGPT2-37D2}.

Primary Subject Area: [Content] Vision and Language

Secondary Subject Area: [Content] Multimodal Fusion, [Experience] Multimedia Applications, [Generation] Generative Multimedia

Relevance To Conference: Radiology report generation is an extension of the image captioning task in the medical field and a crucial research topic in multimedia computing. Exploring novel methods for generating radiology reports is essential for reducing the workload on doctors and advancing automated medical diagnostics. In this context, we propose a Divide-and-Conquer strategy, named DCG, that constructs a more comprehensive organ-disease graph, encompassing a broader range of diseases than previous methods, and serves as prior knowledge to enhance radiology report generation. Extensive experiments conducted on benchmark datasets IU-Xray and MIMIC-CXR demonstrate the superiority of our proposed method and also provide new research directions and technical perspectives for the field of multimedia processing and analysis, as covered by the ACM Multimedia conference.

Submission Number: 2992