Multimodal Graph-LLM: Leveraging Graph-Enhanced LLMs for Multimodal Healthcare Predictions

Inyoung Choi, Sukwon Yun, Jiayi Xin, Jie Peng, Tianlong Chen, Qi Long

Keywords: EHR, multimodal, LLM, graphs, healthcare

Abstract: Multimodal healthcare research is crucial for improving clinical decision-making by integrating diverse data types, such as clinical notes, lab results, and imaging. Large Language Models (LLMs) are widely recognized for their exceptional text-based reasoning capabilities, making them effective in processing complex clinical narratives. However, they struggle to incorporate multimodal data, limiting their broader applicability in healthcare analysis. In this work, we propose MG-LLM (Multimodal Graph-LLM), a novel framework that leverages the strengths of LLMs while enhancing them with multimodal alignment and data integration through Graph Neural Networks (GNNs). GNNs propagate information across similar patients, model temporal relationships between visits, and align information from different modalities, creating enriched multimodal context vectors. These context vectors are then injected into the intermediate layers of the LLM, allowing it to harness both textual reasoning and multimodal data for more accurate predictions. We evaluate MG-LLM on the MIMIC-IV and MIMIC-CXR datasets, demonstrating significant improvements in clinical prediction tasks compared to baseline models. Our results showcase the potential of combining the text reasoning power of LLMs with GNN-driven multimodal alignment for robust, comprehensive healthcare analysis.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11363