Modal-Centric Insights Into Multimodal Federated Learning for Smart Healthcare: A Survey

Di Wang, Wenjian Liu, Longxiang Gao, Y. Neil Qu, Hu Zhang, Jihong Shi

Abstract: Federated Learning (FL) has progressed, providing a distributed mechanism where data need not be consolidated, thereby enhancing the privacy and security of sensitive healthcare data. Recent advancements in multimodal machine learning have empowered the development of accurate and robust AI systems in the medical domain, especially within centralized database systems. Multimodal federated learning (MFL) combines federated learning’s distributed nature with the flexibility to work with various data types. This capability is particularly relevant in medical contexts, where healthcare data can range from electronic health records (EHR) to radiology imaging, genomic data, and wearable device information. The key advantage of MFL in medical settings is the ability to create robust models without centralizing sensitive patient information. However, there are no existent research to comprehensively review the application status of MFL in smart healthcare systems from the modal-centric aspect. Motivated by this, we explore the systematization of knowledge about multimodal federated learning on medical data, find the challenges of this area and related solutions. With this research, we hope to provide a systematical review of this under-explored domain and provide insights for forthcoming researchers.