FedMRG: federated medical report generation via text-aware learning rate adjustment and multi-level prototype collaboration

Hichem Metmer, Xiaoshan Yang

Abstract: Medical report generation (MRG), which aims to automatically generate textual descriptions of medical images (e.g., chest X-rays), has gained significant research interest as a means to reduce the radiology reporting workload. However, existing MRG methods heavily rely on large-scale datasets, raising significant privacy concerns. In this paper, we introduce FedMRG, a Federated Medical Report Generation task that facilitates collaborative learning across multiple hospitals while preserving privacy. FedMRG addresses two key challenges: (1) text richness imbalance and (2) Feature contribution diversity. To tackle these challenges, we propose a novel two-step framework: (1) federated cross-modal pre-training and (2) fine-tuning with limited annotations. To address text richness imbalance issue, we introduce the Text-Aware Learning Rate Adjustment (TALRA) module, which ensures balanced participation from clients with varying levels of textual data richness. To tackle feature contribution diversity, we propose the Multi-Level Prototype Collaboration (MLPC) mechanism, which efficiently shares and integrates multi-level prototypes across various clients with different data modalities. Extensive experiments on four benchmark datasets demonstrate the effectiveness of the proposed method for MRG in a decentralized, yet collaborative learning environment.