Model-Heterogeneous Federated Prompt Learning

ICLR 2026 Conference Submission25616 Authors

Keywords: federated learning, prompt learning, heterogeneous model, vision-language models

Abstract: Large-scale vision-language models (VLMs) have shown remarkable transferability across tasks, and their integration into federated learning (FL) frameworks offers promising privacy-preserving learning capabilities. Recent advances in federated prompt learning (FPL) leverage prompt tuning to reduce computational and communication overhead. However, existing FPL methods assume a homogeneous model setting, where all clients share the same VLMs, which is an unrealistic constraint given the heterogeneous computational capacities of clients in real-world scenarios. To bridge this gap, we propose model-heterogeneous federated prompt learning (MHFPL), a novel setting where clients with diverse VLM backbones collaboratively learn prompts. We further introduce FedAPPR, a principled framework for MHFPL built on two key components: (a) server-level adversarial prompt alignment for aligning client semantics via adversarial training, and (b) client-level proximity regularization to further constrain prompt drift between clients. Extensive experiments on six datasets with diverse architectures and data distributions demonstrate the superiority and generality of FedAPPR compared to baselines, confirming it as an effective solution for FL scenarios with varying VLMs.

Primary Area: learning theory

Submission Number: 25616