Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

Bikang Pan, Wei Huang, Ye Shi

Keywords: Federated Learning, Vision-Language Foundation Models, Prompt Learning, Theoretical Analysis

TL;DR: This paper constructs a theoretical analysis framework for prompt-based federated learning via feature learning theory, and introduces a prompt portfolio mechanism to address severe data heterogeneity and balance generalization and personalization.

Abstract: Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.

Primary Area: Learning theory

Submission Number: 6698