SecFPP : Secure Federated Prompt Personalization for Vision Language Models

ICLR 2026 Conference Submission22464 Authors

Keywords: Prompt Personalization, Federated Learning, Privacy Protection

TL;DR: This paper presents a novel protocol to achieve federated prompt personalization for VLMs with provable privacy guarantees and state-of-the-art performance under severe data heterogeneity.

Abstract: Prompt learning has emerged as an effective and widely-adopted approach for customizing pre-trained vision language models (VLMs) to user-specific downstream tasks. To tackle data shortage and heterogeneity across multiple users, federated prompt personalization (FPP) has received significant attention as an effective method to harmonize customized performance and pre-trained model generalization capability. However, user-specific prompts, as valuable intellectual assets, face increasing privacy risks such as prompt stealing attacks. Though conventional privacy-preserving techniques such as differential privacy can mitigate these risks by adding noise masks to prompt parameters, they can incur severe performance degradation due to prompt sensitivity. In this work, we propose SecFPP, a secure federated prompt personalization protocol, that reconciles the trade-off among model generalization, local personalization, and privacy preservation. SecFPP delivers state-of-the-art performance under severe data heterogeneity, while using secure multiparty computation primitives to provide formal privacy guarantees without utility loss. The proposed protocol employs a decoupled prompt adaptation strategy by decomposing user prompts into federated and local components, thereby improving personalization performance in multi-granular unbalanced data distributions. We develop a privacy-preserving adaptive clustering algorithm for federated prompts to capture different domains or dataset heterogeneity while using the local prompts to adapt downstream tasks and capture the class heterogeneity. We validate the security of SecFPP theoretically and empirically. Extensive experiments comparing SecFPP with non-private and privacy-preserving baselines demonstrate its superior personalization accuracy. Moreover, comparisons with existing privacy-preserving frameworks highlight that SecFPP significantly improves the privacy-performance trade-off in FPP, simultaneously delivering strongest privacy guarantees and enhanced personalization.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 22464