CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimization

Xinting Liao, Behnoosh Zamanlooy, Masoumeh Shafieinejad, D. B. Emerson, Ruinan Jin, Deval Pandya, Xiaoxiao Li

Keywords: Federated learning, Prompt Injection Attack, Prompt Optimization

TL;DR: The textual collaborative prompt optimization reveals a new attack surface for prompt injection risks.

Abstract: Textual Collaborative Prompt Optimization (TCPO) extends Textgrad (Yuksekgonul et al., 2025) to a decentralized setting by allowing multiple clients to jointly improve prompts for large language models (LLMs) while keeping their data locally. Its reliance on free-form textual updating and aggregation introduces a new and largely unexplored attack surface, i.e., malicious instructions can be injected into local prompts and propagated through server-side prompt aggregation. Unlike conventional prompt injection attacks that define success as misleading LLM generating response shifted from the original task, attacking TCPO is more challenging as malicious instructions must simultaneously remain effective after aggregation, persist under subsequent benign prompt optimization, and stay stealthy enough to evade server-side defenses. To expose this risk, we propose CPInj, a collaborative prompt injection attack that contaminates the aggregated global prompt with malicious instructions, degrades downstream task performance, resists purification by prompt optimization on benign clients, and evades advanced detection-based defenses on the server. We find that current defense methods are ineffective against CPInj. To mitigate this attack, we further propose a defense-oriented aggregation method, i.e., APAgg, which purifies malicious instructions and recovers the utility of TCPO. We conduct extensive experiments across three LLM families and five reasoning tasks, which verify that our proposed attack reveals a critical vulnerability in TCPO. Although we take a first step toward mitigation, the attack remains highly effective and far from fully resolved, calling for more robust defense for TCPO.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 27