Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis

ICLR 2026 Conference Submission22686 Authors

Keywords: prompt injection defense, intent analysis, reasoning model

Abstract: Indirect prompt injection attacks (IPIAs), where large language models (LLMs) follow malicious instructions hidden in input data, pose a critical threat to LLM-powered agents. In this paper, we present IntentGuard, a general defense framework based on instruction-following intent analysis. The key insight of IntentGuard is that the decisive factor in IPIAs is not the presence of malicious text, but whether the LLM intends to follow instructions from untrusted data. Building on this insight, IntentGuard leverages an instruction-following intent analyzer (IIA) to identify which parts of the input prompt the model recognizes as actionable instructions, and then flag or neutralize any overlaps with untrusted data segments. To instantiate the framework, we develop an IIA that uses three ``thinking intervention'' strategies to elicit a structured list of intended instructions from reasoning-enabled LLMs. These techniques include start-of-thinking prefilling, end-of-thinking refinement, and adversarial in-context demonstration. We evaluate IntentGuard on two agentic benchmarks (AgentDojo and Mind2Web) using two reasoning-enabled LLMs (Qwen-3-32B and gpt-oss-20B). Results demonstrate that IntentGuard achieves (1) no utility degradation in {all but one setting} and (2) strong robustness against adaptive prompt injection attacks (e.g., reducing attack success rates from 100\% to 8.5\% in a Mind2Web scenario).

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 22686