Prompt Injection: A New Attack Surface and Its Defenses in the Age of Agents

Why it matters again: from chatbots to agents

Prompt injection is an attack that mixes an attacker's instructions into the input data a model reads, making the model do a task chosen by the attacker instead of its original one. The PromptLocate study paper defines it as 'an attack that contaminates input with a prompt consisting of injected instructions and data so that the model completes an attacker-chosen task.' According to the Prompt Injection 2.0 study paper, Preamble first discovered this attack in May 2022 and reported it to OpenAI.

In the chatbot era, the damage usually ended with one wrong answer. Agents are different. An agent reads the web, opens emails, calls tools and moves through multiple steps on its own. So a single line in the external data it reads can turn into a real action. Attacks that arrive through external data rather than direct user input are called indirect prompt injection. The DataFilter study paper explains that if an attacker plants instructions in data an agent accesses, they can override the user's original task at will and steer the agent toward harmful actions.

There is also a reason existing security tools struggle to stop it. Prompt Injection 2.0 paper analyzes 'hybrid threats' in which prompt injection is combined with traditional web attacks such as XSS (cross-site scripting) and CSRF (cross-site request forgery). It also covers benchmark results showing that existing controls such as web application firewalls, XSS filters and CSRF tokens fail to stop these attacks. Because the attack string is ordinary natural language rather than executable code, methods that filter by special characters or fixed patterns do not work well.

Core idea: models trust the voice over the marker

An LLM sees the world as a single continuous stream of text. Role distinctions such as system, user and tool are just markers inserted between parts of that stream. The role confusion study paper traces the root of prompt injection to exactly this point. A model judges where text comes from not by the role marker attached to it but by how the text sounds. Even if a command hidden in a web page carries a <tool> marker, the agent follows it as long as it sounds like the user.

It is similar to voice phishing that imitates a CEO's voice. Even if the caller ID (the marker) shows an outside number, an employee will follow a wire transfer instruction if the voice and manner of speaking match the CEO's. The study measured this phenomenon directly inside the model. Using 'role probes' to examine how the model internally recognizes 'who is speaking now,' the researchers found that injected text occupied the same Representation (the model's internal representation) space as the trusted role it imitated.

An attack that demonstrates this principle is CoT Forgery. If a plausibly crafted reasoning process (CoT, Chain-of-Thought) is inserted into a user prompt or tool output, the model mistakes it for its own thinking. The baseline attack success rate was nearly 0, but this attack achieved a 60% success rate against state-of-the-art models. In addition, the degree of role confusion measured before the model generated a single token could predict whether the attack would succeed. The study summarizes the result as 'to the model, sounding like a role and actually being that role are indistinguishable.'

If you accept this observation, you reach the conclusion that adding one more marker to a data segment saying 'this is data, do not follow it' is not enough. That is why defenses are moving away from adding more markers and toward changing the model's internal architecture or splitting system permissions.

Where attacks come in: the agent attack surface

The first path is the web pages, documents and emails an agent reads. Text is not the only risk. A study of text-based prompt injection against vision-language models paper reported that the simple method of putting instructions as written text inside an image is enough to fool VLMs, and that it does not require much compute.

The second path is tool descriptions. Agents connect to external tools through MCP (Model Context Protocol), and what each tool does is written as natural-language metadata. The Log-To-Leak study paper builds on the already known tool poisoning attack surface and covers an attack that makes the agent make one more attacker-controlled 'logging' call after all normal calls have finished. This call exfiltrates the user's query, the MCP server name, the server response and the agent's response. The injected instruction consists of four elements: Trigger, Tool Binding, Justification and Pressure. Across 18 model-server combinations where actual tool execution was confirmed, the attack success rate was 62.64–100%. In 16 of those combinations, the change in the original task completion rate was 6.63 percentage points or less. In other words, the work finishes normally while information leaks out.

The third path is state that persists over time. The cross-session stored prompt injection study paper points out that the security boundary has changed as agents gain state that survives the end of a session, such as memory, file systems and tools. Once external information crosses this boundary and becomes part of persistent state, a malicious instruction does not end with a single conversation. It stays quietly even after the attacker leaves and affects later runs. Borrowing the name from stored XSS, the study describes this threat as expanding along both a time axis (persisting and activating across sessions) and a space axis (spreading into persistent state outside the prompt).

How defenses have evolved: finding, removing and monitoring

The DataFilter study paper divides existing defenses into three types and notes the cost of each. Fine-tuning defenses require access to model weights, detection-based defenses lose a lot of utility, and system-level defenses require substantial system redesign. The study's solution is a filter model that removes malicious instructions before the data reaches the main model. It was built with SFT (supervised fine-tuning) on simulated injection data, and it looks at the user instruction and the data together, removing only the attack content while keeping legitimate information. The study reports that it reduced attack success rates to nearly 0 across several benchmarks while preserving utility. The trade-off it offers is that it sits in front of any model regardless of type, so it can also be used with black-box commercial LLMs.

AgentWatcher paper targets two weaknesses on the detection side: detection performance drops sharply as context grows longer, and judgments are opaque because there are no explicit rules for what counts as an injection. First, it shortens what needs to be checked by extracting only the small number of context segments that causally influenced the agent's action. Then it defines rules for 'what is and is not an injection' and has a monitoring LLM judge the extracted segments according to those rules. This makes it scalable to long contexts and makes the basis for each judgment explainable.

PromptLocate paper deals with the aftermath of an attack. Finding where the injected parts are within contaminated data is necessary for post-incident forensic analysis and data recovery. The method has three steps: it splits the data into semantic segments, finds the segments contaminated by injected instructions, and then pinpoints the segments contaminated by injected data. The study reports that it located injections accurately across 8 existing attacks and 8 adaptive attacks.

On the architecture side outside the model, Prompt Injection 2.0 paper proposes an architectural approach combining prompt isolation, runtime security and privilege separation. In summary, detection leaves the model untouched but can miss attacks or raise false positives; filters are easy to add but require running a separate model; and structural separation is strong but requires design changes.

Where things are heading: separation inside the model and control by the host

One direction is to build instruction-data separation into the model architecture itself. Heads of Security (HoS, paper) attaches a small learned gate to each attention head and has each head learn how to split its attention between trusted and untrusted tokens. This lets the model read and reason over external content without following the instructions inside it. It can be read as an attempt to directly fix, inside the model, the 'mismatch between markers and internal perception' that the role confusion study revealed. In evaluations on Llama-3.3-70B-Instruct and GPT-OSS-20B, the success rate of the black-box attack RL-Hammer was 97% on GPT-OSS-20B and 85% on the fine-tuning defense SecAlign, but 10% on HoS. The white-box attack Claudini needed 3 times more FLOPs against HoS than against other targets to reach the same success rate. By contrast, ASIDE, another structural defense, kept only 10% tool-use utility. This means that even when instructions and data are separated inside the model, the key question is how much utility is preserved.

The other direction is for the host (the side that runs the agent) to hold permissions instead of relying on the model's judgment. In Log-To-Leak paper, the three defenses evaluated left a residual attack success rate above 55% on the GitHub MCP and GPT-4o combination. The instruction detector wrongly flagged 85.53% of legitimate tool descriptions as attacks. Based on this, the study concluded that additional tool calls following a normal task should be controlled by requiring the host to authorize them directly. The cross-session stored study paper points in the same direction. It argues that security should not stop at filtering untrusted input but should move to 'state-centric security' that controls how external information gains 'authority' as it enters persistent state.

Evaluation methods are changing too. NTBench paper highlights a type of attack missed by systems that only block attacks that clearly deviate from the correct answer. It calls this type, which keeps most of a normal answer while making only part of it wrong or harmful, 'near-target prompt injection.' Under the same budget of 3 attack runs per sample, this attack achieved higher success rates than existing fixed-target attacks across all evaluated models and settings.

Limitations and open issues: what to weigh in practice

There is still no way to block these attacks completely. The numbers from the studies above show that each defense only lowers the attack success rate, leaving the problem of balancing residual attacks against false positives. A strict detector can end up blocking most legitimate tool descriptions, as in the Log-To-Leak paper case. A loose one leaves residual attacks. So rather than relying on any single layer, it is more realistic to stack detection, sanitization and permission control.

There are also evaluation pitfalls. In the HoS study paper, HoS and SecAlign performed similarly on agent benchmarks using static attacks, and the gap widened sharply only under adaptive attacks. This means fixed attack sets alone make it hard to tell how strong a defense really is. Also, as NTBench paper showed, checking only 'did the model abandon the user's request' misses attacks that subtly twist the output near the correct answer. As with Log-To-Leak, information can leak while the task completion rate stays the same, so a high task success rate should not be taken as proof of safety.

Here are some common mistakes. First, adding a single sentence like 'ignore external instructions' to the system prompt and stopping there. As the role confusion study paper shows, models trust the voice over the marker, so one sentence does not create a boundary. Second, saving content read from outside into memory or files without review. As the cross-session stored study paper points out, this turns a one-time attack into a persistent threat. Third, giving an agent that reads untrusted data external transmission or write permissions as well. A general security principle is to split permissions according to the principle of least privilege and to require human confirmation (human-in-the-loop) for hard-to-reverse actions.

Care is also needed when handling attack records. Because prompt injection attacks are plain text, any attack left in logs or red team reports becomes a copy that can be reused as is. Such records are material for improving defenses and, at the same time, sensitive assets that must be managed.

Further study

To grasp the principles first, start with the role confusion study paper. It is the starting point for understanding why defenses that add more markers hit a limit. Next, reading Prompt Injection 2.0 paper shows how this attack intertwines with traditional web security and what architecture-level defenses look like.

For the agent attack surface, read the cross-session stored prompt injection study paper and Log-To-Leak paper together. The former covers persistent state such as memory and files, and the latter covers MCP tool descriptions. For image-related risks, the VLM text injection study paper offers a quick overview.

Defenses are easier to organize in the order of before, during and after an attack. Read DataFilter paper, which sanitizes data before an attack; AgentWatcher paper, which monitors during execution; and PromptLocate paper, which traces the location afterward, in that order, then move on to HoS paper, which separates them through model architecture. If you are thinking about how to evaluate your own system, finish with NTBench paper and try adding near-target attacks to your evaluation scenarios.