LLM and agent security guide: from prompt injection to tool permissions

What it is and why it matters

LLM security is less about protecting the model itself than about protecting the apps and agents built around it. The instruction hierarchy study paper points out that prompt injection targets not the model but the apps built on top of it. If an app can access confidential data or take real actions, a single injection can leak user data or hijack the LLM's behavior.

The problem grew because LLMs spread quickly in setups that attach search results, documents and APIs. The indirect prompt injection study paper says that LLM-integrated apps blur the line between 'data' and 'instructions'. Traditional software treated input text only as data, but an LLM reading text can take sentences within it as commands.

To see the full range of threats at a glance, a shared taxonomy is needed. The OWASP Gen AI Security Project source lists the 2025 top 10 risks for LLM applications as prompt injection, sensitive information disclosure, supply chain vulnerabilities, data poisoning (contamination of training and embedding data), improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. A Korean translation is also available.

NIST's AI 100-2 E2025 report source provides a layered taxonomy and glossary of adversarial machine learning attacks, organized by type of ML method, the lifecycle stage at which the attack occurs, and the attacker's goals, capabilities and knowledge. It aims to create a 'common language' so that attacks such as data poisoning, evasion attacks and privacy compromise are called by the same names. The OWASP list works well as a checklist for app developers, and the NIST taxonomy as a shared vocabulary for communicating with security teams.

Prompt injection: direct, indirect, and the lethal trifecta

Prompt injection is an attack in which an attacker slips in instructions that override what the system designer intended. The instruction hierarchy study paper splits it into two kinds. Direct injection is when the app's user types the instructions straight into the input box. Indirect injection is when the attack instructions are inside third-party input, such as web search results or tool output. Simon Willison's post source stresses that prompt injection and jailbreaking (attacks that get a model to say forbidden content) should be treated as separate problems. The reason given is that if the two are treated as the same, developers will dismiss injection as "not relevant to our app."

The first systematic study of indirect injection paper showed that an attacker can remotely control an app without accessing it directly. Planting instructions in data that is likely to be retrieved is enough. The study demonstrated attacks on real systems such as the GPT-4-based Bing Chat and code-completion engines. It found that retrieved prompts can act like 'arbitrary code execution' and can even change which APIs are called and when. It grouped the harms into data theft, worm-like spreading, information ecosystem contamination, persistent compromise and denial of service.

Willison's post source calls one combination that is especially dangerous in agents the 'lethal trifecta'. It occurs when an agent has access to private data, is exposed to text or images an attacker can control (untrusted input), and can send data to the outside. When all three come together in one agent, an attacker can easily trick it into reading the data and then sending it to the attacker. API calls are not the only exfiltration channel. An HTTP request, an image load or even a single link the user clicks can also serve as a channel.

The root cause is that LLMs cannot reliably judge how much weight an instruction should get based on its source source. This is because every input is ultimately concatenated into a single sequence of tokens before it goes into the model. Anthropic's post source gives the example of instructions hidden in white text in an email that cause a 'confidential' email to be forwarded externally. It explains that browser agents carry greater risk because they have a wide attack surface and can take many kinds of actions.

MCP and tool security

MCP (Model Context Protocol) is a standard that lets agents plug in and use multiple tools. Willison's post source sees a risk in the way MCP encourages mixing tools from different sources. The tools that can reach private data are often the same ones that can carry malicious instructions. In the GitHub MCP attack the post uses as an example, all three parts of the trifecta were inside a single server. The agent could read a public issue posted by the attacker, access private repositories and create a pull request that sent private data out.

The model reads tool descriptions as text, along with the results that tools return. So attacks that hide instructions in tool descriptions or results (often called tool poisoning) work on the same principle. This is also why the instruction hierarchy study paper treats tool output as the lowest-privilege input.

For authentication and authorization, the official MCP security guidance source sets concrete rules. The first rule addresses the 'confused deputy' problem. Some MCP proxy servers connect to a third-party API with a static client ID. If an attacker exploits the consent cookie on such a server, they can intercept authorization codes without the user's consent. The guidance requires proxy servers to always obtain per-client consent. The consent screen must show the requesting client's name, the scopes and the redirect URI the token will be sent to. Redirect URIs must be checked for an exact match, without wildcards. OAuth state values should be single-use and expire within a short time (e.g., 10 minutes).

The second rule bans 'token passthrough'. Suppose an MCP server passes a token that was not issued to it on to downstream APIs without validating it. This bypasses controls such as rate limiting and request validation, and audit logs can no longer show who made a request source. The guidance explicitly forbids this practice. It explains that keeping token audiences separate from the start makes it easier to extend the security model later.

Layers of defense: design principles

The sources agree on one principle: a single layer of defense cannot stop these attacks. Willison's post source is skeptical of guardrail products that claim to "catch 95% of attacks" and points out that 95% is a failing grade in web security. Anthropic source reports that it reduced the attack success rate against its browser extension to around 1%. It also notes that this 1% is still a meaningful risk and that the problem has not been solved. So defenses that make the model stronger need to be combined with structural defenses that limit the scope of damage.

The main model-side defense is the instruction hierarchy. This study paper trained GPT-3.5 to give system messages priority over user messages, and user messages priority over third-party text such as tool results. It used synthetic data to teach the model to ignore lower-level instructions that conflict with higher-level ones. The authors reported a 63% improvement in defense against system prompt extraction. Robustness against jailbreak attacks that were not in the training data rose by more than 30%. However, 'over-refusal', where the model also refuses harmless requests, increased somewhat. Anthropic source also says it uses reinforcement learning to train the model to refuse injections. It uses classifiers to check untrusted content that enters the context window, and human red teamers keep trying to attack the model.

The key to structural defense is breaking one leg of the trifecta. Willison's post source quotes the point that an agent that has read untrusted input must be constrained so that this input cannot trigger consequential actions. The post also notes that vendors mostly fixed real incidents by closing exfiltration channels. In practice, this leads to four principles. Least privilege gives tools only the permissions the task requires. Sandboxing isolates code execution and file access. Human approval (human-in-the-loop) is required before consequential actions such as sending email, making payments or deleting data. Output checks inspect links, images and external addresses in the output. The OWASP list source points to exactly this area with its 'improper output handling' and 'excessive agency' entries.

The last layer is logging. The MCP security guidance source bans token passthrough partly because it breaks audit logs. In the same way, you need a record of who called which tool with what permissions to investigate incidents and fix controls.

Operational governance and adoption steps

Technical defenses hold up over time only if they are part of an organization's risk management framework. Google's SAIF source maps the AI development process from a security perspective. It lists 15 risks and the controls that address each one. It also has a section on agent security for autonomous systems and an agent risk self-assessment that shows the relevant risks and controls after you answer a few questions. The NIST AI RMF source is a voluntary framework released on January 26, 2023. It helps teams build trustworthiness considerations into the design, development, use and evaluation of AI products. It comes with the Playbook, an implementation guide, and the Generative AI Profile (NIST-AI-600-1, July 2024), which covers risks specific to generative AI. Version 1.0 is currently being revised.

For a first rollout, the following order is recommended. 1) List the data your app handles, the tools connected to it and the external content it reads in. Then check each item against the OWASP Top 10 risks source. 2) Check whether any agent has all three parts of the lethal trifecta source. If one does, start by changing the design to break one leg. 3) Cut tool permissions to the minimum, and check MCP servers' consent and token handling against the official guidance source. 4) Add human approval for consequential actions and a sandbox for code execution. 5) Add input classifiers and output checks, and have a red team try real attacks source. 6) Use the SAIF self-assessment source and the NIST AI RMF source to set owners, review cycles and incident response, and make these part of your operating framework.

The main point of this order is that it does not start from the idea that 'the model will block it.' Change the structure first, then lower the probability of a successful attack with the model and checkers. Finally, bring everything into the organization's management framework.

Checklist

At the design stage, check the following. Does any agent have private data, untrusted input and external transmission all at once source? Have you kept information that must not leak, such as passwords or keys, out of the system prompt (system prompt leakage, w1)? Are tool results and external documents kept separate from user instructions and treated with lower privilege paper?

Check tools and MCP as follows. Does each tool have only the permissions it truly needs? Do you avoid mixing in MCP servers of unclear origin source? Does the MCP proxy obtain per-client consent and require an exact match for redirect URIs? Does it avoid passing tokens on without validation source?

For execution and operations, check the following. Does a person approve actions such as sending email, deleting data or making payments before they happen? Does code run inside a sandbox? Are links, images and external addresses in the output checked? Is there a limit on resource use, such as token usage (unbounded consumption, w1)? Are tool calls logged? Do you run red-team attacks regularly source?

Common mistakes and limitations

The most common mistake is to rely only on writing "do not follow external instructions" in the system prompt. Willison's post source notes that you can add such wording. However, there are endless ways to phrase malicious instructions, so you can never be sure it will work every time. LLMs also do not always behave the same way, even with the same input, which adds to this uncertainty.

The second mistake is treating a guardrail product's detection rate as a security guarantee source. The third is treating prompt injection as the same problem as jailbreaking and thinking "our app just needs to avoid saying embarrassing things" source. The fourth is trusting the vendor to block attacks. Willison says vendors quickly closed exfiltration channels in their own products. But once users start mixing tools on their own, there is nothing the vendor can do.

The defense techniques themselves also have limits. Instruction hierarchy training paper greatly improved robustness but also increased over-refusal. Anthropic source says that attack techniques keep evolving, so prompt injection remains an active research topic and is not a solved problem. The indirect injection study paper also noted that there were no effective mitigations when it was published. For now, the only option is to combine defenses that lower the probability of a successful attack with designs that limit the scope of damage.

Further study

To start with the concepts, read the indirect prompt injection study paper first. Its core argument that apps blur the line between data and instructions, along with its classification of attacks, is the basis for later discussion. Next, read Willison's post on the lethal trifecta source to understand what to cut off when designing an agent.

For model-side defenses, see how the instruction hierarchy study paper trains a model to prioritize system messages, user messages and tool results. Then read Anthropic's browser agent post source for a real case that combines reinforcement learning, classifiers and red teaming.

For practical checks and operations, use the OWASP LLM Top 10 source as a checklist, the MCP security guidance source as a rulebook for tool integrations and NIST AI 100-2 source as a glossary. To set up an organization-wide framework, start with the SAIF source self-assessment and the NIST AI RMF source Playbook and Generative AI Profile.