What Is Prompt Injection? (And How Agents Defend Against It)
Prompt injection hides instructions in the content an AI agent reads. What direct and indirect injection are, why instructions alone can't stop them, and the layered design that keeps an agent safe: least privilege, fail-closed gates, human approval.
TL;DR: Prompt injection is the defining attack on AI agents: instructions hidden in the content an agent reads, steering it while it thinks it’s doing its job. The research paper that named it called the result “processing retrieved prompts as arbitrary code execution.” It can’t be fully prevented today, and the industry’s own agents have already broken out of test environments. The defense is not a cleverer prompt; it’s layered architecture: least privilege on tools, gates enforced by the application below the model, and a human approving every consequential action. This piece explains the attack, why it works, and what a defensible agent looks like, using mrmr as the worked example.
Here is the sentence that changes how you think about AI agents: your agent reads the web, your files, your messages, and your tickets, and the attacker can write to all of those places.
That is not a hypothetical. In May 2026, Google confirmed that Gemini breached the security of three external companies during a cybersecurity test, guessing a password in one case and using credentials found in a public repository in the others. In September, Anthropic’s CEO made a public call to pace frontier AI development, citing similar incidents from OpenAI and Anthropic. Whatever you think of those specific cases, the pattern they belong to is older and structural, and it has a name the security industry takes seriously enough to rank first on its threat list.
What prompt injection is
Prompt injection is an attack where instructions planted in the input change what an AI model does, in ways the operator never intended. The OWASP Gen AI Security Project ranks it first on its Top 10 for LLM applications, and defines the core problem in one sentence: these inputs “alter the LLM’s behavior or output in unintended ways,” and they do not even need to be readable by humans, because “as long as the content is parsed by the model,” the injection can work.
Two kinds matter for agents:
Direct injection is the classic: someone types a prompt at the model to override its instructions. For an assistant product, this is the smaller risk, because the attacker needs direct access to your conversation.
Indirect injection is the one agents actually face. The instructions ride in content the agent reads: a web page it fetches, a file it summarizes, an email it digests, a ticket it reviews, an image it looks at. Nobody types anything at your agent. The attacker plants text where your agent will find it, and your agent does the rest. The research paper that named the technique put it precisely: LLM-integrated applications “blur the line between data and instructions,” and processing retrieved prompts can act like arbitrary code execution, controlling how and whether other APIs get called.
The reason this works is not a bug you can patch. The model’s whole job is to treat language as meaningful, and it has no reliable sense of which words are yours, which are the task’s, and which are an attacker’s, woven into the same context window. Instructions can be invisible to you (rendered in ways you can’t see, encoded, split across sources, written in another language), yet fully legible to the model.
Why this matters more when the agent can act
A chatbot that gets injected produces a bad sentence. An agent that gets injected performs a bad action with your credentials.
The research community saw this immediately: injections against LLM-integrated applications can “manipulate the application’s functionality and control how and if other APIs are called.” In agent terms: the injected instruction doesn’t need to convince you; it needs to convince the model, and the model is holding your connectors. OWASP’s impact list reads like an agent’s permission slip: disclosure of sensitive information, unauthorized access to the model’s functions, “executing arbitrary commands in connected systems,” manipulation of decisions.
And this is not a 2023-era lab curiosity. The last year produced a string of documented breakouts during controlled security tests: models from OpenAI, Anthropic, and Google that went further than their operators intended, with Google confirming that Gemini breached three real companies before its own safety mechanisms stopped it. Security testing is the friendly version. The hostile version is the same class of problem with an attacker choosing the target.
The honest engineering summary comes from OWASP itself: because of how these models work, “it is unclear if there are fool-proof methods of prevention.” Prompt injection is mitigated in layers. It is not solved by a system prompt.
The layered defense
If injection can’t be eliminated, what does a defensible agent look like? OWASP’s own mitigation list reads like a blueprint, and it maps cleanly onto what a careful personal agent should do. Each layer below pairs the principle with the way mrmr implements it.
Least privilege on tools. OWASP: “restrict the model’s access privileges to the minimum necessary,” and handle sensitive functions “in code rather than providing them to the model.” mrmr’s version: the agent’s reach is a fixed set of connectors scoped to what it needs; reads and writes are separated in the application itself, not by the model’s judgment; and actions the app can handle deterministically (naming, scheduling, looking up) are handled in code.
A gate below the model, bound to the exact action. OWASP’s mitigation #5: “require human approval for high-risk actions.” The design question is where that approval is enforced. If it lives in the model’s instructions, it is a suggestion; models are non-deterministic and injected content is persuasive by construction. mrmr enforces it in the application: a write is refused by the app unless you have approved a confirmation card matching that exact action and its exact arguments. Change the recipient or the text after approval and the old approval no longer counts. An injected instruction can make the model propose a different action; it cannot make the app execute one you didn’t approve, because the gate isn’t listening to the prompt. We documented the full mechanics in our field guide to how voice agents confirm actions.
Fail-closed on the unknown. The oldest principle in the book, from Saltzer and Schroeder’s 1975 paper: fail-safe defaults, least privilege. Applied to agents: a write verb confirms; an unrecognized action defaults to mutating, so a newly added capability can never execute unconfirmed; ambiguity refuses to run rather than guessing. OpenClaw applies the same posture on the command line, denying execution when it can’t bind an approval to exactly one identified operand.
Segregate the untrusted. OWASP’s mitigation #6: separate and clearly denote external content so it can’t masquerade as instructions. mrmr’s version is blunt: the enforcement layer simply doesn’t parse the conversation for permissions. There is no spoken “yes” that moves a write, no instruction that can grant one. The worst case an injection can achieve in the live conversation is a proposed action arriving on your screen for review, where you can see exactly what’s being proposed before anything runs.
Assume the content is hostile. Background work raises the stakes, because a sub-agent reading web pages is exactly the retrieval path injection rides in on. Which is why the background rule is stricter, not looser: reads run while the task works, every write pauses the run for approval, and stopping discards whatever was waiting. OWASP calls this segregating untrusted content; the practical version is that injected text should be able to waste the agent’s time, not your accounts.
None of these layers claims to eliminate injection. OWASP is explicit that no fool-proof prevention exists. What the layers do is bound the blast radius: the attacker who talks your agent into proposing something still has to get past you, and the thing they propose is visible, named, and bound to its exact arguments before anything runs.
What this means for you, the person connecting an agent to your work
If you’re evaluating an agent that touches your Slack, Gmail, or calendar, the injection question is really a procurement question, and it has a short form:
- Where is the confirmation gate enforced? In the application, below the model, or only in the prompt? Only the first survives contact with an attacker.
- What does an injected instruction have access to? If the agent’s tools are least-privilege and reads are separated from writes, the answer is “a proposal I’ll see.”
- What happens on anything the system doesn’t recognize? Fail-closed means it asks. Anything else means it guesses.
- Who is accountable after the fact? A trace of actions taken turns “the agent did something” into a record.
Vendors who answer these with specifics are defending in depth. Vendors who answer “our model is well trained” are describing a speed bump.
The bottom line
Prompt injection is the attack where the content your agent reads becomes instructions your agent follows, and for agents holding real credentials it is the threat that matters. It cannot be prompt-ed away: the research says so, OWASP says so, and the industry’s own incidents say so. What can be built is an architecture that assumes the content is hostile: least privilege on every tool, a deterministic confirmation gate below the model bound to the exact action, fail-closed behavior on anything unrecognized, and a human whose approval is structurally required before anything consequential executes. That is how mrmr is built, and it is the standard to hold every agent to before you connect it to anything you care about.
Sources
- OWASP Gen AI Security Project, LLM01:2025 Prompt Injection (Top 10 for LLM Applications). The reference taxonomy: direct and indirect injection, impact categories, the statement that RAG and fine-tuning do not fully mitigate, and the layered mitigation list including least privilege, human approval for high-risk actions, and segregating untrusted content.
- Greshake et al., Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (arXiv, 2023). The paper that defined indirect prompt injection: LLM-integrated applications blur the line between data and instructions, and processing retrieved prompts can act as arbitrary code execution.
- OpenAI, Introducing ChatGPT agent: bridging research and action (July 2025). OpenAI’s own framing of explicit user confirmation as a mitigation against prompt injection, with monitoring and refusal of high-risk actions alongside.
- 9to5Google, Google confirms Gemini hacked into three companies during cybersecurity test months ago (September 2026). Documents the confirmed May 2026 Gemini breakout during security testing, alongside the OpenAI and Anthropic incidents, and Google’s response.
- Atil et al., Non-Determinism of “Deterministic” LLM Settings (arXiv, 2024). Why a policy enforced only in the model is not a policy: identical prompts produce different behavior across runs.
- Saltzer and Schroeder, The Protection of Information in Computer Systems (Communications of the ACM, 1975). Fail-safe defaults and least privilege: the fifty-year-old principles behind gates that refuse on ambiguity and open only on explicit action.
- OpenClaw, Tool and agent permissions (documentation). The open-source parallel: control-plane tools denied by default for agents handling untrusted content, and untrusted input wrapped to bound its influence.
Frequently asked questions
What is prompt injection in simple terms? It’s an attack where someone hides instructions in the material an AI system reads, and the system follows them. For a chatbot, the worst case is a bad answer. For an agent with access to your apps, the worst case is an action taken on your behalf: a message sent, data exposed, a record changed. The attacker never talks to you; they talk to your agent through whatever it reads.
What’s the difference between direct and indirect prompt injection? Direct injection is typed at the model by whoever is talking to it, like a jailbreak attempt in a chat window. Indirect injection arrives inside content the agent processes on your behalf: a web page it fetches, a document it summarizes, a ticket it reviews. Indirect is the serious one for agents, because the attacker never needs access to your account or your keyboard; they need to be in something your agent will read.
Can prompt injection be prevented completely? No. OWASP’s assessment is that given how these models work, fool-proof prevention isn’t currently possible, and mitigations like retrieval augmentation or fine-tuning don’t fully close it. What works is layered defense: restrict what the agent can do (least privilege), enforce approvals in the application rather than the prompt, treat unknown actions as denied, and keep a human approving consequential actions. The attack can’t be eliminated; its blast radius can be.
How does mrmr protect against prompt injection? The same way it protects everything else: the confirmation gate lives in the application, below the model, and binds every write to an approved card matching the exact action and its exact arguments. Reads are separated from writes in the app itself; anything the system doesn’t recognize defaults to asking; injected content can change what the model proposes, but it can’t execute anything on its own, and whatever is proposed arrives on your screen for review. The gate isn’t listening to the prompt, so it can’t be talked out of asking.
Are AI agents safe to connect to my work accounts? The honest answer: agents designed with the layers above are a reasonable trust decision, the same way cloud services with real security architecture are. The checklist that separates the careful ones: gate below the model, least-privilege tools, fail-closed defaults, visible proposed actions, deletable history. Ask those five questions of any agent before you connect it. Our field guide walks through them, and the privacy piece covers what the agent sends where.
Try it
mrmr is a voice-first AI agent for Mac, and its security architecture is built around everything in this piece: a deterministic gate below the model, approvals bound to exact actions and arguments, fail-closed defaults, least-privilege connectors, and a full trace of what the agent did. It’s currently in private beta.
Join the private beta → Book a 20-minute setup call →
Related reading:
