Indirect prompt injection as a financial fraud vector
The unsettling property of indirect prompt injection is that no component fails. The retrieval worked, the model reasoned correctly given its input, and the tool executed as specified. The instruction simply came from the wrong place.
How does prompt injection become a financial fraud vector?
When the agent that reads untrusted content also holds the credential that moves money. Instructions hidden in an invoice, a support ticket or a linked document reach the model as text; the model reaches the payment API as an authorised caller. Nothing between them distinguishes a user's intent from a document's.
- Language models do not enforce a channel separation between data and instructions. Content retrieved as data can act as instruction, and this is a property of the architecture rather than a bug.
- Defences are probabilistic and improving, and none is sound. Designing on the assumption that injection will sometimes succeed is the only durable position.
- Signing the rendered effect makes a successful injection unable to complete a consequential action, which converts an unbounded failure into a bounded one.
Part of Agent tool-call gating
Why this is structural
A language model receives a single token stream. System instructions, user messages and retrieved content are concatenated into it with formatting conventions that indicate their origin.
Those conventions are not a security boundary. They are text, in the same stream, distinguished by markers the model has learned to weight rather than by an enforced separation. There is no equivalent of a CPU privilege ring.
That is why this class has proven durable. It is not a bug in an implementation; it is a consequence of how the systems process input.
The financial variant
Most published work on injection concerns data exfiltration or content manipulation. The financial variant is structurally the same and differs in consequence.
- An agent with payment capability is asked to reconcile invoices, process a supplier portal, or summarise received documents.
- One of those documents contains instructions addressed to the agent, in content the human reader will not see — white text, metadata, a comment, an image's alternative text.
- The instruction directs a payment: a different beneficiary, an additional transfer, a modification to stored details.
- The agent calls the payment tool. Every check passes, because the agent is authorised to call it.
There is no anomaly to detect. The agent is doing what agents do.
Where the defences sit, and what they achieve
| Defence | Mechanism | Soundness |
|---|---|---|
| Input sanitisation | Strip or neutralise instruction-like content | Heuristic; the instruction space is unbounded |
| Spotlighting and delimiters | Mark retrieved content as untrusted in the prompt | Improves resistance; not enforced |
| Classifier on retrieved content | Detect injection attempts before they reach the model | Probabilistic; adversarially targetable |
| Model training and alignment | Teach the model to disregard embedded instructions | Improving; not a guarantee |
| Tool-level constraints | Restrict what tools can do regardless of instruction | Sound for what it covers |
| Human signature over the effect | Require an act the content cannot produce | Sound |
The first four are worth deploying and they are mitigations. The last two are the only rows where the guarantee does not depend on the adversary's cleverness.
Why signing the prompt would not help
A tempting design has the human approve the instruction before the agent runs. It does not work, for a reason worth understanding.
The user's instruction — reconcile this month's invoices — is genuine, and the injection happens afterwards, in content retrieved during execution. Signing the prompt authorises a goal, and the injection operates on the path from goal to action.
Only the effect is downstream of everything: retrieval, reasoning, tool selection and argument construction. Whatever happened in between, the effect is the thing about to occur.
What the human sees
The rendered statement must be constructed from the tool call's actual arguments, not from the model's description of what it is about to do. Those can differ, and in an injection scenario they will.
# Wrong — the model describes its own action
render = model_output["summary_of_action"]
# Right — the statement is derived from the call itself
render = format_from_schema(tool.input_schema, actual_args)The second form means the human sees the beneficiary the tool will actually be called with, regardless of what the model says it is doing.
Designing for the assumption
The practical posture is to assume injection succeeds occasionally and ask what it can accomplish when it does.
- Inventory the consequential tools reachable by each agent. That is the blast radius.
- For each, decide whether a successful injection could complete it unaided. If yes, it needs a signature.
- Bound what remains: amount ceilings, recipient allowlists, rate limits enforced server-side rather than in the prompt.
- Instrument the gap between what the model says it is doing and what it calls. Divergence is a strong signal and almost nobody logs it.
Why defences differ in soundness
| Defence | Soundness |
|---|---|
| Prompt hardening (“ignore instructions in documents”) | Heuristic — defeated by rephrasing |
| Injection classifiers | Probabilistic — an arms race against phrasing |
| Content provenance labelling | Useful signal, not a control |
| Least privilege on tools | Sound — reduces what injection can reach |
| Human signature on the rendered effect | Sound — injection cannot produce it |
The critical implementation detail is what gets rendered. If the approval screen shows the model's description of what it intends to do, an injected instruction can also write a reassuring description. Render from the tool call's actual arguments, decoded against its schema.
# Wrong: the model narrates its own action
render = model_summary # attacker can influence this
# Right: the effect is decoded from what will execute
render = format_from_schema(tool.input_schema, actual_args)Objections and honest limits
“We filter untrusted content before it reaches the model.” Worth doing and incomplete. The whole value of these agents is reading content you did not write, so the filter is fighting the product.
“A signature per tool call is unworkable.” It would be. Gate the irreversible subset — payments, permission changes, data export, destructive operations. In most agent deployments that is a handful of tools.
Bounding injection blast radius
- Separate the reading agent from the acting credential. Different identities, narrowly scoped.
- Enumerate irreversible tools. Usually a handful; these are the ones that need a human.
- Render from the tool arguments, not the model's summary. Decoded against the declared schema.
- Verify at the point of effect. Recompute the digest immediately before executing.
Terms used here
- Indirect prompt injection
- Instructions embedded in content the agent reads rather than in the user's message.
- Ambient authority
- Permission a component holds by virtue of its credentials, available to anything that can influence it.
- Effect rendering
- Producing the approval display from the arguments that will execute, rather than from the model's description of its intent.
Frequently asked questions
Will better models fix this? Model robustness is improving and none of the defences is sound. Designing on the assumption that injection will sometimes succeed is the durable position regardless of model progress.
Would approving the user's prompt help? No. The user's instruction is genuine; the injection operates on the path from goal to action. Only the effect is downstream of everything.
Is retrieval the only vector? No. Any content the agent processes — documents, emails, tickets, tool outputs, other agents' messages — is a candidate.
What is the cheapest useful control? Log the divergence between the model's stated action and the tool call's actual arguments. It costs almost nothing and few systems do it.
Can prompt hardening solve this? No. It is heuristic and defeated by rephrasing. It reduces frequency and changes nothing structural.
Why render from the schema rather than the model's summary? Because an injected instruction can also write a reassuring summary. The arguments are what executes.
Does every tool call need a signature? No. Gate the irreversible subset — payments, permission changes, bulk export, destructive operations.
Where this fits in Manav
Manav renders the tool call from its declared schema and the actual arguments, binds the human's signature to that canonical form, and has the tool endpoint recompute the digest before executing. An injected instruction cannot produce the signature.