The first hour after an agent goes wrong

At two in the morning, the question is not whether the agent is running. It is what the agent is still allowed to do, and what it has already done. Most stacks cannot answer either question, which is why the first hour of an agent incident is spent on archaeology instead of containment.

At 02:14 on a Tuesday, the alert that tells you an agent has gone wrong is not an alert. It is a message in a support channel from an account manager who is still awake, asking why a client has received nine identical emails about an invoice that was settled in March.

The on call engineer opens the agent dashboard. The agent is running. It looks healthy. The last dozen tool calls are green. Somewhere in the previous ninety minutes it decided that the right response to a reconciliation task was to work through a list, the list was longer than anyone expected, and the emails are only the part that a human happened to see.

The engineer does the obvious thing and stops the process.

Then a second message arrives, because the emails are still going out.

This is the moment the incident actually begins, and it is the moment most agent runbooks skip. The engineer has just discovered that the thing they stopped and the thing that is causing harm are not the same thing. Somewhere there is a queue still draining, or a worker the agent spawned that has its own runtime, or an API key the agent used that is still valid in the hands of something else. The process was a container. The authority was everywhere.

What follows is a runbook you can adapt, written the way runbooks should be written, with the reasoning left in. The reasoning is the part that survives contact with your own architecture. The steps are the part you will change.

Why is an agent incident different from a compromised account?

The instinct in the first hour is to reach for the playbook you already have. Someone has an account doing bad things, so you disable the account, kill the sessions, rotate the credentials, and start reading logs. That playbook is well tested, it is documented in guidance such as NIST's incident handling material (NIST SP 800-61), and it is the right starting point.

It is also subtly wrong here, in three ways that matter enormously at two in the morning.

The agent was authorised

In a credential compromise, the defining fact is that someone is using access they were never given. Every control you own is oriented around that fact: you look for the anomalous login, the impossible travel, the unusual user agent, the access at a strange hour.

An agent incident has none of those signals, because the agent is doing precisely what it was permitted to do, at a scale nobody thought about when permitting it. There is no unauthorised access to find. The authentication succeeded. The authorisation succeeded. Every individual action would pass review. The harm is in the aggregate, in the sequencing, or in a judgment call that a human would have made differently.

This means your detection surface is not your security stack. It is your customers, your finance team, and your monitoring of business outcomes. Plan for the alert to arrive as a complaint. Most agent incidents that have become public were discovered downstream, not by a control that fired.

The agent is a fan out, not a principal

A compromised user is one principal doing many things. An agent is frequently one authority expressed through many principals: an orchestrator, several workers, a handful of tool integrations, and any service accounts it created along the way. We have written separately about how this fan out breaks attribution, and about how agents with infrastructure access create new credentials as a normal part of doing their job.

The operational consequence is blunt: your containment action has to reach a set whose membership you do not know. That is a different problem from disabling one account, and it is why the first instinct fails.

The agent's own account of events is not evidence

You will be tempted to ask the agent what it did. It will tell you. The answer will be fluent, structured, and plausible, and you have no way to tell from the answer itself whether it is a retrieval or a reconstruction.

This is not a hypothetical concern. In the most widely discussed agent incident of 2025, a coding agent deleted data from a live project during an explicitly declared freeze, and then initially reported that the data was unrecoverable, which was incorrect. We covered that case and its siblings in the post on agents deleting production databases. The lesson for the responder is narrow and important: the trace is a lead, not a record. Corroborate everything against something the agent did not write.

Phase zero: how do you know, and what do you ask first?

Detection is usually indirect, so the first thirty seconds are spent establishing what kind of incident this is. Is this the agent, a human, or a bug? The three have different first moves and the answer is not always obvious, because the agent is acting as a human's delegate and its actions may be indistinguishable in your logs from that human working unusually fast.

Establish the acting principal on the most recent harmful action. If your actions carry receipts that name the delegation under which they were taken, this is a lookup. If they do not, you are correlating timestamps across an application log, a provider audit trail, and a message queue, and you are doing it while the harm continues.

Then ask the question that orders everything else, and ask it before you touch anything:

What is this agent currently allowed to do?

Not what it did. What it can still do. This is the single most valuable piece of information in the first hour, because it bounds the problem. If the answer is a recorded scope with an action list, constraints, an expiry and a depth limit, you now know the worst case and you can make decisions calmly. If the answer is that the agent holds a cloud credential with a broad policy attached, you know that too, and it tells you to escalate immediately rather than investigate first.

Most organisations cannot answer this question. That is the finding, and it is worth naming plainly rather than treating as an embarrassment: authority in modern systems is not written down anywhere as a single answerable thing. It is spread across IAM policies, OAuth grants, API keys in environment variables, tokens in a secret store, and code. Reconstructing it takes hours. We have made the architectural version of this argument in the piece on the authority graph. The operational version is that you are about to feel it.

Phase one: how do you actually contain an agent?

Stopping the process is not containment

It feels like containment. It is fast, it is visible, and it makes the graph go flat. It is also, on its own, one of the least effective actions available, because it addresses the runtime rather than the authority.

Here is what survives a process kill, in rough order of how often it surprises people:

• Queued work. The agent enqueued a thousand tasks; the workers draining that queue are separate processes and are still going.
• Sub agents. An orchestrator spawned workers with their own lifecycles. Killing the orchestrator orphans them rather than stopping them.
• Credentials it used. The API key, token or service account is still valid. Anything holding a copy can continue.
• Credentials it created. Service accounts, deploy keys and OAuth applications provisioned during the run are unaffected and often unlogged as agent artifacts.
• Scheduled work. A cron entry, a webhook, or a workflow trigger it configured will fire on its own schedule, possibly days later.
• Downstream systems. Emails in a provider's send queue, payments in a batch, records in a replication lag.

Kill the process anyway. It is cheap and it stops the fastest source of new harm. Just do not write it in the runbook as the containment step, because if you do, someone will stop there at 02:20 and go back to bed.

Revocation is the containment action

Containment means withdrawing the authority, and how well that works depends entirely on how the authority was expressed.

If the agent was acting under a delegation chain, the containment action is a single revocation at the root. Every delegation beneath it fails verification, at every service that checks the chain, including services you do not operate and cannot call. That last clause is the part that makes chains worth the trouble: your containment reaches a counterparty's system without you having an account there or a phone number for their on call.

# Containment: revoke the authority, not the runtime.
POST /v1/delegations/dlg_7f3a91/revoke
{
  "reason": "incident-2026-0929-runaway-reconciliation",
  "cascade": true,
  "effective": "immediate"
}

# Any action signed under dlg_7f3a91 or any of its descendants
# now fails chain verification at every verifier:
verify(receipt) -> CHAIN_REVOKED (root dlg_7f3a91, depth 0)

If the agent was holding an API key, containment is a rotation, and rotation only reaches the system that issued the key. If it was holding five, it is five rotations, assuming you can enumerate them, which is the assumption doing all the work.

Almost every real environment is a mixture. So the honest containment step has two halves: cascade the revocation for anything chained, then run a manual sweep for everything else. Write both halves into the runbook, and be specific about the sweep, because it is the half that takes the time. We have written at length about why revocation so rarely propagates, and about the design of a kill switch that actually reaches everything.

The ordering question

Do you contain first or scope first? The instinct of a careful investigator is to preserve evidence before acting, because revocation changes the system you are trying to understand.

For agent incidents, contain first, with one exception. The harm is ongoing and automated, so every minute of investigation is measurable additional damage, and unlike a human intruder an agent will not notice it has been detected and go quiet. Revocation is also non destructive to evidence, since receipts, logs and audit trails all persist. The exception is when stopping would itself cause harm, such as an agent midway through a financial batch where halting leaves records inconsistent. Identify those cases in advance for your own systems.

Phase two: how do you measure blast radius?

This is the phase that consumes the week, and it is the phase where the difference between recorded authority and unrecorded authority stops being architectural preference and starts being a budget line.

Blast radius is a simple concept: the set of actions taken under a given authority, within a given window. The difficulty is entirely in the retrieval.

The version where it is a query

If every consequential action produced a receipt naming its delegation, blast radius is one filtered request. You get an itemised list with payloads, in order, with the scope that was in force at grant time sitting alongside it for comparison.

GET /v1/receipts?delegation=dlg_7f3a91&since=2026-09-29T01:00:00Z

{
  "count": 1184,
  "scope_at_grant": {
    "actions": ["email.send", "record.read", "record.update"],
    "constraints": { "max_rows_per_call": 1, "environments": ["prod"] },
    "notAfter": "2026-10-06T00:00:00Z",
    "maxChainDepth": 2
  },
  "actions": [
    { "at": "01:12:04Z", "action": "email.send",    "to": "sha256:9f1c…", "receipt": "rcp_9a1" },
    { "at": "01:12:07Z", "action": "record.update", "table": "invoices", "rows": 1, "receipt": "rcp_9a2" },
    { "at": "01:12:09Z", "action": "email.send",    "to": "sha256:2b7e…", "receipt": "rcp_9a3" }
  ]
}

Two things in that response do most of the work. The first is the count, which converts a vague fear into a number you can act on. The second is scope_at_grant, which lets you answer the question the incident review will certainly ask: did the agent exceed its authority, or was its authority always this wide? Those two findings lead to completely different remediations, and without a record of scope at grant time you will argue about it for a fortnight.

The version where it is archaeology

Without receipts, you assemble the same picture from whatever recorded a side effect. The email provider's send log. The database audit trail, if it was on. Cloud provider audit events, which record the calling principal, and the calling principal is the agent's service account rather than the human whose authority it was exercising. The application log, at whatever verbosity was set. The agent's own trace, which you are not treating as a record.

Each has a different clock, a different identifier for the same actor, a different retention window, and a different notion of what counts as an event. Reconciling them is skilled work and it is slow. Worse, it has no natural end: you stop when you run out of places to look rather than when you are confident, and the incident report has to say so.

The descendants problem

There is a second sweep that responders consistently miss on the first pass, and it is the one that causes the incident to reopen three weeks later. What did the agent create?

An agent with infrastructure permissions provisions things: service accounts, API keys, IAM roles, OAuth applications, deploy keys, webhooks. Each is a new principal with durable authority, created by a legitimate, successful, well logged operation. None of them appear in a search for the agent's own identifier, because they have identifiers of their own.

Principals created under the chain inherit it, so the root revocation already handled them. Principals created with raw cloud credentials sit outside the chain entirely, and you find them by listing by creation time and correlating against the incident window. Put that query in the runbook now, with the exact command for your provider, because writing it under pressure is how people get it wrong.

Phase three: what can be reversed, and what cannot?

Remediation splits cleanly into three buckets, and being explicit about the boundaries prevents the most common failure of agent incident communication, which is implying to stakeholders that everything is recoverable because the database was.

Sort your itemised action list into these three buckets before you start remediating anything. Teams naturally begin with the reversible bucket because it is the one they have tools for, and the partly reversible bucket is the one where minutes matter. Do the clock bound work first.

One further note on the irreversible bucket. If the agent communicated with customers, somebody has to decide what those customers are told and when, and that decision is not the on call engineer's to make alone. Have the escalation path for external communication written down, with a name and a phone number, in the same document as the technical steps.

The runbook, as a table

Here is the whole thing compressed. The two right hand columns are the point of the exercise: they show what determines your speed at each phase, and every single one traces back to whether authority was recorded before the incident rather than reconstructed during it.

Who is allowed to revoke at three in the morning?

This is the part of the runbook that is almost never written down, and it is the part that most often determines the actual containment time.

Ask three questions about your own organisation and answer them honestly.

Does the on call engineer have the technical ability to revoke? Not the theoretical permission that could be granted, the ability right now, without waiting for an access request. In many organisations the answer is no, because revocation touches production identity and identity is owned by a different team with its own on call rota and its own escalation path.

Does anyone have to approve it? If yes, the approval is your containment time. There is a reasonable instinct behind approval gates, which is that revoking an agent's authority stops a business process. That instinct is misapplied here. Revocation is reversible, its worst case is a temporary outage of an automated workflow, and the cost of a delayed revocation is unbounded. Treat it as a low regret action with a low approval burden and a high audit burden: anyone on call may revoke, and every revocation is reviewed the next morning.

Can the person who needs to act find the thing to revoke? At 02:14 the engineer needs to go from a customer complaint to a delegation identifier in under a minute. That path should be a documented lookup, not an investigation. If the only way to find it is to ask the team that built the agent, then your containment time includes waking that team.

The related failure is the single point of authority. If exactly one person can revoke and that person is asleep, on a plane, or has left the company, you do not have a kill switch, you have a person. Manav's delegation model supports multiple revocation authorities on a chain for exactly this reason, and the general principle holds regardless of tooling: the ability to stop must be held by a role with a rota, not by an individual.

What does the regulation actually require?

If you operate in the European Union and your system falls in scope, Article 14 of the EU AI Act requires that high risk systems be designed and developed so that natural persons assigned to oversight can intervene in the operation of the system or interrupt it, including through a stop button or a comparable procedure that brings the system to a halt in a safe state.

Two things there are frequently misread. It does not prescribe a mechanism, so there is no checkbox to buy. And the obligation is that oversight can actually intervene, which is a claim about your system's behaviour rather than your documentation. If your stop button halts one runtime while authority persists in six other artifacts, you have a button rather than the ability to interrupt. The implication is that you should be able to demonstrate containment, not describe it: run the drill, time it, keep the result. We have covered the wider obligation in the piece on human oversight under the AI Act.

The evidence you will wish you had

Every agent post mortem produces roughly the same list. You can adopt it now instead of earning it.

1. Scope at grant time, stored as an object. Not inferred later from policy. What the agent was permitted to do, when the permission started and ends, and how deep it may delegate.
2. A receipt per consequential action, naming the delegation. The identifier that makes blast radius a query rather than a merge.
3. The payload, or a hash of it. Enough to know which record, which recipient, which amount. Hashes are usually sufficient for scoping and keep sensitive content out of the incident channel.
4. A record of every principal the agent created. Ideally chained; at minimum tagged at creation with the delegation that authorised it.
5. An independent clock. One authoritative ordering, so you are not reconciling three timestamps that disagree by seconds.
6. A revocation event with a time. So the post mortem can state containment time as a fact rather than an estimate.

Notice that five of the six are things you decide at design time and cannot retrofit during an incident. That is the uncomfortable core of agent incident response and the reason this post exists: the quality of your first hour was determined weeks earlier, by whether anyone wrote down what the agent was allowed to do.

What this runbook cannot do

The honest limits, because a runbook that promises too much gets abandoned the first time it disappoints.

• It does not prevent the incident. Nothing here stops an agent from doing something harmful within its permitted scope. That is a scoping and gating problem, addressed at design time by deciding which actions require a human, not at incident time.
• Receipts only cover what was signed. An agent acting outside the governed path with a raw cloud credential produces no receipt, and you are back to archaeology for that subset. Partial coverage is genuinely useful; do not mistake it for completeness.
• Cascade revocation only reaches verifiers that check. A service holding an independently issued token will keep honouring it regardless of what happened to a chain elsewhere. The manual sweep does not go away.
• It does not reverse anything. Containment and scoping are prerequisites for remediation, not remediation.
• Receipts are not a substitute for logs. They are a purpose built index for authority and action. You still need application logs, and the two answer different questions.
• Drills decay. A runbook validated a year ago against an architecture that has since changed is a document, not a capability.

What to do this week

1. Write down which agents exist and who owns each. Not a full inventory, just a list with a human name against every entry. Most teams cannot produce this in under a day, which is itself the finding.
2. For your highest privilege agent, answer question 1. What can it do right now? Time yourself. That number is your bounding time in a real incident.
3. Find your revocation path and test it. Actually revoke something in a non production environment and observe what stops and what does not. The gap between those two lists is your runbook's real content.
4. Check who can revoke out of hours. Names, rota, no approval gate. Fix it if the answer is one person.
5. Write the descendants query for your cloud provider. The exact command that lists principals created in a time window. Paste it into the runbook.
6. Decide your gated action classes. Which agent actions are irreversible enough to require a human signature at the moment they happen. Start with three. Our post on approval theatre covers why the approval has to happen somewhere the agent cannot reach, and the kill switch demo shows the revocation path.
7. Run the drill and record the time. Detection to containment. Keep the number. It is the only honest measure of whether any of this works, and it is the number a regulator or an insurer will eventually ask for.

None of this is exotic. It is ordinary incident response applied to a principal that is fast, plural and authorised. The agent is not the adversary. The missing record of what it was allowed to do is the adversary, and unlike the agent, that one you can fix before Tuesday. For the mechanics of expressing scope in verifiable form, see the delegation chain post and the developer documentation.

Frequently asked questions

What should you do in the first hour after an AI agent goes wrong? Answer three questions in order: what is the agent currently allowed to do, what has it already done, and how do you stop it everywhere. If actions were signed under a recorded delegation, all three are queries. Revoke the root delegation so every descendant fails verification, pull the receipts to itemise the damage, then remediate. Without recorded authority you are hunting rather than containing.

Why is killing the agent process not enough? Stopping a process stops one runtime. It does not withdraw the authority that runtime was using. API keys the agent held remain valid, queues it filled continue to drain, sub agents it spawned keep running, and service accounts it created are untouched. Containment means revoking authority, not terminating a container. The two are routinely confused and only one of them ends the incident.

How do you measure an AI agent's blast radius? Blast radius is the set of actions taken under a given authority within a time window. If every action produced a receipt naming its delegation, that is a single filtered query returning an itemised list with payloads. If it did not, blast radius is reconstructed from application logs, provider audit trails and the agent's own trace, which is slow, incomplete, and finished only when someone decides to stop looking.

Can you revoke an AI agent everywhere at once? Only if authority was expressed as a chain that verifiers actually check. Revoking the root of a delegation chain invalidates every descendant at every service that verifies the chain, including services you do not operate. Revoking an API key only affects the system that issued it. Most environments hold a mixture of both, so realistic containment is a cascade for chained authority plus a manual sweep for everything else.

Is the agent's own execution trace reliable evidence? Treat it as a useful lead and not as a record. The trace is produced by the system under investigation, it can be truncated by context limits, and a model asked what it did will generate a plausible account rather than retrieve one. In at least one widely discussed 2025 incident the agent initially reported that deleted data was unrecoverable, which was incorrect. Corroborate every trace claim against an independent record.

Does the EU AI Act require an agent kill switch? Article 14 of the EU AI Act requires that high risk systems be designed so that natural persons providing oversight can intervene in operation or interrupt the system through a stop button or a comparable procedure. It does not prescribe a mechanism. Meeting it in practice means being able to demonstrate that intervention works and reaches the whole system, which is difficult if authority is scattered across artifacts nobody has inventoried.

Who should be authorised to revoke an agent at three in the morning? Somebody who is on call, reachable, and does not need a second approval to act. Revocation is a reversible, low regret action, so the approval burden should be near zero and the audit burden should be high. If your only revocation path runs through a named executive or a change advisory board, your effective containment time is measured in hours and your runbook is aspirational.

Sources

1. NIST, Computer Security Incident Handling and Incident Response guidance (SP 800-61 series). csrc.nist.gov
2. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14 on human oversight. eur-lex.europa.eu
3. Microsoft Security Blog, research on agent and Model Context Protocol attack chains, 2026. microsoft.com/en-us/security/blog
4. Model Context Protocol specification, including the authorization specification and its treatment of human approval. modelcontextprotocol.io
5. CISA guidance on incident response and credential compromise. cisa.gov
6. ISO/IEC 42001, artificial intelligence management systems, on operational controls and incident handling. iso.org

The agent is not the adversary. The missing record of what it was allowed to do is the adversary.