An AI called your restaurant. Was there a person behind it?

Businesses have started answering calls from AI voice agents, and their own AI receptionists have started answering them back. Neither side can establish the thing that actually matters, which is whether a real human asked for this and will turn up.

A host at a forty cover restaurant picks up the phone on a Tuesday afternoon. The voice on the other end is polite, unhurried and completely fluent. It would like a table for four on Saturday at eight, and it has a slight preference for the back room. It answers a clarifying question about dietary requirements without hesitating.

The host takes the booking. Something about the call was very slightly off, in a way that would be hard to describe to a colleague. The pauses were a fraction too even. But there is a queue at the door and the call was entirely reasonable, so the table goes in the book.

On Saturday, nobody comes.

Here is the part that matters. The host has no way to know which of three things happened. A real person may have asked their assistant to book a table and then forgotten to cancel, which is a normal customer problem that restaurants have always had. Or an agent may have been booking tables speculatively across a dozen restaurants for a user who intended to keep one, which is a newer problem and a much more expensive one. Or there may have been no person at all, and the table was taken by software with a commercial interest in holding it.

Those three outcomes require completely different responses from the business, and the phone call gives it no way to tell them apart. It never did give it a way, but until recently the cost of dialling a number and speaking convincingly was a human minute, and that cost did the work of a control.

What is actually documented here, and what am I reasoning about?

This topic is new enough that it deserves an unusual amount of care about the difference between evidence and inference, so let me separate them explicitly before going further.

Documented: AI voice agents that place and answer calls on behalf of users and businesses are commercially available and in production use for reservations, appointments, rescheduling and customer service. Synthetic voice is also used for fraud, which is the subject of a separate piece on contact centre deepfakes. New York enacted a Restaurant Reservation Anti-Piracy Act, signed in December 2024 and effective from February 2025, restricting third party services from listing or selling restaurant reservations without a written agreement with the restaurant, which tells you the reservation hoarding problem was serious enough to legislate. The Federal Communications Commission has ruled that AI generated voices in robocalls fall within the existing prohibitions of the Telephone Consumer Protection Act, and has pursued enforcement over a synthetic voice political robocall.

Also documented, though I would treat the numbers with care: surveys of contact centre leaders have consistently reported that a majority regard synthetic voice as a serious threat and doubt their ability to detect it. Those surveys are generally commissioned or published by vendors selling detection, and they are self reported, so use them as directional evidence of concern rather than as measurement.

Not documented, and I will not pretend otherwise: I have no reliable figure for what share of inbound calls to businesses currently originate from agents, nor a published incident where a business suffered a specific quantified loss by accepting an agent call. Voice agent vendors publish claims about reducing missed calls and no shows, and those claims are marketing material from parties with an obvious interest, so I am not going to repeat their numbers.

What follows is therefore an argument about a structure, built from mechanisms that are well understood in adjacent domains, rather than a report on measured losses. The structure is sound and the losses are coming. That is the honest framing.

Why is this different from the deepfake problem?

Because the caller is not pretending to be anyone.

The contact centre deepfake problem is impersonation: an attacker clones a customer's voice to convince an agent they are that customer. The defence, such as it is, has been voice biometrics and liveness analysis, and the reason that defence is losing is the same reason detection loses everywhere in this series. The generator improves faster than the classifier, and the attacker gets unlimited attempts with feedback.

An agent caller is a different situation entirely. It is not claiming to be a human. It says, quite openly, that it is calling on behalf of someone. The question is no longer "is this voice genuine" but "is there a principal, who is it, and did they actually ask for this". That is not a signal processing problem and no amount of audio analysis will answer it, because the answer is not present in the audio at all.

This distinction is worth holding on to, because it changes what a solution can look like. Impersonation might in principle be caught by a sufficiently good detector. Absent authority cannot be, because there is nothing to detect. A perfectly ordinary sounding agent with no human behind it produces exactly the same waveform as a perfectly ordinary sounding agent with a real customer behind it.

The three callers a business now has to distinguish

Look at the middle three rows. They are acoustically indistinguishable, and they have opposite correct responses. No listening based control can separate them, and no amount of staff training will either, because there is nothing for the staff to hear.

Why can a business not simply refuse to take agent calls?

Because a meaningful and growing share of its customers will be calling that way, and refusing them is refusing revenue.

It is worth being honest about the incentives on all three sides, because a proposal that ignores any of them will not survive.

The business wants to fill tables, appointment slots and service capacity, and would prefer to do it without absorbing bookings that evaporate. A restaurant that adopts a blanket policy of hanging up on anything that sounds synthetic will hang up on real customers, including customers using accessibility tools and customers whose assistants are entirely legitimate. It will also fail, because voice agents are getting better at sounding human, so the policy degrades into a coin flip performed by a stressed host during service.

The agent operator wants its agent to work. Every business that refuses agent calls is a hole in the product, and the operator's incentive is therefore to make the agent as indistinguishable as possible from a human caller. That is the arms race dynamic that makes voluntary self identification unstable: any operator that discloses honestly is disadvantaged against one that does not, unless disclosure buys something.

The user simply wants their table. They do not care about the mechanism, and they will be annoyed at whichever party gets in the way.

The only stable resolution is one where disclosure is rewarded rather than punished. If an agent that presents verifiable credentials gets served promptly, and an agent that presents nothing gets treated with the same suspicion as an anonymous caller, then honest operators have a reason to be honest and the arms race stops being one sided.

What would a verifiable agent caller actually present?

A delegation. Specifically, a signed object that says: this human authorised this agent to perform this class of action, within these limits, until this time, and here is how to check that offline.

The general shape of these chains is set out in the piece on delegation chains, and the same object works here. For a booking it might carry:

{
  "delegation_id": "dlg_9f24c7",
  "principal_key":  "ed25519:MCowBQ...",     // the human's public key
  "delegate_key":   "ed25519:MFkwEw...",     // the calling agent
  "scope": {
    "actions":  ["reservation.create", "reservation.cancel"],
    "party_max": 6
  },
  "constraints": {
    "concurrent_holds_max": 1
  },
  "notBefore": "2026-09-20T00:00:00Z",
  "notAfter":  "2026-09-27T00:00:00Z",
  "maxChainDepth": 1,
  "revocationId": "rev_44c1a0"
}

Read the constraints line, because it is doing the work that matters commercially. A delegation that permits one concurrent held reservation is a delegation that cannot be used to hold six tables across six restaurants on the same evening. The business is not detecting speculative booking behaviour after the fact. It is checking, before accepting, that the authority being presented does not permit it.

The business verifies the signature against the principal's published key, checks that the requested action falls inside the scope, checks the validity window and the revocation status, and then makes an ordinary commercial decision about a caller whose principal is now a known, accountable party. It does this without contacting the agent operator and without asking Manav anything, which matters for a restaurant on a bad connection at seven on a Saturday.

How does a cryptographic object travel over a phone call?

It does not, and any proposal that claims otherwise should be treated with suspicion. This is the honest engineering objection to everything above and it deserves a direct answer rather than a hand wave.

A voice call is a narrow band audio channel designed for speech. It cannot carry a signature. You are not going to have an agent read out a base64 encoded Ed25519 signature to a host holding a pen, and anything involving spoken hexadecimal is a fantasy that has never survived contact with a real telephone.

So the realistic design carries a reference over the voice channel and the actual verification happens elsewhere. The agent states a short delegation reference, or transmits it as tones, and the business's system resolves that reference over a data channel it already has, which is the internet. The call carries a pointer. The network carries the proof. This is the same pattern as a package tracking number: the number is short enough to say aloud and useless on its own, and the system it points into holds the real record.

For businesses whose phone system is already a software platform, which by now is most of them, this resolution can happen automatically while the call is still connected, and the host simply sees a verified marker appear next to the booking. For a restaurant with a handset and a paper book, it does not happen at all, and that is a real limitation rather than an implementation detail.

The account change case is different, and simpler

Everything above concerns bookings, where the worst outcome is an empty table. For a bank, a utility or an airline, calls also request account changes: update the address, add a payee, change the email, reset access. Those calls are where the money is, and the design there is both simpler and stronger.

Do not try to establish anything about the caller at all. Move the action off the call.

The agent handling the call, human or otherwise, initiates the change in their system, which triggers a signature request to the account holder's enrolled device. The account holder approves it there, seeing the actual change rather than hearing a description of it. The call becomes a conversation about what the customer wants, which is what a call is good for, and stops being an authorisation channel, which it was never good for.

This is the same control described in the contact centre piece and in the piece on bribed support staff, and it has the property that makes it worth the trouble: it does not matter who is calling. A cloned voice, a rogue agent, a legitimate agent, a real customer, or a bribed employee all hit the same wall, because none of them holds the account holder's device. The business stops needing to answer the question the phone call cannot answer.

Where does this sit with disclosure law?

Alongside it, and usefully so. Regulators have moved toward requiring disclosure when a caller is artificial, most visibly through the Federal Communications Commission's determination that AI generated voices in robocalls are covered by the Telephone Consumer Protection Act, and through various state level bot disclosure requirements.

Disclosure and verification solve different halves of the problem, which is worth spelling out. Disclosure tells the business that the caller is an agent, which is genuinely useful and which honest operators will comply with. It does not tell the business whether a human authorised the call, and it does not bind the disclosure to anything, so a caller with fraudulent intent simply does not disclose. Verification supplies the second half: not merely that this is an agent, but whose agent, for what, and until when.

The combination is what a business actually needs. A disclosed and verified agent is a customer arriving through a new channel. A disclosed but unverified agent is an unknown caller, to be treated with the caution any unknown caller warrants. An undisclosed agent is an operator that has told you what it is by concealing it.

What this cannot do

It needs adoption on both sides, and Manav has integrated with neither. Agent platforms would have to present delegations, and business telephony platforms would have to resolve and verify them. Neither exists today. The primitives that do exist are the delegation chain, the per action signature, the companion device flow and the offline verifiable receipt. Everything described here as a calling flow is a design, not a product.

It does nothing about a caller who presents nothing. A fraudulent caller will not offer a delegation, and the business falls back to whatever it does today, which is callbacks and knowledge questions and judgment. Verification raises the ceiling for cooperative callers and does not raise the floor for uncooperative ones. What it does do is make the absence of a delegation informative, which it currently is not.

It does not stop a real human authorising something regrettable. If a person genuinely tells their agent to hold four tables and pick one, the delegation will be valid and the business will still lose three covers. The constraint field can prevent that specific pattern if the business insists on it, but a determined user with several agent accounts is outside the model.

It does not help the restaurant with a paper book. A business with no software in the call path cannot verify anything, and a significant share of the businesses most damaged by reservation hoarding are exactly those businesses. Any honest account of this design has to admit that it reaches the well tooled first, which is the opposite of where the need is greatest.

It creates a two tier phone system if it is done carelessly. If verified agents get served and unverified callers get held in a queue, then the people harmed are those without agents, which skews old, poor and rural. Verification should buy speed for the verified, not penalties for everyone else, and a business that inverts that has built something worse than what it replaced.

What to do this week

1. Write down which inbound calls can change something. Separate calls that request information, calls that book capacity, and calls that alter an account. These are three different risk classes and most businesses treat them as one.
2. Move account changing actions off the phone entirely. This is the highest value action on the list and it does not require anyone else to adopt anything. The agent initiates, the customer confirms on their own device.
3. Stop training staff to identify synthetic voices. It does not work, it will work less well every quarter, and it makes staff responsible for a judgment they cannot make. Train them instead on which actions require confirmation through another channel.
4. Instrument no shows by booking channel. If bookings taken by phone from fluent, oddly even callers no show at three times the rate of walk ins, that is a number worth having before you need to argue for a policy.
5. Set a concurrency policy for bookings, and enforce it on identity rather than on phone number. A phone number is trivially many, and holding capacity across a market is the actual harm.
6. Check your state's position on bot disclosure and reservation resale. If you operate in New York, the reservation legislation already gives you standing you may not know you have.
7. Ask your telephony vendor what their plan is for agent callers. They will either have thought about it or be visibly surprised, and either answer is useful.

The delegation objects described here are documented in the developer documentation, and the agent authorisation flow can be seen in the agent demo.

Frequently asked questions

How can a business verify that an AI voice agent is calling on behalf of a real customer? Not by listening. The workable method is to require the agent to present a delegation reference during the call, which the business resolves over a data channel to a signed object naming the human principal, the permitted actions and the validity window. The call carries a pointer and the network carries the proof.

Should businesses accept calls from AI agents? Yes, for the same reason they accept online bookings. A growing share of customers will reach them that way, and refusing means refusing revenue. The useful distinction is not between human and agent callers, it is between callers who can demonstrate an accountable principal and callers who cannot.

Can voice biometrics detect AI callers? Increasingly poorly, and it is answering the wrong question anyway. Voice analysis attempts to determine whether a voice is synthetic. It cannot determine whether a human authorised the request, which is what matters, because a legitimate agent and a rogue agent sound identical.

Is it legal for an AI to call a business without saying so? Disclosure requirements vary by jurisdiction and are tightening. The Federal Communications Commission has determined that AI generated voices in robocalls fall under existing Telephone Consumer Protection Act restrictions, and several states have bot disclosure rules. Compliance is a separate matter from verification: disclosure tells you it is an agent, not whose agent.

What stops an AI agent from booking tables it will not use? A delegation with an explicit concurrency constraint. If the authority presented permits one concurrent held reservation, the business can decline a second without needing to detect a pattern across restaurants, because the limit is checked before the booking is accepted rather than inferred afterwards.

Does this replace callbacks and knowledge questions? For account changing actions, yes, and it should, because both are defeated by breached data and cloned voices. For callers who present no delegation, the business falls back to existing practice, so those procedures do not disappear. They stop being the only control for the callers who can do better.

Sources

1. New York State Senate, Restaurant Reservation Anti-Piracy Act, signed December 2024 and effective February 2025, restricting third party listing and sale of restaurant reservations without agreement. nysenate.gov
2. Federal Communications Commission, rulings and enforcement concerning AI generated voices in robocalls under the Telephone Consumer Protection Act. fcc.gov
3. IETF Secure Telephone Identity Revisited working group, the standards underlying STIR/SHAKEN caller attestation. datatracker.ietf.org/wg/stir
4. RFC 8224, Authenticated Identity Management in the Session Initiation Protocol, for what call signing does and does not establish. datatracker.ietf.org/doc/rfc8224
5. FIDO Alliance specifications, including cross device authentication flows relevant to out of band signature requests. fidoalliance.org/specifications
6. FBI Internet Crime Complaint Center, public service announcements on account takeover and telephone based social engineering. ic3.gov/PSA

Stop asking whether the caller is human. Ask whose human, and make them prove it somewhere other than the phone.