Why Voice AI Pilots Work in the Demo and Fail on Real Calls
The production engineering checklist that separates a prototype from a dependable voice agent.
The demo goes well, as demos generally do.
Clean audio from a headset in a quiet room, one call at a time, and a tester who knows what the agent can do and phrases things the way it expects. Everyone in the room agrees it is ready.
Then it meets a carrier, four hundred simultaneous callers, a hands-free kit on a motorway, a caller who interrupts the greeting, and an account number that sounds like three others.
The gap between those two experiences is not a model problem. Speech recognition and language models are the most mature part of the stack. What breaks in production is everything around them: the call path, the integrations, the knowledge the agent reads from, and the governance over what it may do.
Most of the difficulty sits at the seams, where this guide spends its time.
Why the demo cannot show you production
The industry evidence is hard to ignore. Gartner predicts more than 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. MIT’s Project NANDA reported in its 2025 State of AI in Business study that the large majority of enterprise generative AI pilots delivered no measurable P&L return. Deloitte’s Tech Trends 2026 puts numbers on the same gap: 14% of organisations have production-ready agentic AI, and 11% are actually running it. Deloitte’s reading of that research names legacy system incompatibility as a driver, another way of describing the seams this article is about.
These are market signals rather than forecasts of your own outcome. Still, the causes they name are fairly consistent, and most trace back to the same thing: the pilot was scoped to show the model works, while production asks whether the agent can complete work at volume without introducing a new failure mode.
Voice compounds the difficulty. In a chat, a two-second pause is invisible, because the customer assumes the system is thinking, and they can re-read what was said. On a call, two seconds of silence reads as a dropped line, a confidently wrong answer tends to be accepted, and there is nothing to scroll back to. Which is part of why a text agent performing well says little about how the same vendor handles voice.
Map the complete call path
Before testing anything, draw the path a call takes and mark every boundary where one system hands to another. Failures cluster at those boundaries, and almost none of them are visible in a demo environment.
Carrier ingress. How the call actually arrives: SIP trunking, bring-your-own-carrier, the codec in use, and what happens to audio quality when it traverses a gateway it was not tuned for.
Recognition and orchestration. Where speech becomes intent, and where the decision to act, ask again or escalate is made.
Keypad input. DTMF is still how account numbers and card details are captured in regulated environments. It has to keep working when an agent sits in the path, and it has to work when the caller mixes speech and keypad in the same turn.
Authentication. Whether the agent can verify a caller to the standard your compliance team already signed off, and what it does when verification fails twice.
Business systems. CRM, order, billing, claims, scheduling. An agent that can only read is closer to a well-spoken IVR. One that can act needs scoped write access, and every write is somewhere a timeout, a stale record or a permissions gap can turn into a customer-facing failure.
Recording and compliance capture. Confirm what is recorded when the agent is on the line, where it is stored, and whether it survives a transfer.
Human escalation. The last boundary and, in our experience, the one that breaks most often. A transfer must carry who the caller is, what has been verified, what was attempted and why the agent gave up. A handoff without context is worse than having no agent at all, because the customer explains themselves twice.
Each of these is a separate integration, and most of them stay invisible until real calls start arriving.
Test the conversation under pressure
Latency, measured at load rather than on average. Analyst benchmarking puts the breaking point for AI voice conversations at around 800 milliseconds; ZK Research principal analyst Zeus Kerravala has described this as the point beyond which the experience deteriorates quickly enough that customers notice. A human-to-human call carries roughly 200ms of delay, and language model processing adds meaningfully on top. The trap is the average: a system reporting a comfortable mean can still sit well past the threshold at peak concurrency, precisely when the most callers are exposed. Test at your busiest hour’s concurrency and report the tail, not the mean.
Interruption and turn-taking. Real callers interrupt. They answer before the sentence finishes, they say “no, wait”, they talk over a menu they have heard before. An agent that cannot stop mid-utterance and re-plan produces the kind of failure customers remember, since being talked over by a machine reads as being ignored. Barge-in is worth testing deliberately, not discovering in production.
Silence, noise and recovery. What happens after three seconds of nothing? A caller in a car, a speakerphone, a poor mobile connection. And when the agent has misunderstood twice, does it have a defined recovery path, or does it loop? Looping is a quick way for an automation programme to acquire internal opponents.
The vocabulary that actually matters. Headline recognition accuracy is measured on general speech, and your calls mostly are not. They are surnames, account numbers, policy references, drug names and product codes, where one character changes the meaning and general-purpose accuracy tends to degrade. Build the test set from real call transcripts, weighted towards the entities the agent must get right, and measure those separately.
Ground the answers, bound the actions
An agent with no governed source of truth will improvise, and on a call an improvised answer arrives with the same confidence as a correct one, with nothing to signal the difference to the caller.
Grounding the agent in a single governed knowledge source is the structural fix, best done before go-live rather than after the first incident. Four things make it hold:
Freshness with an owner. Knowledge decays quietly. A policy changes, a promotion ends, a process is revised, and the agent carries on citing the old version until somebody updates it. Name the owner and the cadence at design time, not after a complaint.
One source, two consumers. The same governed content should serve the voice agent and your human agents. This is the part that pays regardless of how containment performs: consistent answers, one update workflow instead of five, and a measurable onboarding benefit for new hires.
Explicit action boundaries. What the agent may do without a human, what requires confirmation, what it may never do, all enforced in the orchestration layer, not implied in a prompt. Gartner’s warning about “agent washing”, and its estimate that only around 130 of the thousands of self-described agentic vendors are genuinely agentic, matters here: a platform that is really a chatbot with actions bolted on will not give you enforceable boundaries.
Audit trail and a shut-off. For any regulated deployment: what the agent heard, retrieved, decided and did. Plus a documented way to switch one capability off in minutes without taking the service down. Teams without that switch tend to disable the whole agent at the first surprise, which is often how a working pilot ends up cancelled.
Replace the vanity metrics
Containment is the number most pilots report, and it is also the one most likely to mislead. It tells you the call ended in the automated channel, which is not the same as the caller’s problem being solved.
The two come apart predictably. An agent optimised for containment can hit its target by making transfer difficult, so the call is contained and the customer either calls back tomorrow or gives up. Both read as success on a containment report, and the second rarely shows up anywhere.
A production scorecard needs at least these:
- Resolution rate: the caller’s issue was actually closed, verified against the downstream system rather than the call outcome code.
- Repeat contact within 7 days: the single best detector of containment that was not resolution.
- Escalation quality: what proportion of transfers arrive with full context, and what the handle time looks like on the human side afterwards.
- Customer effort: how hard the caller had to work, measured on automated and human interactions alike.
- Cost per resolved interaction: not cost per contained call, which improves whenever the agent gets less useful.
Containment is worth reporting alongside these rather than in place of them, and the targets are worth setting from your own intent mix and baseline. Containment goals lifted from a vendor case study describe someone else’s call drivers.
The production-readiness checklist
Before you increase call volume, answer these. Any question without an owner and a tested answer is an open risk.
Call path. Is the full path from carrier to escalation documented? Does DTMF still work? Is recording complete across a transfer? Does the handoff carry identity, verification state and attempt history?
Conversation. Has latency been measured at peak concurrency and reported as a tail rather than a mean? Has barge-in been tested deliberately? Is there a defined recovery path after two failed attempts? Is entity recognition measured separately, on your own vocabulary?
Knowledge and actions. Is there one governed source? Who owns freshness, on what cadence? Are action boundaries enforced in the orchestration layer? Is there an audit trail and a per-capability shut-off?
Measurement. Is resolution measured against the downstream system? Is repeat contact tracked? Are targets derived from your own baseline?
Operations. Who owns the agent’s performance in ninety days, when products, policies and call drivers have moved? What is the review cadence, and who authorises a change?
The pilot is not the hard part
Enterprise voice AI platforms such as Parloa, PolyAI, Cisco AI Agent, Genesys AVA, NICE Cognigy, Zendesk and Forethought have each solved substantial parts of this problem, in different ways and for different environments. None of them can solve the parts that live in your estate: your carrier configuration, your CRM permissions, your knowledge debt, your compliance posture, your escalation behaviour.
That is where the work actually sits, and a demo cannot show it to you. A voice agent behaves less like a model you install than like a production system with telephony, integration, knowledge and governance problems attached, and it becomes dependable in proportion to how many of those have an owner and a test.
In our experience the organisations that reach production are rarely those with the most impressive pilot, but those that worked through the dull questions early.
Servion has engineered enterprise voice environments for three decades, from carrier integration to production escalation design, and now runs agentic voice deployments across multiple platforms. Benchmark an existing voice AI pilot against the production-readiness checklist →
ABOUT THE AUTHOR
Bruce Eidsvik is Chief Growth Officer at Servion, where he leads go-to-market strategy and helps enterprise clients navigate the evolving CX technology landscape. With deep expertise in contact center transformation and vendor ecosystems, Bruce guides organizations from evaluation to deployment across some of the most complex CX environments in the world.
