Subscribe
Tags
Voice AI
AI Agents
Latency & Performance
PUC
My internet provider rolled out an AI agent to replace their phone tree. I called it a few weeks ago.
It asked what I was calling about. Then six seconds of dead air, long enough that I said "hello?" into the void like it was 1998. It came back, said it recognized my number, asked if I was calling about that account. Yes. Then it told me it was going to authenticate me, and a completely different voice took over. Robotic. Every fifteen seconds: "MFA SENT. VERIFY MFA." I asked where it had sent anything. "INVALID MFA."
I eventually found an email with a six-digit code and read it out. The friendly voice came back to tell me it couldn't help, and transferred me to a person. The person asked what I was calling about, and said they'd need to authenticate me.
The entire AI portion of that call was worse than useless. It made me do everything twice.
Here's what should bother you about it: every individual piece worked. The model held a conversation. The transcription was accurate. The voices sounded fine. Authentication authenticated. Nothing in that stack failed, and the call was still garbage.
The call was garbage because of the places where the pieces met.
In this post
You can't eliminate a seam, the boundary where two parts of your voice app connect. You can only move it. Every seam costs you latency, coherence, or ownership of the failure.
A conversational turn has about 1,500 milliseconds before a human feels the lag. Placement is arithmetic: every seam in that turn spends against the same budget.
Cramming everything into one prompt doesn't remove your seams. It buries them somewhere you have no handle to grab.
Why do voice agents fail when every component works?
Because the failures live between the components, not inside them. Every application is a set of parts wired together, and every place two parts connect is what I'll call a seam. Speech-to-text hands off to the model. The model hands off to speech synthesis. The agent reaches out to your CRM. It hands off to a human. Your app is a pile of these boundaries with logic strung between them.
Can you design a voice pipeline with fewer moving parts?
You can move the parts around. You can't get rid of them, and that's the part people get wrong. If you think you've eliminated a seam, you've moved it, hidden it inside a library, pushed it down a layer, or bought it from a vendor who now owns it instead of you. It didn't go away. It's conserved. The only thing you actually control is where it lives.
I've believed a version of this for twenty years about single points of failure. I started out building large-scale web apps, and the lesson was the same: you never remove the single point of failure. You move it somewhere you can manage it, where a failure costs you the least. Same move here.
What actually goes wrong at the boundaries between components?
Three things, and a seam can charge you all three at once.
It costs time. Every hop is milliseconds you don't get back.
It costs coherence. State and context and identity have to cross the boundary, and when they don't, the two sides stop agreeing about what's going on.
It costs breakage. Seams break, and whoever holds the seam owns what happens when it does.
Every failure in the rest of this post is one of those three, or two of them at once.
Why does my voice agent go silent for several seconds?
Usually because it's waiting on a round trip that would be invisible in a web app. On the web a seam is nearly free, and you've been correctly trained to believe that.
Two hundred milliseconds is invisible on a page load. You'll wait two seconds for a dashboard to render and not consciously register it. So the normal patterns are fine: call out to a service, wait for the response, render. If a request is slow, you show a spinner and the user waits, mildly annoyed at worst. Nothing about that is bad engineering. It's the right design for a medium where the user is looking at a screen and understands that things load.
A conversation is not that medium. There's no spinner. There's no visual signal that work is happening. There's just silence, and silence in a conversation means something specific to a human being: either you didn't hear them, or something is broken. Two hundred milliseconds you'd never notice on a page is a beat of hesitation on a call. Two seconds is dead air, and the caller says "hello?"
You already know this failure from the web, incidentally. It's the N+1 query in a render path, or the chatty API call inside a loop, where each individual request is fast and the aggregate is a disaster. The difference is that on the web you find out when the page feels sluggish. On a call, you find out when the caller talks over your agent, or hangs up.
Voice agents built on web reflexes demo beautifully. One call, local stack, clean network, a few turns. Then they meet a real conversation that runs thirty turns on a real phone connection, and the thing that was fine at three turns becomes unbearable at thirty.
How much latency can a voice AI turn tolerate?
About a second and a half. That's roughly the budget before a human on the phone starts feeling the lag, before they say "hello?", before they assume the line dropped, before they talk over your agent.
Fifteen hundred milliseconds. Now count your seams. Transcription, the model, the synthesis, maybe a lookup in the middle. Every one spends against the same budget. This stops being philosophy and turns into arithmetic. Eight boundaries in a turn and each one has almost no room. Add one slow query and you've blown the whole thing, which means clawing time back everywhere else or moving that query off the critical path.
Getting things off the critical path is the actual skill, because not every seam has to be fast. Say you need a CRM lookup. If you can fire it at the start of the call, stuff the record into context, and use it a minute later when it comes up, it doesn't matter where that lookup lives or how slow it is. It resolved while nobody was waiting.
But if the caller says something three turns in that you couldn't have predicted, and you need that record right now, mid-sentence, to answer them. That one has to be fast. The only way to make it fast is to kill the round trip, which means it has to live where the call lives. Same lookup. Opposite placement. What decides it isn't a rule we can hand you. It's the shape of your specific call flow, and whether the turn closes inside the budget.
One blown turn isn't a catastrophe, though. A call can absorb a slow turn or two. It won't feel natural in that moment, and people forgive it, the same way they forgive a person on the phone taking a beat to think. What they don't forgive is thirty turns that each land right at the edge of the budget. No single turn was technically over, and the whole call is exhausting.
And when you know a gap is coming, you can cover it. If the lookup can't be async and it's going to take four seconds, the agent can do what a human does: say "hang on, let me pull up your account." The four seconds cost the same, but they're no longer unexplained silence. You bought the time back by setting an expectation.
That's worth noticing, because it's a third way to place a seam. You can move it closer to the call, you can move it off the critical path, or you can leave it exactly where it is and change what the caller expects while it runs.
Which is what makes my ISP call such a clean example. I'm fairly sure those six seconds were a lookup on my phone number to find my account, because the first thing it said afterward was that it recognized my number. That lookup had at least three better homes.
They could have fired it the instant the call arrived, before the agent ever spoke, so the record was sitting in context by the time I finished my sentence. They could have bought the time even more cheaply by letting the phone ring one extra cycle before answering. Nobody perceives a fourth ring as latency. It's the one place in a phone call where a second of work is completely invisible, and it has no equivalent on the web, which is why almost nobody reaches for it.
And if neither was possible, if the lookup genuinely had to happen after the greeting, they could have said so. "Let me pull up your account." Same six seconds, no dead air.
The failure wasn't the six seconds. It was six seconds that nobody accounted for.
Why does my AI agent forget what the caller already told it?
Go back to the moment the voice changed on my ISP call. That wasn't latency. That part was fast, actually.
That was coherence. Somebody bolted the authentication step on as a separate system, its own service, its own voice, its own vocabulary, and wired it into the flow without sharing anything across the boundary. The auth service didn't know my name, even though the agent that transferred me already had it. It didn't know how to talk like the rest of the call. So the join between two systems leaked straight into my experience.
I could hear the seam. Most seams are invisible and you have to argue they exist. That one announced itself in a stranger's voice.
The handoff to the human was the same failure, worse. The interaction record didn't cross the boundary. Everything the agent learned about me stayed on its side and died there, so I paid the difference by repeating myself. A latency budget can't see any of this. Every hop in that call could be under budget and you'd still ship it.
What happens to live calls when your LLM provider goes down?
You've got a pipeline. AssemblyAI turns speech into text. You hand the text to OpenAI. You hand what comes back to Rime for speech. Three vendors, two seams, and it works beautifully in the demo.
Now OpenAI goes down. Or it just gets slow, which is worse, because every call limps instead of failing clean. Or they deprecate your model on a Tuesday. Any one of those, and if you stitched this together yourself, every call in flight dies at once.
So are you going to build the thing that saves you? For one agent? You'll need to detect the degradation, fail over to another provider, rewrite your prompts and calls to handle that provider's quirks — and you do know that OpenAI and Groq and OpenRouter and Azure all serve the "same" model with their own little differences, right? — rehydrate the context for every call that's mid-sentence when you switch, and then fail back when the first provider recovers, all without dropping a call.
When you take a seam into your own layer, you don't just own the happy path. You own the break. And every seam has a default behavior when it breaks, whether you defined one or not. The default is always bad. The default is six seconds of silence. The default is a confident wrong answer because the lookup 500'd and nobody told the model. The default is "MFA SENT" on a loop forever. You design the failure behavior, or the failure behavior designs your worst call.
Should I put all my agent logic in one big prompt?
That's the tempting move. Don't wire up a pipeline full of boundaries you have to manage. Hand the whole thing to one model. Write the agent as one giant prompt. Zero seams to place. Clean.
Conservation says you can't. You didn't remove those seams. You buried them. Routing, guardrails, retrieval, business logic, all fused inside a single model call, in the one place you have the least control over.
We call this prompt-and-pray: putting the rules in the prompt and hoping the model follows them, with no deterministic check outside the model to enforce anything. The failure people notice is the model ignoring an instruction. The failure that costs more is that a single giant prompt takes away your handle. There's nothing left to grab.
You can't hand the model new information mid-call. You looked up the account while it was talking, and there's no way to tell it. You can't pull a function out of its hands: payment capture belongs with a processor the AI never sees, and now it can't be. You can't know when someone's jailbroken your prompt, let alone reset the context to recover.
Bury the seams and you bury every handle that isn't the prompt itself.
How do you build an AI that listens to a call without speaking on it?
This is the capability that makes seam placement concrete, and most stacks can't give it to you at all.
An agent that listens to a call and never speaks on it. A silent leg on the media, hearing everything, feeding a human in real time, catching the thing they'd miss. Augmenting the person instead of replacing them.
Think about what that takes. The model has to hear one leg of the call, reason about it, and get something useful back inside the same turn, before the moment is gone. A coach that arrives two seconds late is worse than no coach.
If you're building on an agentic framework, you can't do this at all. The abstraction assumes the agent is the call, so there's no seat for a silent third party on the media. And you can't bolt it on from your application layer either, because your application never touches the audio. It only sees text after the fact, on one side of a round trip you've already paid for.
On SignalWire it's a SignalWire Markup Language (SWML) method:
That's the integration. The hard part didn't go anywhere. The failover, the latency work, the tap on the right leg of the media, all of it is still there. It just doesn't live in your code anymore. It moved into the layer built to hold it, which is the whole point: the seam is conserved, so you put it where it does the least harm. For this one, that place is the media path, whether you own it or someone hands it to you.
Where the seams live
You can't avoid the seams. The only thing you ever get to decide is where they live.
Put them close to the media, tight to a layer built to hold them, and the caller never feels them. Scatter them to the edges because that's where they were easiest to drop, and your user pays for every one: in latency, in a stranger's voice reading them an error code, in explaining their problem twice to two systems that never spoke to each other.
The AI wave didn't repeal any of this. It hid the seams inside a model and convinced a lot of smart people they'd disappeared.
They didn't. Go find yours, and put them somewhere better.
Have questions about building better voice AI agents? Join the discussion in our developer community on Discord.
Related Resources
