Contact Sales

All fields are required

You can't prompt your way to great voice AI | SignalWire
CEO

You can't prompt your way to great voice AI

State management is the hidden problem behind voice AI

Start for free Talk to an Expert
Anthony Minessale

I. The Illusion of Control

You think you are building a voice AI product. You are actually building a distributed state synchronization system that happens to involve voice.

Every "voice AI platform" is an attempt to solve the same fundamental problem: keeping logic attached to a live call. The problem is that calls are stateful, real-time, and unforgiving. HTTP is stateless. The mismatch creates fragility that compounds with every feature you add.

When a caller speaks, your system needs to know: What was said before? What is the current intent? Is a transfer in progress? Is the caller on hold? Did they interrupt? Is another agent already handling this? These are state questions. And if that state lives outside the call, you are in a constant race to keep it synchronized.

This is the trap. You start by building a simple voice agent. You end up maintaining a brittle state machine spread across webhooks, databases, caches, and API calls. Every new feature adds complexity. Every edge case adds another synchronization point. Every scale milestone reveals race conditions you thought you had solved.

The companies that succeed in voice AI are not the ones with the best models. They are the ones who solved the state problem. Most have not. Most are faking it, hoping the race conditions stay rare enough that customers do not notice. That works until it does not.

II. The External Scaffolding Trap

Here is how most voice AI platforms work:

  1. A call arrives. The platform sends a webhook to your server.
  2. Your server receives the webhook, processes it, updates your database, and calls an API to control the call.
  3. Something happens on the call. Another webhook fires.
  4. Repeat until the call ends.

This looks simple. It is not.

The state lives in YOUR code. Your Redis instance tracking call sessions. Your Postgres database storing conversation history. Your in-memory cache holding the current intent. Your session store mapping call IDs to customer records. All of this is your responsibility.

And all of it is racing against the call.

The call does not wait for your database write to complete. The call does not care that your webhook handler is still processing the previous event. The call moves forward in real-time, and your external state is always playing catch-up.

Race conditions are not edge cases. They are the normal case at scale:

  • A webhook arrives. You update state. You call the API to play audio. But the caller already hung up. Your state says the call is active. It is not.
  • Two webhooks fire simultaneously because the caller spoke while your TTS was playing. Your state update from the first webhook gets overwritten by the second. Context is lost.
  • Network hiccup. Your callback never arrives. The call hangs in limbo. Your monitoring does not catch it because your state says everything is fine.
  • You initiate a transfer. Before the API response returns, the caller says "actually, never mind." Now you have a transfer in progress that your logic does not know about.

You are not building a voice product. You are building distributed systems infrastructure that happens to involve voice. And distributed systems are hard. They take years to get right. Most teams never do.

III. The "Bolt-On AI" Pattern and Why It Breaks

The standard architecture for AI voice looks like this:

Caller speaks
 ↓
Audio captured by platform
 ↓
Sent to Speech-to-Text service
 ↓
Text sent to YOUR backend (webhook)
 ↓
Your backend calls LLM
 ↓
LLM response sent to Text-to-Speech service
 ↓
Audio sent back to platform
 ↓
Platform plays audio to caller
 ↓
Caller responds
 ↓
Repeat

Count the arrows. Each arrow is a network hop. Each arrow is latency. Each arrow is a potential failure point. Each arrow is a state synchronization problem.

But the real problem is interruption.

The caller does not wait politely for your pipeline to complete. They speak whenever they want. They interrupt. They change their mind. They hang up mid-sentence.

When the caller interrupts:

  • Your STT service is still processing their previous utterance
  • Your LLM is generating a response to something they no longer care about
  • Your TTS is converting text that should not be played
  • Your platform is playing audio the caller is talking over

Who wins? How do you know to stop? Your external state does not know. It cannot know. By the time the "interruption detected" webhook arrives at your server, you have already committed to a response path that is now wrong.

Barge-in handling, endpointing, turn-taking, silence detection: all of these require knowing call state in real-time. External state is always stale. By the time you know what happened, it already happened, and the call moved on.

This is why bolt-on AI feels broken. The latency is not just slow responses. It is the constant misalignment between what the AI thinks is happening and what is actually happening on the call.

IV. The WebSocket Illusion

Some platforms claim to solve this with WebSockets. "We replaced webhooks with persistent connections. Problem solved."

No. You moved the problem. You did not solve it.

With WebSocket-based architectures like Twilio's ConversationRelay:

Now YOU manage thousands of persistent connections. Every active call is a WebSocket connection to your server. At scale, this means managing connection pools, load balancing sticky sessions, and handling the operational complexity of stateful infrastructure.

Now YOU handle reconnection, heartbeats, and message ordering. WebSockets drop. Networks hiccup. Messages arrive out of order. Your code needs to handle all of this gracefully. Most implementations do not.

Now YOU synchronize WebSocket state with your database state with API call state. You have not eliminated state synchronization. You have added another layer. The WebSocket knows one thing. Your database knows another. The platform's API returns a third. Which is correct? You need to reconcile constantly.

The call control is still external. The WebSocket gives you faster event notification. But when you want to actually do something (transfer the call, place on hold, bridge to another party), you still call an API. That API call still races against the call itself.

Scalability becomes your problem. One WebSocket per call means your infrastructure scales linearly with call volume. At 10,000 concurrent calls, you need infrastructure to handle 10,000 persistent connections. That is operational overhead that has nothing to do with your product.

The WebSocket is just a faster webhook. The fundamental architecture is the same: external state, external control, external complexity. You are still building distributed systems infrastructure. You just made it harder to scale.

Even OpenAI's Realtime API follows this pattern. Yes, you get streaming audio and lower latency to the model. But the call control is still yours. The state management is still yours. You still need to build the infrastructure to connect that WebSocket to actual phone calls, handle transfers, manage holds, and synchronize state. The model got faster. Your architecture problems did not go away.

V. What "State From Within" Actually Means

There is another way. The call itself holds the state. Not your backend. Not your database. Not your cache. The call.

SWML (SignalWire Markup Language) is a declarative state machine that travels with the call. When a call arrives, it receives a SWML document that defines what should happen. That document contains the logic, the state transitions, the AI configuration, the tools available, and the actions to take.

When the AI decides to place the caller on hold:

  • It does not call an API
  • It does not send a webhook to your backend asking you to call an API
  • It returns an instruction as part of its tool response
  • The platform executes that instruction atomically, in the same context as the decision

There is no race condition because there is no round-trip. The decision and the action happen together. The state update and the call control happen together. There is no window where your external state and the call state can diverge.

When the caller interrupts:

  • The platform knows immediately because it controls the audio stream
  • The platform stops TTS playback immediately because it controls the player
  • The platform notifies the AI immediately because the AI runs inside the platform
  • No webhook delay. No API call delay. No state synchronization delay.

This is not a minor optimization. This is a fundamentally different architecture. The call is not a remote resource you control via API. The call is a runtime environment where your logic executes directly.

VI. RELAY: Live Connection Since 2018

SignalWire introduced RELAY in 2018: real-time bidirectional control over calls via persistent connection.

But RELAY is not "faster webhooks." RELAY is a fundamentally different model.

With RELAY, you subscribe to call events and issue commands in a single persistent session. You receive events as they happen. You send commands that execute immediately. You receive confirmations that the commands succeeded. All in one connection, all in real-time.

But here is the key: RELAY is optional power, not required scaffolding.

You do not need RELAY to build on SignalWire. You can use pure SWML: define your call flow declaratively, let the platform execute it, receive webhooks when you need to make decisions. For most applications, this is simpler and sufficient.

But when you need fine-grained control, RELAY is there:

  • Monitor a call in real-time without affecting it
  • Inject a whisper to an agent mid-call
  • Fork media to a transcription service
  • Take over a call from a SWML flow when something unexpected happens

You can mix them. SWML defines the flow. RELAY observes and intervenes when needed. The platform handles the state in both cases. You handle the logic.

This is the difference between scaffolding and platform. Scaffolding is code you write to hold things together. Platform is infrastructure that works correctly so you can focus on your product.

Other platforms give you scaffolding and call it a product. SignalWire gives you a platform and lets you build your product on top.

VII. The FreeSWITCH Inheritance

SignalWire's founding team created FreeSWITCH, the open-source telephony engine that powers some of the largest voice networks in the world. FreeSWITCH has processed trillions of voice minutes. It has been battle-tested for over a decade in the most demanding environments.

FreeSWITCH solved the state problem a long time ago.

Conference bridges where dozens of participants join and leave dynamically. Call parking where calls wait indefinitely and can be retrieved from any endpoint. Attended transfers where the transferring party can return to the original caller if the target does not answer. Whisper and barge where supervisors monitor and intervene in live calls. Media forking where audio streams are duplicated to recording or analysis services without affecting the call.

Every complex call scenario has already been solved at the platform level. The state management for these scenarios is not something you need to build. It exists. It works. It has worked at scale for years.

SignalWire exposes this power through simple, secure APIs. You are not recreating call state management. You are using infrastructure that already handles it correctly.

When you build on SignalWire, you inherit decades of engineering. When you build on bolt-on platforms, you inherit their shortcuts and their race conditions.

VIII. Your State, Our Management

The platform manages call state. But what about YOUR state? The customer ID, the account balance, the authentication token, the order in progress, the context you need in every callback?

You attach it to the call. The platform carries it. You get it back everywhere.

In SWML and RELAY, session variables let you attach arbitrary data to the call. Set them at any point. Read them at any point. They travel with the call through transfers, holds, conferences, and AI interactions. When a webhook fires, your session variables are in the payload. No external lookup required.

# Set session variables in SWML

- set:
 variables:
 customer_id: "12345"
 account_tier: "premium"
 auth_token: "abc..."

In the AI kernel, global_data takes this further. You can attach data that the AI never sees but that your tools always receive. The AI cannot speak this data. It cannot include it in responses. It cannot reason about it. But when the AI calls a tool, your backend receives the full global_data payload.

This is critical for security and architecture:

  • Authentication tokens travel with the call. Your tools verify permissions without external lookups.
  • Payment details stay hidden from the AI. The AI says "process the payment." Your tool receives the card token from global_data and executes securely.
  • Order state accumulates invisibly. The AI adds items. Your tools update the order in global_data. The AI never sees the running total. It cannot be tricked into applying invalid discounts.
  • Conversation metadata collects automatically. Timestamps, turn counts, sentiment scores, transcription snippets. All available for post-processing without you collecting it.

When the call ends, you get everything: the session variables you set, the global_data you accumulated, the metadata the platform collected. One payload, complete context, no assembly required.

This is the difference between "the platform manages state" and "you manage state externally." With external state, you store the customer ID in Redis, keyed by call ID. You hope the call ID matches. You hope Redis is available. You hope the TTL has not expired. You hope no race condition corrupted the data.

With platform-native state, the data travels with the call. It cannot get out of sync because there is only one copy. It cannot expire because the call owns it. It cannot race because updates are atomic within the call context.

Your tools become simpler. Your callbacks become richer. Your external dependencies become fewer.

IX. Every AI Enhancement Works The Same Way

The pattern applies to every AI capability, not just conversational agents.

Dynamic AI Agents

The state machine controls the AI. The AI does not control the state machine. The AI is stateless: it receives context, it produces a response and optional tool calls, it does not maintain session state. The platform is stateful: it tracks conversation history, call state, tool execution, and workflow progress.

When the AI calls a tool, the platform executes it. If the tool returns SWML actions (hold, transfer, dial, play audio), the platform executes them atomically. The AI does not manage state. The AI makes decisions. The platform makes them happen.

Live Transcription

Audio streams from the call to the transcription service. Results stream back. The platform handles the audio routing, the buffering, the delivery of results. You subscribe to transcription events via RELAY or receive them via webhook. You do not manage audio buffers. You do not manage transcription state. You receive text.

Live Translation

The same pattern. Speech in one language, text out in another, synthesized speech in the target language, all routed correctly. The platform handles the complexity. You configure the flow and consume the output.

Human-AI Handoff

Seamless because the platform knows the call state. When an AI agent escalates to a human, the platform transfers the call with full context. The human agent sees the conversation history. The caller does not repeat themselves. The handoff does not lose state because there is no external state to lose.

Multi-Party Conferencing with AI

FreeSWITCH conferencing plus AI participants. The platform handles mixing audio from multiple parties. The platform handles who can speak and who can hear. AI participants join like any other participant. The platform routes their audio, processes their responses, and manages the state of the entire conference.

Every permutation follows the same principle: the platform manages state, you manage logic. You do not rebuild state management for each new capability. You configure the capability and the platform handles the rest.

X. The Race Condition Graveyard

Here is what breaks when state lives outside the call. These are not hypotheticals. These are patterns we see in production systems built on bolt-on platforms.

The Zombie Call

The caller hangs up. The platform sends a webhook. Your webhook handler fails (timeout, exception, network issue). Your retry logic kicks in, but by then the call is gone. Your state says the call is active. It is not. Your system tries to play audio to a dead call. The API returns an error. Your error handler updates state. But another process already read the stale state and initiated a transfer. Now you have orphaned state that never cleans up.

The Double Update

The caller says something while your TTS is playing. Two webhooks fire: "speech detected" and "playback interrupted." Both arrive at your server within milliseconds. Both read the current state. Both decide on an action. Both update the state. One overwrites the other. You lose context. The AI responds to the wrong thing. The caller is confused.

The Stale Response

Your LLM takes 800ms to generate a response. During that 800ms, the caller says "actually, never mind" and asks a different question. Your system does not know. The "never mind" webhook is queued behind the LLM processing. You play the stale response. The caller now has to clarify. Latency doubles.

The Phantom Transfer

You initiate a transfer. The API call is in flight. The caller hangs up. Your transfer succeeds (from the API's perspective). The target's phone rings. They answer. The original caller is gone. You have a confused target and no original caller. Your state shows a successful transfer.

The Concurrent Agent

Two systems both monitor the same call (maybe AI and human supervisor). Both decide to take action. Both send commands. Both succeed from their perspective. The call receives conflicting instructions. Behavior is undefined.

These are not edge cases. They are the normal case at scale. The only question is frequency. At 100 concurrent calls, you might see these weekly. At 10,000 concurrent calls, you see them hourly. At 100,000 concurrent calls, you see them constantly.

The teams that build on external state spend their engineering time chasing these bugs. They add retry logic, distributed locks, idempotency keys, state reconciliation jobs, orphan cleanup crons. They build infrastructure to work around the fundamental architecture flaw.

The teams that build on platform-native state do not have these bugs. The bugs cannot exist because the architecture does not allow them.

XI. What Technical Leaders Get Wrong

"We can build this ourselves."

Yes, you can. You will spend years on state management instead of months on product. You will discover edge cases through production incidents. You will build increasingly complex infrastructure to handle problems that should not exist. And your best engineers will spend their time on plumbing instead of differentiation.

The teams that have done this successfully had massive resources and long timelines. They were building the platform, not building on a platform. If your business is voice infrastructure, build voice infrastructure. If your business is something else, do not rebuild what already exists.

"We need full control."

You have it. RELAY gives you every primitive: play audio, stop audio, record, transcribe, transfer, bridge, conference, whisper, barge, fork media, inject DTMF. SWML gives you every pattern: IVR flows, AI agents, conditional branching, state machines, tool execution. The control is there. The difference is that the platform handles the state management for you.

Full control does not mean building everything yourself. It means having access to every capability you need. SignalWire provides both.

"We don't want vendor lock-in."

The lock-in is in the state management code you are writing. That is the real trap.

If you build custom state synchronization infrastructure on top of Twilio, you are locked into your own code. Migrating means rewriting your state management. The platform is the easy part to replace. Your infrastructure is the hard part.

If you build on SignalWire's platform-native state, your code is simpler. There is less of it. Migration means redefining SWML documents and updating API calls. The complex state management code does not exist to migrate.

Vendor lock-in concerns are valid. But the answer is not "build everything yourself." The answer is "choose infrastructure that makes your code simple enough to migrate."

"Our architecture is simpler."

It looks simpler because you have not built it yet. A few webhooks, a Redis cache, some API calls. What could go wrong?

Everything, at scale. The simplicity is an illusion. The complexity is deferred, not eliminated. It will arrive as production incidents, as customer complaints, as engineering sprints spent debugging race conditions instead of shipping features.

SignalWire's architecture looks more complex because it handles the complexity for you. That complexity exists. The question is where it lives. In your code, where you maintain it forever? Or in the platform, where it is already solved?

XII. Building Two Steps Ahead

Your product roadmap does not stop at basic AI voice agents.

Today: A voice agent that answers calls and handles simple queries.

Tomorrow: Multi-agent orchestration. The first agent screens the call, collects information, and hands off to a specialized agent. The specialized agent resolves the issue or escalates to a human. The human takes over with full context. All seamlessly, all in one call.

Next month: Cross-call coordination. An AI agent places a caller on hold, dials a specialist, briefs the specialist on the situation, gets approval, and bridges the parties together. Two calls, coordinated in real-time, with AI managing the workflow.

Next quarter: Real-time translation. Caller speaks Spanish. Agent speaks English. AI translates both directions in real-time. The conversation flows naturally despite the language barrier.

Next year: Compliance recording with live redaction. Every call is recorded for compliance, but sensitive data (credit card numbers, SSNs) is redacted in real-time before it hits storage. AI detects sensitive data, the platform redacts the audio, you never store what you should not store.

If your state management is external, every one of these features requires rearchitecting. Multi-agent orchestration means coordinating state across agents. Cross-call coordination means synchronizing state across calls. Real-time translation means managing state for the translation pipeline. Each feature adds complexity to your already-fragile state infrastructure.

If your state management is platform-native, every one of these features is configuration. Define the SWML flow. Enable the capability. The platform handles the state.

You are not just building for today. You are building for the product you will need in two years. The architecture decisions you make now determine whether those future features take days or quarters to implement.

XIII. Platform vs. Scaffolding: The Only Choice That Matters

Scaffolding is code you write to hold things together until you can do it properly. It is temporary by intent but permanent by inertia. It accumulates. It becomes load-bearing. It cannot be removed without rebuilding.

Most voice AI implementations are scaffolding. Webhook handlers that retry and reconcile. Caching layers that try to predict call state. Background jobs that clean up orphaned sessions. Monitoring that detects race conditions after they happen. All of it is scaffolding holding together an architecture that does not fit the problem.

Platform is infrastructure that does it properly from the start. It handles the hard problems so you do not have to. It scales because it was designed to scale. It handles edge cases because it has seen them all before.

SignalWire is not a shortcut. It is the correct architecture. The one you would build if you had a decade and unlimited resources. The one FreeSWITCH proved at scale. The one that treats call state as a first-class concern instead of an afterthought.

Every hour you spend on state synchronization is an hour not spent on your product. Every engineer debugging race conditions is an engineer not building features. Every production incident caused by external state is customer trust you cannot recover.

The choice is not build versus buy. The choice is where you want to spend your engineering effort. On problems that are already solved? Or on problems that make your product unique?

Stop fighting your architecture. Start shipping your product.