Smart Code, Dumb Models

The obvious way to build an AI voice agent is to hand the whole call to a large language model. Write a long prompt that describes your business, your policies, and every step of the process. Give the model some tools and let it figure out the rest. It demos beautifully. Then it goes into production and the problems start: the agent pauses too long before answering, skips a step it was told never to skip, and handles the same request three different ways on three different calls.

We build voice agents the other way. Our approach is smart code, dumb models. To see why, start with what inference is actually good at.

What inference is for

LLMs look like magic because they handle ambiguous, open-ended work that's hard to specify in code. Nobody can write the function that turns every way a caller might say "I need to move my appointment" into the right intent, but a model does it easily. That's the category where inference earns its keep:

• Figuring out what the caller wants

• Deciding which approach to take to a problem

• Choosing which tool to use for a particular job

• Handling the weird requests and edge cases nobody wrote code for

Every item on that list is a judgment call. There's no fixed procedure to follow, and the right answer depends on context the developer couldn't predict.

What inference is bad at

Inference is a terrible fit for deterministic, repeated, well-specified tasks. If you can define the steps, inference is the wrong tool for the job.

Take identity verification. Before an agent discusses an account, it needs to ask for the account number, look it up, ask for the billing ZIP code, and compare the two. You can put "always verify the caller before discussing the account" in the prompt, and the model will follow it almost every time. Code follows it every time. Almost every time is fine for a demo and unacceptable when the agent is reading someone's account balance to a stranger.

The model also does that work slowly. Every step you hand to inference is a step the model has to read, reason about, and generate tokens for, on every turn of the call. A prompt that carries your entire business process can run to thousands of tokens the model processes before it says a word. The usual fix for an unreliable model is a bigger, smarter one, which is slower still.

Speed matters more on a phone call than almost anywhere else. People answer each other in a few hundred milliseconds. When an agent takes two or three seconds to respond, callers talk over it, repeat themselves, or decide it's broken. A chat window can show a spinner while the model thinks. A phone call can't.

Then there's testing. Code that runs the same steps in the same order can be tested with ordinary unit tests. The same input produces the same output, and when it breaks you can find out why. A prompt can only be sampled. You write evals: define the criteria for a good outcome, run the prompt against a large set of conversations, and measure how often it gets there. That's the right way to test inference, and it's real work. It also only tells you the behavior is right most of the time, which is a fine answer for understanding a caller and the wrong one for a step that has to happen on every call.

Splitting the work between code and models

Smart code, dumb models gives each tool the job it's good at.

Code runs the conversation. It decides which step comes next, what data to fetch, which rules apply, and when the call is done. Anything you can write down as a procedure lives in code, where it runs the same way every time and costs nothing to repeat.

The model handles what code can't. It understands what the caller said, works out what they want, phrases the response the way a person would, and copes when the caller says something nobody planned for. Because that job is narrow, a small, fast model does it well. "Dumb" doesn't mean useless. It means the model isn't carrying the whole business process in its head on every turn.

The result is an agent that responds as fast as a person, follows the rules every time, can be tested like any other software, and costs less to run. You haven't given up the magic. You've pointed it at the part of the call that needs it.

If you're building the future of voice AI, build it with us. Sign up for a free account and join the discussion on our community Discord.