Building Voice Agents That Get Sh*t Done

When we started building Sunday, a phone agent for fitness studios, I thought the hard part would be the voice pipeline: stream tokens into TTS, send audio through STT, detect when someone stopped speaking, and keep the whole thing fast enough to feel natural.

That part mattered. It just stopped being the differentiator much faster than I expected.

LiveKit and Pipecat made the real-time plumbing easier to assemble. Vapi and Retell offered most of the stack as a service. Deepgram Flux, which we used, combined transcription and turn detection into one model. The infrastructure kept getting better while we were building on top of it.

After thousands of calls, the problems that consumed us were further up the stack: getting callers to give Sunday a chance, keeping long workflows on track, dealing with silence, and knowing whether a successful tool call actually meant the caller's problem was solved.

The first problem was getting anyone to talk to Sunday.

Years of bad phone systems have trained us to escape the moment we hear an automated voice. I do it too. I press zero, say "representative," or try whatever might get me to a person faster.

Our first greeting did not help:

Thanks for calling [studio]. I'm Sunday, your AI concierge on a recorded line, how can I help?

It took seven or eight seconds to say. Before callers had told us what they wanted, Sunday already sounded like the phone tree they were trying to avoid.

We eventually replaced it with something much closer to what the front desk already said:

This is [studio] on a recorded line, how can I help?

That small change materially improved our conversation rate. Sunday still said it was an AI when someone asked. We were not trying to trick anyone; we just wanted callers to explain why they were calling before deciding it could not help.

We spent a lot of time trying voices from Cartesia, Rime, and other TTS providers. The voices we liked sounded like someone who might work at the studio, not someone narrating an audiobook. Shorter sentences, small pauses, the occasional "um" or "hm," and acknowledgements before slow operations did more for the experience than perfectly written prose.

A better voice model could not rescue a seven-second preamble or a badly timed turn boundary.

Once more callers got past the opening, we got to find out whether Sunday could actually help them.

A call entered through the studio's existing number, forwarded to Telnyx, crossed SIP into LiveKit Cloud, and triggered room dispatch for an agent process. While the call was connecting, the agent fetched studio configuration, caller context, and the state it needed for the first turn.

The agent process owned the live conversation: prompt assembly, model calls, tool use, knowledge retrieval, and handoffs to AgentTasks. A separate backend owned configuration and durable state.

We were a small team, so fine-tuning and hosting our own conversational model never made sense. We used the LiveKit SDK and LiveKit Cloud for the real-time layer, managed providers for STT, TTS, and the LLM, and kept orchestration under our control.

We rewrote the prompts constantly. We tried reasoning settings from none to high and eventually landed on GPT-5.4-mini with low reasoning. We adjusted endpointing and interruption behavior, cached speech we could predict ahead of time, and played short acknowledgements before operations that we knew would be slow.

As the main agent accumulated instructions and tools, it gained more ways to get distracted halfway through a task. A booking flow did not need to know how cancellations worked, and a cancellation flow did not need every tool available to the general agent.

We moved those bounded workflows into LiveKit AgentTasks using its supervisor pattern. The main agent kept control of the call while a short-lived task took over with a smaller prompt and tool set.

A cancellation task only needed to resolve the membership, check the policy, get confirmation, perform the change, and interpret the provider response.

The smaller context and narrower tool set made these workflows easier to keep on track. But as Sunday took on more of them, more turns included long reasoning, a knowledge-base lookup, or a heavy tool call. That made the next problem impossible to ignore.

There is no loading spinner on a phone call. There is just silence.

Most turns were fast, but the occasional long pause could derail an otherwise good conversation. The caller would repeat themselves, interrupt Sunday, or assume the call had died. Averages hid the moments people actually remembered.

I had seen 650 ms repeated as the point a voice agent needed to stay under to feel natural. It is a useful target, but not a hard law. Humans usually leave gaps of around 200 ms between turns because they start planning a response before the other person has finished speaking (Levinson and Torreira, 2015). LiveKit uses a broader target of under one second, while Deepgram reported a 660 ms median as being in the natural conversational range (LiveKit, Deepgram).

We aimed for roughly 650 ms between the caller finishing and Sunday beginning to speak. That was realistic for normal back-and-forth, but not for every request. Some turns needed longer reasoning, a knowledge-base lookup, or a call to an external provider.

We handled those cases in two ways. First, we removed delays we could avoid. Sunday loaded the studio configuration and caller details while the phone was still ringing, and we generated reusable audio such as the greeting ahead of time.

When the work itself was slow, Sunday immediately played a short acknowledgement such as "hm, let me look into that." The clip was already generated, so it could play while the agent continued reasoning, searching the knowledge base, or waiting for a tool. The answer still came afterward, once the work was actually complete.

For example:

Caller: Can you check whether I'm eligible to cancel?

Sunday: Hm, let me look into that.

The agent reasons, searches the policy, and calls the membership provider while the cached audio plays.

Sunday: You're eligible to cancel. I can do that now if you'd like.

The acknowledgement did not make the operation faster. It made the state of the conversation obvious and kept the caller engaged while the operation ran. But to reduce the actual wait, we needed to understand everything that happened between the caller finishing and Sunday beginning to speak.

At first, we only knew the total time. Endpointing, the LLM, tool calls, query embedding, vector search, and TTS all collapsed into the same silence.

We added timing around each stage so we could break the latency down turn by turn. The surprising result was how much time we were spending in the knowledge-base lookup—particularly before the vector search even started.

We generated an embedding for every natural-language query using OpenAI's text-embedding-3-small API. That network call took roughly 300 ms, consuming almost half of our 650 ms target on its own. We tried other hosted embedding providers, but eventually decided the network call was not worth keeping.

We shipped BAAI/bge-base-en-v1.5, a small embedding model, with the agent backend. Query embedding dropped from roughly 300 ms to 30 ms—a 10× improvement—while Turbopuffer continued to handle the vector search.

That implementation is already a little dated. Turbopuffer now offers native embeddings in private beta, which can turn query text into a vector as part of the search instead of requiring us to run a separate embedding service. It is a good example of how quickly this stack moves: infrastructure we had to build ourselves can become a database feature soon after.

Fast retrieval is useless if the context is wrong.

The speedup mattered, but latency was only one part of the knowledge-base problem. We had already started evaluating retrieval quality before changing the embedding path.

We learned that "retrieval quality" was not one metric. A bad result could mean the customer's knowledge base did not contain the answer, our chunking separated important context, the embedding failed to represent the query, the search missed the right chunks, or we did not return enough context to the agent.

Instead of treating one score as the answer, we separated those failure modes:

1. Coverage: For a curated set of real production questions, did the customer's source material contain the information needed to answer?
2. Retrieval: When we knew which source material was relevant, did the expected chunks appear in the top results?
3. ANN recall: Did Turbopuffer's approximate index return the same nearest neighbors as an exhaustive search?
4. Context quality: Given the question and returned passages, did a human or LLM judge consider the context relevant and complete enough to use?

Turbopuffer's recall endpoint helped with the third check. It compares approximate nearest-neighbor results with an exhaustive search over sampled vectors. That can tell us whether the index is losing nearest neighbors, but not whether those neighbors contain the right answer. It does not replace the curated questions or the context review.

This gave us a strategy for finding where retrieval failed instead of collapsing every failure into "the knowledge base is bad." It also gave us a repeatable baseline whenever we changed customer content, chunking, embeddings, ranking, or the number of passages returned.

Context played a large role in guiding Sunday's answers and decisions. For questions that depended on the knowledge base, a stronger model could not make up for missing or incomplete information. That made this evaluation crucial, even though it still did not measure the agent as a whole.

Did Sunday actually handle the call?

The retrieval-quality checks and latency traces told us a lot about one part of Sunday. They still did not tell us whether Sunday had handled the call.

Containment was too blunt for that. A call that never reached the front desk could mean Sunday solved the problem, the caller hung up, or the conversation reached a path we did not support. We also transferred callers who asked for a person. Preventing those handoffs would have improved containment while making the product worse.

So we built our own production evaluation monitor. After every call, it marked the conversation as handled by Sunday, escalated, or needs review. An LLM judge also read the conversation and produced a deeper assessment of what happened.

At the end of each day, a job gathered that customer's calls, grouped them using deterministic rules and additional LLM judges, and posted a Daily Shift Summary to Slack. The summary let our team see how Sunday had performed for that customer without reading every transcript.

Over time, those summaries exposed patterns across customers: requests Sunday handled well, recurring failure modes, fixes we needed to make, and the value the agent was providing. A single containment percentage could not show us any of that.

Throughout this work, we kept coming back to a simple rule: traces over hunches.

We versioned every part of the agent, timed every stage of a turn, and paired those traces with retrieval evals and daily call summaries. When something felt slow or unreliable, we wanted the evidence to show us where it broke before we changed the model or architecture.

That is why I would not start by hosting a larger model. I would make the model smarter only after the traces showed it was the bottleneck.

We started this project thinking voice quality and model intelligence would carry most of the weight. They helped Sunday sound human, but the less glamorous work—constraining workflows, tracing slow turns, evaluating retrieval, and reviewing production calls—made Sunday useful and earned our customers' trust.

That's what it means to build a Voice Agent that gets sh*t done.