Voice agents are not pipelines. They are competing loops.

The demo diagram is a pipeline

The first voice-agent diagram everyone draws is a pipeline:

That gets you a demo. It does not get you a natural conversation.

The pipeline diagram hides the control problem. Real conversation is not a clean sequence where one person finishes, the other thinks, then the other speaks. People interrupt. They backchannel. They say "wait." They correct themselves. They talk over the first half of a sentence because they already know where it is going.

That means a production voice agent has more than one loop running at once.

There is the conversation loop:

There is the tool loop:

There is the turn-taking loop:

There is the audio-output loop:

And then there is the loop nobody talks about enough:

That is the interruption loop. It is also the loop users notice first when it fails.

The interruption loop is where the abstraction leaks

In text agents, "observe" usually means reading a message or a tool result. In voice agents, "observe" can mean very different things. It can mean: I received an interim transcript. Or it can mean: I saw speech energy in the last 20ms audio frame.

Those are not equivalent observations. They live at different levels of the stack. They arrive at different times. They create different failure modes. And they lead to completely different product feel.

A transcript-triggered voice agent observes the caller through speech-to-text. That means the interruption loop looks like this:

A frame-triggered voice agent observes the caller through raw audio. That loop looks more like this:

The transcript loop is simpler. It is easier to build. It is easier to inspect. It lines up nicely with the rest of the LLM application, because everything is text.

The frame loop is closer to the actual conversation. It can react before speech has been transcribed. But it is not free. Now the app has to decide whether the caller is interrupting, backchanneling, coughing, breathing, talking to someone else, or just sitting in a noisy room. It also has to coordinate cancellation across speech generation, playback, model state, and transport state.

I built the simple loop first

I wanted to understand the floor of the simpler architecture before arguing for the more complicated one. So I built a phone agent where the application did not own raw audio frames.

The app used Telnyx Call Control for transport, call control, and speech generation, with Google as the transcription engine behind Telnyx’s transcription webhooks. The numbers in this post are one platform, one engine, one configuration. The lesson is about the architecture, not the vendor. The app learned that the caller was speaking when it received an interim transcript. When it decided the caller had interrupted, it sent a playback-stop command.

This is the architecture a lot of people reach for first because it is clean. There is a phone call, a stream of webhooks, a stream of transcripts, and a set of commands the application can send back to the platform. No media server. No local VAD. No custom frame processing. No parallel audio path. Just:

That simplicity is real. So is the latency cost.

I measured the interruption loop

The benchmark was not trying to measure the whole agent. It measured one loop:

More precisely, the harness measured from the harness leg’s call.speak.started event to the agent leg’s call.speak.ended event.

That distinction matters. This is a control-plane event measurement, not a direct waveform measurement of what a human ear heard. The next version should include caller-side audio recording and acoustic stop detection. For this run, I am reporting the events the platform exposed and the application could act on.

The test harness dialed the agent from a second number, bridged both legs, waited for the agent to speak, and injected a known spoken stimulus. Both call legs landed in the same process, so the harness could pair events without a cross-machine synchronization problem.

A trial only counted if the agent actually issued the stop command. That exclusion rule matters because one early version of the harness caught a contamination case: sometimes the stimulus landed between agent utterances, which produced a fake long tail that was not a real barge-in. Those trials were excluded.

The final pooled result was 34 confirmed interruptions. Median interruption-loop latency was 1328 ms.

The target was 45 trials. I am publishing at 34 because the median held at 1328 ms at every checkpoint on the way there, at N of 19, 32, and 34. If additional trials are collected, they will be appended to the raw data on the branch.

That number is bad enough to notice. But the important part is the breakdown.

Component medians do not have to sum exactly to the total median, so do not overread the arithmetic. The shape is the finding.

Most of the time passed before my code had a signal it could act on. Once the application received the first interim transcript, the app-side interruption handling was essentially immediate. The agent issued the stop command in about 1 ms median.

That does not mean the whole system was fast. It means the slow part was earlier in the loop.

This was not mainly an LLM problem

That is the part I think people miss. If a voice agent feels slow, the instinct is to look at the model. Use a faster model. Stream tokens. Shorten the prompt. Tune the system message. Cache more. Parallelize tools.

All of that can matter. But none of it helps a loop that has not observed the user yet.

In this benchmark, the application could not decide to stop speaking until it received an interim transcript. The first interim transcript arrived about 1093 ms after the harness speak-start event.

The LLM was not the bottleneck in that loop. My Python code was not the bottleneck in that loop. The prompt was not the bottleneck in that loop. The loop boundary was the bottleneck.

The interruption loop observed the world through transcripts. That meant it could not interrupt on speech until speech had become text. That is the architectural lesson. Not "this provider is bad." Not "single-vendor is bad." Not "all voice agents have a 1.3 second barge-in floor." The lesson is narrower and more useful:

That sounds obvious once stated. It is less obvious when you are staring at a clean call-control API and everything looks like normal application code.

Transcript loops are still useful

I do not think the transcript-triggered loop is wrong. For a lot of phone agents, it is probably the right loop. If the product is mostly turn-based, the simplicity is worth a lot. Appointment reminders. Basic IVR replacement. Transactional flows. Notifications. "Press one" flows rewritten as speech. Anything where the agent speaks, the caller answers, and interruption is rare.

In those products, the transcript loop has real advantages:

• simpler state
• fewer moving parts
• easier debugging
• less custom media infrastructure
• more legible logs
• cleaner failure recovery

It also lines up with how most LLM applications are already built. The app receives text. The app reasons over text. The app emits text. The platform turns that text into speech. That is a good abstraction for many jobs.

The problem is when we pretend that abstraction is neutral. It is not. It decides when the agent can observe the human.

Frame loops buy speed by accepting mess

The obvious way to get below the transcript loop is to stop waiting for a transcript. Own the media stream. Watch audio frames directly. Run a voice activity detector or turn detector near the audio. Treat likely caller speech as an event before speech-to-text has produced words.

That changes the interruption loop:

Now the agent can potentially react much earlier. But the tradeoff is real. A frame-level interruption loop has to answer questions the transcript loop mostly avoids:

• Was that speech or noise?
• Was it the caller or background audio?
• Was it a backchannel or an interruption?
• Should "yeah" cancel the agent?
• Should a cough cancel the agent?
• Should speech from a TV cancel the agent?
• What if the agent’s own TTS leaks back into the input?
• What if the caller starts speaking, then stops immediately?

The faster loop is more responsive because it observes earlier. It is also more exposed because it observes less interpreted data.

The real design question

The question is not: which voice agent stack is fastest? That is too vague to be useful. The better question is: what signal closes the interruption loop?

• Does the agent stop because a transcript appeared?
• Does it stop because a VAD fired?
• Does it stop because a realtime speech model emitted an interruption event?
• Does it stop locally, or send a command to a remote call-control API and wait for a completion event?

Those choices are the architecture. They determine what can be optimized in application code and what is upstream of the application entirely.

In my measured transcript-loop implementation, the app could be perfect and still wait for the first interim transcript. That is the piece the benchmark made visible.

This changes how I evaluate voice-agent vendors

I used to think about voice-agent stacks mostly as a pipeline: transport, STT, LLM, TTS. That is still useful, but it is incomplete. Now I ask loop questions:

• Where does turn detection happen?
• What signal detects interruption?
• Can the app see raw audio frames?
• Can playback be cancelled locally?
• Can TTS generation be cancelled separately from audio playback?
• Does the system expose partial transcripts, speech-start events, VAD events, or only final turns?
• Are interruption events observable in logs?
• Can I reproduce the timing from raw data?

The vendor’s latency number matters. But how they get the number matters more.

A system that interrupts on transcript has one shape. A system that interrupts on acoustic onset has another. A system that hides the whole loop inside a managed realtime session has a third. Those are not just implementation details. They are product behavior.

This loop analysis is the core of how we evaluate production voice agents at ProofNorth.

What I am measuring next

The benchmark in this post measured the transcript-triggered loop. The next benchmark is the frame-triggered loop. Same task. Same kind of harness. Same reporting standard. Same uncomfortable honesty about what is measured and what is inferred.

The hypothesis is straightforward: if the application can observe speech before transcription, it should be able to interrupt earlier than a transcript-triggered loop. But I do not want to publish that as a number until it is measured head-to-head.

I am also re-testing how different transcription engines behave behind this call-control surface. Early results were inconsistent enough that I am not publishing an engine comparison until it is measured properly. That gets its own field note.

The important part for now is the loop boundary. In text agents, "observe" usually means the model received another message. In voice agents, "observe" might mean a transcript arrived one second later, or an audio frame arrived right now.

That head-to-head is the next field note. If the frame loop is faster, the data will show by how much. If it is not, that result gets published too.

Links

• offleashthe transcript-loop implementation measured here.
• BENCHMARK.mdmethodology, raw timing definitions, and reproduction notes.
• Raw benchmark datathe trial-level event timings behind the medians.
• telnyx-onsetthe frame-loop implementation I am measuring next.