VAD stopped the voice agent 640 milliseconds earlier

The problem with the two-repo fight

The last time I measured this, I tested a transcript-triggered voice agent built on Telnyx Call Control, offleash. It had a 1.3 second interruption floor.

You might expect this post to pit that old transcript repo against a new frame-based repo, onset, winner takes the headline.

That is not what this is.

A head-to-head between two different architectures is a whole-system observation, not an experiment. If the old repo loses by 900 ms, is it because of the transcription surface? The TTS path? The media ownership? The provider events? A comparison cannot tell you.

That two-repo comparison was formally preregistered, demoted to secondary for exactly this reason, and gated on offleash being rerunnable through the same acoustic boundary, which it was not. So the last post’s 1328 ms stays historical context, and it should not be compared with any number in this post, including this post’s transcript condition: different agent, different measurement boundary, different transport.

Instead, the primary comparison flips exactly one thing inside a single media-stream agent, onset:

• Condition A: a local VAD decision on the raw audio frames is the sole interruption-eligible trigger.
• Condition B: the arrival of the first eligible transcript is the sole eligible trigger.

Shared between the two conditions: the runtime, the call and media topology, STT connection, TTS path, playback queue, caller fixture, and analysis rules. One variable moves. The result is actually attributable.

The obvious measurement was invalid

The original plan was to measure the interruption at the provider’s media-stream surfaces. Twenty-six bounded live attempts proved this was a bad idea. Four behaviors controlled the outcome:

• A bidirectional stream did not expose a usable provider-outbound track for the same leg.
• Inbound-direction surfaces on the agent leg mirrored the agent’s own injected audio rather than providing an isolated caller reference.
• Receive-only streams inherited the leg’s media context and did not honor the codec override the plan needed.
• A Call Control-answered harness endpoint did not transmit RTP, which starved the delivery-gated design.

The provider surfaces were not broken. They behaved consistently; they could not satisfy this measurement topology, which needed an isolated view of what a caller actually hears. To get that, the measurement had to move to the caller’s side of the wire.

The rescue: a common acoustic boundary

I replaced the provider-side harness with an external SIP media endpoint acting as the caller.

This harness places the call, transmits the synthetic caller fixture ("I need a table for two") 1,000 ms into the agent’s greeting, records every frame of returned agent audio, and timestamps everything on a single monotonic clock.

This is critical: the headline metric runs from the harness-side emitted-stimulus boundary to the harness-side returned-audio stop boundary. No cross-process clock math. No webhook timestamps. No provider-reported durations.

One eligibility rule matters for reading the tables below. The fixture is timed to land mid-utterance, but the agent’s greeting has natural pauses. If the fixture happened to land on one, there was no active speech to interrupt, so the trial was marked ineligible. Ineligible trials stayed in the dataset and the denominator; they were never rerun.

The detector calibration loop

Detecting when the agent "stopped talking" requires a silence detector. To be clear about roles: this detector is part of the measurement, deciding when returned audio actually stopped. It is not the agent’s VAD, which is the interruption trigger under test in Condition A. I used a finite grid over labeled audio spans to find the simplest passing threshold and shortest passing hold.

Here is what it took before I believed the detector:

1. 01Campaign 1: 300 ms hold

   Passed the initial labels, but qualification caught a defect: the detector labeled a natural pause in the agent’s speech as a stop. Old response audio resumed immediately after. Campaign discarded in full.

2. 02Campaign 2: 400 ms hold

   The recalibrated grid caught three more natural-pause false stops, plus a classifier omission for terminal failures. Campaign discarded in full.

3. 03Final campaign: 600 ms hold

   The final detector passed qualification entirely. Zero false stops, zero stale-audio resumptions, and zero lost turns among eligible trials.

Both recalibrations were driven by pooled natural-pause evidence, never by between-condition latency results, so the detector could not drift toward a preferred answer. No latency from a discarded campaign ever entered the final analysis.

The 600 ms logic: the 600 ms value is a confirmation interval. The detector reports the stop boundary at the beginning of the silent run, not 600 ms after it. If a trial measures a 400 ms latency, it still required 600 ms of future silence to confirm that the 400 ms mark was the actual stop.

The final results

For the final run, I scheduled 40 attempts per condition in a balanced-block order under a frozen seed. Failed and ineligible attempts were never replaced. In voice systems, reliability metrics are as critical as speed metrics. Keeping those rows in the dataset costs nothing and buys everything: a benchmark that quietly retries its failures is reporting its retry policy, not its system.

The trials were independent and unpaired, and the preregistered analysis reports bootstrap intervals for each condition’s median, 361.5 to 440.0 ms for VAD and 900.6 to 1211.5 ms for transcript, rather than a hypothesis test. I am not claiming statistical significance. I am showing two medians whose intervals sit far apart, with all 80 trials, including every failure, in a committed per-trial CSV.

Look at the spread, not just the medians. The VAD condition is tight, with a worst successful trial at 720 ms. The transcript condition is wide, with a p90 near 1.7 seconds. A transcript-triggered agent does not have a barge-in latency; it has a barge-in distribution, and the tail is what callers remember.

Architectural reality and product takeaways

The application code itself was not the bottleneck in either condition. Once the trigger fired, the internal bookkeeping was sub-millisecond:

Measurement warning: do not subtract these internal agent timestamps from the external SIP harness clock. They live in different processes. They merely prove the application acts immediately once allowed.

The gap exists because of when the agent is allowed to decide. The VAD condition acts as soon as sustained caller energy shows up in the frames. The transcript condition has to wait for speech to become text on a remote server and travel back.

The scope is deliberately narrow: this harness, this fixture, this VAD configuration, this implementation. It is not a claim that every VAD beats every STT engine, and it says nothing about other thresholds, voices, providers, or physical devices with speakerphone echo. The full limitations list is in the repo’s benchmark plan.

The harness is part of the release. It is a SIP caller that measures a dialable voice agent acoustically, from the caller’s side of the wire, and the agent under test is untouched. It dials a number, injects a timed fixture, records every returned frame, and timestamps everything on one clock. It currently drives onset, and nothing in the measurement path knows what it is calling. The spec, the qualification protocol, and the per-trial manifests are all in the repo.

The product takeaway: for raw speed, the earlier acoustic signal wins clearly. But a transcript loop still buys you something.

VAD mostly says that speech-like energy exists. It does not know whether the caller is saying "stop," giving an agreeable "mm-hmm," talking to someone else in the room, or just dropping a pan on the floor.

Waiting for words costs you about 640 ms in this implementation. Whether every sudden sound should cancel your agent remains your product decision.

This kind of loop analysis is the core of how we evaluate production voice agents at ProofNorth.

What I am measuring next

The engine question from the last post is still open: interim-forwarding behavior for non-Google STT engines through Call Control transcription webhooks was inconsistent in early testing, and it needs a real sample size before anything gets published. That is the next field note.

Methods appendix

Fixture "I need a table for two," injected 1,000 ms after confirmed active agent playback; frozen greeting natural-end reference 3,200 ms. Wire audio PCMU 8 kHz mono; analysis PCM16 16 kHz mono; 20 ms frames. Detector: RMS, minus 42 dBFS activity and silence thresholds, 100 ms arm, 600 ms hold, selected by a preregistered rule over a condition-independent grid and validated in a separate qualification population. Final collection seed 2026071306; analysis seed 2026071305; 10,000 bootstrap resamples per condition median; no replacement policy throughout.

• Preregistration, comparison hierarchy, gates, failure taxonomy, and limitations: BENCHMARK_PLAN.md in the onset repo.
• Final analysis: bench/PHASE3_REPORT.md.
• Provider-surface attempts and the SIP amendment: bench/PHASE2_REPORT.md.

Links

• onsetthe full-duplex media-stream agent measured here, plus the SIP harness, spec, and per-trial manifests.
• Per-trial dataall 80 trials, including every failure and ineligible attempt.
• offleashthe transcript-triggered Call Control agent from the last post.
• The previous field notewhy voice agents are competing loops, and the transcript-loop measurement that started this.