Where the Latency Actually Goes: Deterministic State Machines, Barge-In, and Cancellation in Full-Duplex Voice Agents.

Abstract

In voice agents, we typically see latency as a model performance issue. We measure three metrics independently to show how responsive the system is: time to first token, speech recognition speed, and time to first synthesized audio. Now something else happens to the caller. The response will begin when the audio in the media path is detected. Then the responsiveness is re-measured, this time by re-measuring the time it takes for the current response to go silent during an interruption. Between these two points there are several separate operations happening, such as updating conversation memory, jitter management, speech synthesis, endpoint detection, transcription, language model inference, and audio pacing. In this paper, we examine the coordination problem that this design induces. The primary argument here is that low component latencies are not enough for full duplex voice interaction. The control and audio events are asynchronous, so the system must maintain the consistency of the active conversation turn. Crucially, cancellation has to be able to revert all work related to the pipeline, including text that has been generated, synthesis that is pending, audio that is queued, and the state of the conversation. The architecture proposed here is founded on three related ideas: a latency model from the caller’s perspective based on the critical path; a deterministic event reducer with explicit turn ownership; a playback-aware commitment boundary that separates generated content and content the caller hears. This paper also discusses adaptive jitter buffering, end-of-turn prediction, barge-in detection, and the uncertainty that comes from deploying voice systems in Georgian and other relatively low-resource languages.