Why latency decides whether a voice agent sounds human

Latency decides whether a voice agent sounds human because people expect the gap between turns in a conversation to be very short. When the agent pauses too long, the caller wonders whether the line has dropped, says "hello?" and starts talking over the reply, and the call loses its rhythm. On a real phone call the delay comes from several places: the phone network, deciding that the caller has finished, understanding what they said, choosing a reply, looking up facts, and producing the audio. Good latency is short, consistent, and never cuts the caller off.

This piece explains each source of delay in plain terms, with what we have learned tuning it on real Indian calls. For how a voice agent works end to end, start with what an AI voice agent is.

People take turns fast

Stivers and colleagues (2009) studied conversation in ten languages, ranging from small indigenous communities to major world languages. The same pattern held in all of them: people avoid talking over each other and keep the silence between turns to a minimum. The average gap between turns did differ by language, but within a range of 250 milliseconds of the cross-language mean.

That is the standard an AI voice agent is judged against, whether or not the caller ever thinks about it. The gap between the caller finishing and the agent starting is voice latency. Callers do not measure it. They feel it.

What a slow reply does to a call

• The caller assumes the line has dropped and says "hello?" or repeats the question.
• The agent's answer then starts while the caller is talking. A good agent stops when interrupted, so both stop, and both start again.
• The caller decides they are talking to a machine and changes how they talk: shorter answers, less patience, a quicker hang-up.
• On a sales call, rhythm is credibility. An agent that hesitates before a price sounds unsure of it.

In the calls we have reviewed, the worst moments were rarely a consistently slow agent. They were one long silence in an otherwise normal call, with the caller filling it by asking whether anyone was there.

Where the delay comes from

Think of each reply as a relay. Every leg adds time, and the caller hears only the total.

The phone network

A real call travels from the caller's handset across their mobile network to your carrier, over SIP trunking to the server running the agent, and back again. Each leg adds a little, and the distance between where the call lands and where the agent runs adds more. A browser demo skips most of this, which is one reason demos feel quicker than calls.

Knowing the caller has finished

This is the least discussed part and, in our experience, the one that matters most. The agent cannot answer until it decides the caller has stopped, and the usual signal is a short stretch of silence. That sets up a direct trade-off.

• Wait too briefly, and the agent answers into a pause in the middle of a sentence. Callers often pause mid-sentence, especially while reaching for an English word or a number in a Hindi or Gujarati sentence: "mera budget ... around 80 lakh hai" (my budget is ... around 80 lakh). An agent that jumps in there is not being quick. It is not listening. We lengthened this window after real calls showed exactly this.
• Wait too long, and every reply on every call carries that extra silence.

Two more details matter on Indian calls. First, "haan", "ji" and "hmm" (yes, yes, hmm) said while the agent is talking are agreement, not an attempt to take over, so the agent should need sustained speech before it treats sound as an interruption. Second, when the agent is quiet and waiting, a one-word answer such as "haan" (yes) or "2 BHK" has to count at once, or the caller answers and hears nothing back. One threshold cannot do both jobs. We use a stricter bar while the agent is speaking and a lighter one while it is listening.

The start of a sentence must not be lost while the system is still deciding that someone has begun to speak. In Hindi that first word can be the negation or the honorific, which changes everything after it. Holding a little audio in reserve and replaying it once speech is confirmed keeps that word.

Noisy lines add one last trap. Traffic, a television or a speakerphone in a busy office can keep the "still speaking" signal alive, so the turn never ends and the agent never answers. From the caller's side it sounds as if the agent hung up while the line stayed open. A ceiling on how long one turn can run, and a rule that speech must be clearly louder than the background to count as continuing, stop a noisy line from becoming a dead one.

Understanding the speech

What the caller said has to become something the agent can act on. Some agents do it in three separate steps (turning speech into text, choosing a reply, turning the reply back into speech), and every handoff between steps adds time. Others listen and speak in a single step, which removes handoffs but not the other legs of the relay. Either way, understanding that starts while the caller is still speaking, instead of waiting for the whole sentence, saves time.

Choosing a reply

The agent reads its brief and the conversation so far, then produces a reply. Three things slow it down:

• A long brief. Every line of it is read before every reply, so a bloated brief is a slower agent as well as a more expensive one.
• A growing conversation. On long calls the history keeps growing, and replies can get slower turn by turn. Teams we work with have heard exactly this late in long calls. The fix was to cap how much history the agent reads, while keeping everything the caller said in a separate running note, so capping the history loses nothing that matters.
• Thinking before speaking. An agent that reasons at length before answering takes longer to start. Putting a bound on that reasoning keeps replies quick.

Anything sent to the agent mid-call adds to its reading, too. A correction such as "the caller has switched to Hindi" should be one short sentence, not a paragraph, because every word of it is time the caller spends waiting. We explain those notes in how a voice agent remembers and follows a language switch.

Looking things up

A good agent checks your documents before it states a fact, and that lookup happens while the caller waits. When we timed our own calls, preparing the search itself was a noticeable share of the pause. Callers ask the same handful of questions, so remembering recent lookups helps. So does a hard time limit, after which the search falls back to a simpler method instead of leaving the caller in silence. Writing to the CRM should never make anyone wait; it can happen alongside the conversation.

Producing the audio

The reply has to become speech, and the first sound has to reach the caller. Streaming audio as it is produced, instead of waiting for a whole sentence, shortens the wait. There is a counter-pressure. If audio arrives in uneven bursts, playing each fragment the instant it lands makes the voice choppy, with holes in the middle of sentences. Holding a small cushion of audio at the start of each reply smooths that out, at the cost of starting a little later. We chose smoothness: a caller forgives a short pause far more easily than a voice that breaks up.

Averages hide the pauses callers remember

"The latency is high" is the right complaint in the wrong unit. The pause a caller notices is made up of the silence window, the agent thinking, lookups, the network and any audio queued ahead of the answer, and each has a different fix.

So measure every reply from the last thing the caller was heard to say to the first sound of the answer. Record the slow replies individually, with what was happening at the time: a lookup in flight, a long answer, a noisy line. Look at the worst replies, not the mean. A caller does not experience an average.

What good looks like

• Replies start promptly once the caller has finished, without cutting into pauses mid-sentence.
• No reply is dramatically slower than the rest.
• "Haan" and "ji" (yes, and a polite yes) do not stop the agent; a real interruption does.
• A one-word answer gets a reply.
• A slow lookup has a limit, and the caller is never left in long silence.
• A caller who goes silent is nudged, then given a polite sign-off before the line is closed, instead of being left on an open line.
• The last replies of a long call come as quickly as the first.

We have left out millisecond targets on purpose. Phones, networks and callers vary too much for one number to mean much. A vendor who can show you measured reply gaps from real calls, including the slowest ones, is managing latency. A vendor who quotes one number from a lab may not be.

How to test it yourself

1. Call on a mobile network, not from a browser on office Wi-Fi.
2. Ask something that needs the brochure.
3. Ask three things in one breath.
4. Pause mid-sentence as if looking for a word, and see whether it jumps in.
5. Say "haan" (yes) while it is talking, and see whether it stops.
6. Answer a question with a single word.
7. Stay on the line for ten minutes, and compare the last replies with the first.
8. Go silent, and see what it does.

The full checklist is in how to evaluate an AI voice agent vendor.

What to do next

Put the agent on a real phone call and listen for the pauses as closely as the words. If you sell property, see how BlackWolf approaches calls for real estate.