AI Voice Agents: Why We Have Not Built One

Status: not built. No prototype, no demo, nothing running. This page is a study of the problem and an account of what would have to be true before we started. We would rather publish that than a page implying a system that does not exist.

Why voice is harder than the text equivalent#

A text assistant that takes four seconds to answer feels slow. A voice agent that takes four seconds to answer sounds broken — the caller assumes the line dropped and starts talking again, which corrupts the turn.

Conversation has a rhythm humans enforce unconsciously. Meeting it is a systems engineering problem, and it is the reason most voice demos are impressive in a quiet room and unusable on a real phone call.

Where the latency budget goes#

A response is not one operation. It is a chain, and each link adds:

• Detecting that the speaker has stopped — not a solved problem. Cut too early and you interrupt; wait too long and every turn feels sluggish. Natural pauses mid-sentence look identical to finished turns.
• Speech to text — streaming helps, but a final transcript still requires the end of the utterance.
• Understanding and deciding — including any retrieval or tool call.
• Text to speech — streaming the first syllables while generating the rest is what makes the difference between acceptable and not.
• Network, in both directions.

Comfortable conversational turn-taking sits in the region of a few hundred milliseconds. The budget is consumed by the chain, not by any single component, which is why swapping in a faster model rarely rescues a slow agent.

The failure modes that only appear on a real line#

Barge-in. People interrupt. An agent that cannot be interrupted mid-sentence is experienced as rude, and callers begin talking over it regardless, so you now have overlapping audio to resolve.

Background noise and other voices. A television, an office, a second person in the room. The transcript contains words nobody said to the agent.

Names, addresses, reference numbers. The information most calls exist to exchange is the information speech recognition handles worst. Any serious voice system needs a confirmation strategy for these, and confirmation costs turns.

Accents and code-switching. In many markets callers move between languages within a sentence. Recognition quality varies sharply, and the variation lands hardest on the callers least able to switch to a text channel.

Silence. A caller thinking, or distracted, is indistinguishable from a caller who has gone.

The recovery loop. When the agent mishears twice, most designs ask again. Callers do not tolerate a third attempt, and a route to a human is not optional.

The parts that are not technical#

Disclosure. In the European Union, transparency obligations under the AI Act have applied since 2 August 2026: people must be told they are interacting with AI. A voice agent that presents as a person is the clearest case of what that addresses. Any deployment needs disclosure designed in from the start, not appended.

Recording and consent. Voice is personal data, and in many jurisdictions recording carries specific consent requirements. This is a legal design constraint, not a feature.

Emotional register. Callers reaching support are frequently already frustrated. Cheerfulness in the face of a problem reads as indifference, and the tone that works in a text channel does not transfer.

What happens when it fails. The escalation path is the product. A voice agent without a fast, obvious route to a person converts a minor problem into a complaint.

What we would need before building#

• A use case where the caller genuinely benefits from voice rather than a form or a message
• A latency budget measured end to end on a real telephone path, not in a browser demo
• A confirmation strategy for names, numbers and addresses
• Barge-in working before anything else is evaluated
• Disclosure and recording consent designed in from the first prototype
• A route to a human that is faster than the caller expects
• Measurement of task completion and escalation rate, not of transcription accuracy

We have not had a problem that justified this. Publishing a demo without one would be a capability claim rather than an experiment.

Where voice is genuinely a good fit#

Hands or eyes are busy — driving, warehouse, clinical, field work. Here voice is not a novelty channel, it is the only usable one.

Accessibility — for some users voice is the accessible interface, and this is an underrepresented reason to build.

High-volume, narrow, verifiable transactions — a small number of intents, with confirmation. Booking, status, simple changes.

Where the alternative is a queue. Even a limited agent beats fourteen minutes of hold music, as long as escalation works.

FAQ#

Do you offer voice agents?#

No, and this page exists so nobody has to ask twice. We have not built one.

Is the technology not good enough yet?#

The components are good. Recognition, generation and synthesis are all strong. The difficulty is integrating them within a conversational latency budget while handling interruption, noise and recovery — a systems problem rather than a model problem, and it is where most projects underestimate the work.

How much latency is acceptable?#

Comfortable turn-taking is a few hundred milliseconds. Beyond roughly a second, callers begin talking over the agent. The number that matters is measured on a real line, including network, not in a local demo.

Should a voice agent say it is AI?#

Yes — and in the EU, transparency obligations have applied since 2 August 2026. Beyond compliance, callers work out that they are talking to a machine quickly, and the ones who feel deceived escalate harder. Confirm your specific obligations with counsel; this is general information, not legal advice.

What is the most common mistake?#

Optimising transcription accuracy while ignoring turn-taking. A system with excellent recognition that cannot be interrupted and pauses awkwardly is worse to use than a less accurate one that feels conversational.

What should we measure?#

Task completion rate and escalation rate. Word error rate is a component metric and a poor proxy — callers do not care whether every word was transcribed, they care whether the thing got done.

Would you build one for a customer?#

Only with a use case meeting the conditions above, and we would say plainly that we have not shipped one before. Publishing an experiment log rather than a capability page is the point of this section.

Related Articles#

See AI agents for the bounded-autonomy discipline that applies to any agent with tools, and AI Testing for evaluating systems whose output varies between runs.

What else is coming for AI Voice Agents