Why we run WebRTC
and not WebSockets

A voice agent has roughly half a second. Below about 500ms from the caller's last syllable to the agent's first, people do not consciously register a gap. Beyond that they notice the pause, and past a point they assume the line is broken and either interrupt or hang up. Published thresholds are in the sources below. Every architectural decision downstream is a negotiation with that budget.

The budget is unforgiving once you write it out. Network round-trip, turn detection, the language model's time to first token and speech synthesis each take a slice of it. There is no slack anywhere, which is why the transport layer cannot be an afterthought.

What TCP does to a conversation

WebSockets run over TCP, and TCP guarantees ordered delivery. That guarantee is exactly the problem. When a packet is lost, everything behind it sits in a buffer until the missing packet is retransmitted and arrives, head-of-line blocking. On a file transfer this is invisible and correct. On a live call it produces a stall of anywhere from tens to hundreds of milliseconds, followed by a burst of buffered audio played back too fast. The caller hears the agent freeze and then gabble.

WebRTC carries media as RTP over UDP and makes the opposite trade. A lost audio frame is simply gone, concealed by the jitter buffer, and almost nobody notices it. A stall while TCP recovers that same frame is noticed by everybody. When the currency is timing rather than bytes, dropping data is the correct engineering choice.

What else you get for free

Choosing WebRTC is not only about UDP. The stack ships with two decades of hard-won real-time audio engineering already inside it, and reimplementing any of it over a WebSocket means rebuilding it badly:

• Adaptive jitter buffer and playout timing. A WebSocket has no concept of a frame arriving too early or too late. It hands you bytes. Smooth playback under variable network conditions is something you would have to write yourself.
• Media-aware congestion control. Google Congestion Control detects a link filling up before packets start dropping and lowers bitrate smoothly. TCP's approach is to saturate the pipe, notice the loss, and halve the rate, a sawtooth that sounds like exactly what it is.
• Acoustic echo cancellation, automatic gain control, noise suppression. Built in, tuned, and running on the client. Speakerphone in a car is a solved problem here and an open one over a raw socket.
• Opus, at a tenth of the bandwidth. Published comparisons put Opus speech at roughly a tenth of the bandwidth of the base64-encoded PCM that WebSocket pipelines typically push (see sources below). On a weak mobile uplink that difference is the difference between a call and an apology.

Where WebSockets are still the right answer

We are not arguing that WebSockets are bad; we are arguing they are being used in the wrong place. They remain the correct choice for server-to-server control traffic, for model APIs, for streaming transcripts to a dashboard, for event and state synchronisation, and for anything where a dropped message is a bug rather than an inaudible artefact. A production voice deployment runs both: RTP over UDP for the media, WebSockets for the control plane beside it.

Two operational notes. Where a corporate network blocks UDP outright we fall back to TURN over TCP, accept the degradation and record it in the deployment notes rather than quoting an unchanged latency figure. And for calls that originate on the public telephone network, the leg into our infrastructure is SIP, WebRTC governs the media path from our edge onward, not the carrier's copper.

What we are watching

WebTransport over QUIC is the interesting one. It offers unreliable datagrams with modern congestion control and finally reached broad browser support during 2026. It does not yet bring the audio pipeline, echo cancellation, gain control, jitter management, that makes WebRTC valuable, so it is not a replacement today. It is a serious candidate for the next generation, and our media layer is abstracted so that swapping it is an infrastructure decision rather than a rewrite.

English text-to-speech has an easy job by comparison: the vowels are on the page. Modern Hebrew is written almost entirely without nikud, the diacritical marks that specify vowels, and Arabic is written without tashkeel for the same reason. Native readers reconstruct the missing information from context so automatically that they forget it is missing. A synthesiser cannot.

The consequence is systematic ambiguity. The three letters שמן are, depending on the vowels you supply, shemen (oil), shamen (fat) or shuman (grease). ספר is sefer (a book), sapar (a barber) or siper (he told). Nothing on the page distinguishes them. Stress placement is unwritten too, and in Hebrew it is contrastive, the same letters stressed differently are different words. Get it wrong and the agent has not merely mispronounced something; it has said something else.

Two viable approaches, and why we use both

The research has converged on two strategies. The first is explicit grapheme-to-phoneme conversion: predict the diacritics, the stress and the reduced vowels, then synthesise from an unambiguous phonetic representation. Recent work in this direction, Phonikud is the notable 2025 example, showed that a lightweight model can add stress and schwa prediction to standard diacritisation while staying fast enough for real-time use. The second strategy skips diacritics altogether: a language model trained on discrete speech tokens learns to resolve the ambiguity from context implicitly, which is what HebTTS demonstrated using weakly-supervised in-the-wild recordings.

Both are correct and they fail differently, so we run a hybrid. A fast deterministic G2P stage handles the general case and gives us something we can inspect and correct. A domain lexicon overrides it for the words that must never be wrong, brand names, drug names, street names, product SKUs, the customer's own company, because a model that is right 97% of the time will still mangle your product name in front of a customer, and that is the one error people remember. Numbers, dates, currency and identifiers get their own deterministic expansion rules, since reading an Israeli mobile number aloud correctly is a formatting problem rather than a linguistic one.

Arabic gets the same treatment with a harder starting point: diacritisation interacts with case endings and dialect, and the vowels a Levantine speaker expects are not the ones Modern Standard Arabic prescribes. The pipeline is per-dialect for that reason, not per-language.

In English an agent can talk to anyone without knowing anything about them. In Hebrew, Arabic, Spanish, French, Russian and dozens of other languages that option does not exist. Verbs, adjectives and pronouns agree with the gender of the person being addressed, and there is no neutral default hiding in the grammar. Asking “would you like a callback?” requires committing to תרצה or תרצי before the sentence can be spoken at all.

Getting it wrong is not a cosmetic error. Callers read it as carelessness at best and as disrespect at worst, and it is the single fastest way for an agent to announce that it is a machine that was localised rather than built for the market.

How the decision is made

We resolve it in strict priority order, and the order matters more than any individual signal:

• Declared preference. If the person has ever stated how they want to be addressed, that is authoritative and permanent. Nothing overrides it.
• Record data. The CRM, the account, the booking. Reliable where it exists and already governed by your data policy.
• Acoustic inference, with a confidence score. Voice-based classification running alongside transcription. Useful, imperfect, and never treated as fact, it produces a probability, not a determination.
• Neutral phrasing below threshold. This is the important one. Both Hebrew and Arabic have constructions that carry no gender agreement; they are simply less convenient, so they have to be written into the response templates deliberately and in advance. Where confidence is low the agent uses them and remains entirely natural.

The design rule is that the agent never guesses in order to sound more fluent. A slightly more formal sentence costs nothing. Confidently addressing a woman in the masculine costs the call. The same machinery selects the synthesis voice and the register, and every one of these decisions is logged, so an unusual output can be traced to the signal that produced it rather than debated.

An agent can be grammatically flawless and still be obviously foreign. What gives it away is behaviour: how fast it talks, how long it lets a silence run, whether it accepts being interrupted, how directly it says no, and what it assumes about the person on the other end. These are not translation problems and no amount of translation fixes them.

What we actually tune per market

• Dialect before language. Modern Standard Arabic is understood across the region and spoken conversationally in almost none of it; an MSA-only agent sounds like a news bulletin. Levantine, Gulf, Egyptian and Maghrebi are deployed separately, with MSA reserved for formal contexts.
• Turn-taking and silence. Israeli phone conversation tolerates overlap and treats a two-second silence as a fault; several European markets treat the same overlap as rudeness. Endpointing thresholds and barge-in sensitivity are set per locale, not globally.
• Register and honorifics. French vous against tu. Arabic honorifics and the address forms that go with them. Hebrew, which is informal by default in a way that reads as disrespectful when transplanted directly into other markets.
• Code-switching. Israeli callers move between Hebrew and English inside a single sentence, particularly for technical and commercial vocabulary; Maghrebi callers do the same with French. An agent that resets to one language mid-utterance breaks the conversation.
• Directness. Whether a refusal is stated plainly or softened, and how a bad answer is delivered. The literal translation of a polite English hedge frequently reads as evasive.
• Formats and calendars. Phone numbers, dates, currency and identifiers read aloud the way locals say them. Working weeks that start on Sunday. National and religious holidays that determine whether the office is open at all.

All of this has to fit inside the same 490ms envelope, which is why it is engineered as light deterministic stages ahead of synthesis rather than as additional model passes. Locale adaptation that costs half a second is not adaptation; it is a regression with better manners.