Show HN: Real-time AI Voice Chat at ~500ms Latency

The best, most human-like AI voice chat I've seen yet is Sesame (www.sesame.com). It has delays, but fills them very naturally with normal human speech nuances like "hmmm", "uhhh", "hold on while I look that up" etc. If there's a longer delay it'll even try to make a bit of small talk, just like a human conversation partner might.

My take on this is that voice AI has not truly arrived until it has mastered the "Interrupting Cow" benchmark.

Spot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.

> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning.

I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multiple generations internally and starts generating after every word, or at least starts asking sub-LLM models "Do I have enough to reply?" and once it does it generates a reply and interrupts.

I'm not sure how most apps handle the user interrupting, in regards to the conversation context. Do they stop generation but use what they have generated already in the context? Do they cut off where the LLM got interrupted? Something like "LLM: ..and then the horse walked... -USER INTERRUPTED-. User: ....". It's not a purely-voice-LLM issue but it comes up way more for that since rarely are you stopping generation (in the demo, that's been done for a while when he interrupts), just the TTS.

Human-to-human conversational patterns are highly specific to cultural and contextual aspects. Sounds like I’m stating the obvious, but developers regularly disregard that and then wonder why things feel unnatural for users. The “median delay” may not be the most useful thing to look at.

To properly learn more appropriate delays, it can be useful to find a proxy measure that can predict when a response can/should be given. For example, look at Kyutai’s use of change in perplexity in predictions from a text translation model for developing simultaneous speech-to-speech translation (https://github.com/kyutai-labs/hibiki).

Great insights. When I have a conversation with another person sometimes they cut me off when they are trying to make a point. I have talked to ChatGPT and grok at length (hours of brain storming, learning things, etc) and AI has never interrupted aggressively to try to make a point stick better

This silence detection is what makes me unable to chat with AI. It is not natural and creates pressure.

True AI chat should know when to talk based on conversation and not things like silence.

Voice to text is stripping conversation from a lot of context as well.

I would also suspect that a human has much less patience for a robot interrupting them than a human.

Neat! I'm already using openwebui/ollama with a 7900 xtx but the STT and TTS parts don't seem to work with it yet:

2025-05-05 20:53:15,808] [WARNING] [real_accelerator.py:194:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.

Error loading model for checkpoint ./models/Lasinya: This op had not been implemented on CPU backend.

This looks great. What hardware do you use, or have you tested it on?

Would you say you are using the best-in-class speech to text libs at the moment? I feel like this space is moving fast because the last time I was headed down this track, I was sure whisper-cpp was the best.

This 100%, yes!

I've found myself putting in filler words or holding a noise "Uhhhhhhhhh" while I'm trying to form a thought but I don't want the LLM to start replying. It's a really hard problem for sure. Similar to the problem of allowing for interruptions but not stopping if the user just says "Right!", "Yes", aka active listening.

One thing I love about MacWhisper (not special to just this STT tool) is it's hold to talk so I can stop talking for as long as I want then start again without it deciding I'm done.

>>they don't allow talking naturally

Neither do phone calls. Round trip latency can easily be 300ms, which we’ve all learned to adapt our speech to.

If you want to feel true luxury find an old analog PTSN line. No compression artifacts or delays. Beautiful and seamless 50ms latency.

Digital was a terrible event for call quality.

Maybe we should settle on some special sound or word which officially signals that we're making a pause for whatever reason, but that we intend to continue with dictating in a couple of seconds. Like "Hmm, wait".

Could that not work with simple instructions? Let the AI decide to respond only with a special wait token until it thinks you are ready. Might not work perfectly but would be a start.

Honestly I think this is a problem of over-engineering and simply allowing the user to press a button when he wants to start talking and press it when he's done is good enough. Or even a codeword for start and finish.

We don't need to feel like we're talking to a real person yet.

Yeah, when I am trying to learn about a topic, I need to think about my question, you know, pausing mid-sentence. All the products jump in and interrupt, no matter if I tell them not to do so. Non-annoying humans don't jump in to fill the gap, they read my face, they take cues, then wait for me to finish. Its one thing to ask an AI to give me directions to the nearest taco stand, its another to have a dialogue about complex topics.

That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming realtime transcription as a trigger. That adds a bit of latency but that gets compensated by way better accuracy.

Edit: just realized the irony but it's really a good question lol

Didn't expect it to be that good! Nice.

Does Dia support configuring voices now? I looked at it when it was first released, and you could only specify [S1] [S2] for the speakers, but not how they would sound.

There was also a very prominent issue where the voices would be sped up if the text was over a few sentences long; the longer the text, the faster it was spoken. One suggestion was to split the conversation into chunks with only one or two "turns" per speaker, but then you'd hear two voices then two more, then two more… with no way to configure any of it.

Dia looked cool on the surface when it was released, but it was only a demo for now and not at all usable for any real use case, even for a personal app. I'm sure they'll get to these issues eventually, but most comments I've seen so far recommending it are from people who have not actually used it or they would know of these major limitations.

Dia is too slow, I need a time to first audio chunk of ~100 milliseconds. Also generations fail too often (artifacts etc)

This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality I'd say there are better models out there.

I doubt TTS will be fast enough for realtime without a Nvidia GPU