Designing Voice Agents for Real Conversations - Chintan Agrawal & Daniel Wirjo, AWS

AI Engineer · 32 min · 297 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:03Hey everyone.
00:04I'm Chintan and I have my colleague Daniel with me.
00:07We are solutions architect on the AWS APJ startup team.
00:11So today we are going to talk about something we've been working on for a while.
00:15Which is turn taking in voice agents.
00:18What I mean by that is how does the system actually know that you have finished speaking and it's time for the agent to respond?
00:25These are all audio engineering problems.
00:27They are not LLM problems because you can have the perfect model, perfect track but the experience still might feel broken if the turn taking is off.
00:38So we are going to walk you through the concepts and three different approaches to solving this.
00:43And Daniel is going to you know make it more concrete by running all the three demo for you.
00:56So to kick things off uh we'll start with the 200 millisecond constraint which is basically the physics of why voice is hard.
01:06Then we look into the pipeline architecture, what are the components, where turn taking actually lives.
01:12We'll then look at three level of solving this problem.
01:15Going from simple silence detection all the way up to running your own turn detection model.
01:21And then we'll also talk about the latency, the budget and production issues that we've seen.
01:30So but before any of that, I just want to show you the problem statement more clearly because you have the same user.
01:37You have the same sentence on the both side but the user outcome is very different because on the left when the user says I want to fly and midway they try to correct themselves,
01:51the agent just does not notice.
01:54Because it keeps going on for almost 2 seconds while the person is sitting there trying to get a word in.
02:00On the right side, the same exact interaction is happening, but the agent is able to capture the interruption in under 200 milliseconds.
02:09And immediately backs off so the user can now speak.
02:13Both the scenarios, the LLM was identical.
02:15It was the same model.
02:16It was the same prompt, but the difference in the user experience is so different.
02:21And the difference is purely because of the audio pipeline.
02:25And how fast does it notices like someone else is talking and it knows when to shut up.
02:31Like that's the problem we're trying to solve today.
02:37So, we'll cover this 200 millisecond constraint because 200 milliseconds is how fast humans switch turns with each other in a conversation.
02:47And the implications are pretty brutal because at 800 milliseconds, things start to feel off.
02:53While at 1.5 seconds, your user just hang up on you because you have like your HR chat agents might get 5 seconds to respond and nobody will care.
03:05But with the voice agents, you don't get that luxury.
03:08We've seen recently like Salesforce was able to pivot the pipeline and they published their results in March 26th.
03:15But even there, best measured response time was 755 milliseconds.
03:19So, that's like almost 4x slower than how humans naturally take turns when talking.
03:26So, the question here becomes how do we close this gap and where we cannot close it like at least how do we get the turn taking right?
03:34So, the user experience does not feel as bad as the raw numbers might suggest here.
03:40So, if you're working with a budget of 755 milliseconds and every millisecond count, you also need to understand what's actually in the pipeline.
03:48Like where does the time go and where does this turn taking fit?
03:54So, the pipeline itself you probably might know the main components, the STT, the LLM, the TTS.
04:01You have the audio in, text out, text in, audio out.
04:04But the piece that's often missing like from people while people are thinking about this is this voice activity detection.
04:13Commonly called as VAD.
04:15It's a tiny component that sits right at the front and it is the one that controls turn taking.
04:21Its job is to basically detect whether the user has stopped talking or not.
04:26And the two other thing that you set alongside main pipeline are also very critical for turn taking.
04:33So, you have the smart turn detection which watches the VAD signal.
04:37Plus the audio features and then it has to make that decision that whether should it respond now or should it wait because the user might still be talking.
04:47And the interruption handler is triggered when someone barges in.
04:51So, what it'll do is it'll propagate a flush of downstream like the TTS would stop and the LLM generation need to be canceled so that the pipeline is ready for the new input
05:01within about, you know, 50 ms. So, if you remember I was talking about like how there are different levels to, you know, building this turn taking in voice agents.
05:15So, we'll start with a level one.
05:17Level one is your Silero VAD.
05:19So, this is a simple component which you fully own and like a lot of production system even today we see are running agents just by using this.
05:30So, coming to Silero VAD, it's a small 300,000 parameter model.
05:37It takes in like a short term Fourier transform.
05:40It has a and it takes raw audio and it'll convert it into a spectral features.
05:46It has four convolutional layers to pick up the pattern and an LSTM that gives it memory so across the frame so it's just not looking at one chunk in isolation.
05:57And then it has a sigmoid that gives a probability of speech and it's like a very small 2 megabyte model.
06:05And the one parameter that minimum silence millisecond is basically the entire user experience of level one.
06:12If you keep it very low, the agent is going to be snapping like it will cut people off while they're still, you know, thinking.
06:19But if you keep it very high, the agent might become super patient and it will never interrupt but you might get to uh get like a dead air where people will wonder if the agent is still connected or not.
06:32So there's like no universally right answer.
06:34It depends on the domain.
06:35If you're working on it from a perspective of a sales agent, you probably want it to be like 200 milliseconds.
06:44If you're in a domain where you need some to give some time to your user to respond, it could be like 1 uh thousand to 1,200 milliseconds.
06:53It depends entirely on what you're building.
06:57So VAD works well like that's why a lot of people are using it and it's generally good at what it does that is to return fast response to the basic question of whether there's someone talking right now.
07:10But uh there are things like it was never designed to handle because there'll be situations where just knowing that is there a silence will not be enough because you also need to answer these questions
07:22of how long to wait like uh the silence duration.
07:26If someone is pausing for 300 milliseconds or 400 milliseconds, VAD is seeing the same exact thing.
07:33It has no idea like whether the person is catching up their breath or whether they've completely finished their thought.
07:41So like they cannot distinguish that.
07:45And the second scenario is like, what if they, you know, the user talks over the agent, like, it barges in.
07:52It does not know how to handle it.
07:54And the same silence can have a different intent.
07:59And it can be like completely different things.
08:01It could be like a completed sentence, it would be an incomplete thought, it could be like a thinking pause.
08:07Or maybe it could be something as basic as like a backchannel acknowledgement.
08:12Uh the situation of the silence is same, but the intent again was very different in all the scenarios, but VAD considers all four equally, and it literally cannot tell them apart.
08:27So, those are the limitation of level one.
08:30And with the level two, basically what you get is that you let the STT service tell you like when the turn is over.
08:39So, Cartesiant does turn detection right inside their STT websocket.
08:45So, when they stream your audio in, the server would handle both the transcription and the turn detection together, and it will emit event when it thinks like the turn is done.
08:57Similarly, Deepgram does the same thing with their what they call as their I think at the endpointing.
09:03And their P50 latency for Cartesiant is about 300, and it's about 250 for Deepgram Nova 3. Both of these are also working really well.
09:12Like, we see a lot of customers use that.
09:16Because they are using like full audio signals plus some linguistic context to make these decisions.
09:22So, which is like much more information than your VAD will ever have.
09:27However, the trade-off is with the transparency.
09:29Because when it is working, it works great.
09:32But when it misfires, or when it cuts someone off at the wrong moment, then you like have no way to figure out because there's no law that will say
09:41uh it was done because of this reason, because of the what was what it saw.
09:47And that decision was made inside someone else's server, and you just got to live with that, basically.
09:55So, that brings us to level three.
09:57So, in level three, you keeps a little VAD running locally for basic signal that is there a speech.
10:05And you add Smart Turn on top of it, which is a small model that runs during silence.
10:12So, Smart Turn V3.2 is the latest model, like, and uh let me share some numbers with you.
10:19And it has about 58.9% recall and 68.4 percent precision.
10:26So, what this means is that about six out of 10 time when someone finishes a sentence, Smart Turn will be able to catch it quickly.
10:34But the other four times, it might not be confident enough, but that's also fine because you still have your VAD timer, which is running underneath as a safety net.
10:44If the Smart Turn doesn't fire, your stop second will kick in, like, at the configured latency anyway.
10:50So, it could be like 300 ms. So, in that scenario, you are never stuck waiting.
10:56You just, like, get the fast respond when the model is confident.
10:59And you get a slightly slower but a safer respond when it's not.
11:05The other number on this list is from, like, a Meta's published paper earlier this year, um around March.
11:12There, they had reported a higher recall at 87.7%, but they haven't released the code, so you really cannot deploy it.
11:22Smart Turn is BSD-2 licensed, and it's a 8-MB small model.
11:26You can pip install it today as well.
11:29So, yeah, like, that works out better for a lot of our customers.
11:33We decide when to start talking.
11:35But like, what about the other direction?
11:37What happens when the user interrupts and while you are already, you know, the agent is talking?
11:44So, on the left side, you can see what happens mechanically.
11:47Like, user opens you mouth, the windy will pick it up in 32 milliseconds and then then like 15 milliseconds, the entire pipeline is flushed.
11:57TTS stops, LLM cancels, everything is clean.
12:00Like, the user never hears the agent.
12:02You know, that part is taken care by Pipe Cat.
12:06But, the important thing is like, should it have stopped there on because think about a normal conversation when someone says, "Yeah."
12:14Like, if they send an acknowledgement while you're speaking, you don't stop, right?
12:20You know, they're just agreeing, but if you they say like, "Okay, wait, no. Like, that was wrong."
12:25You have to stop immediately.
12:27Same thing here.
12:28Like, the red is the it's a correction.
12:31Stop on everything.
12:32They amber could be just some sort of a filler background noise.
12:36But, you want to go ahead and keep going, maybe in a bit quieter.
12:41Blue means like a signal for cough for a background noise.
12:45So, you can just ignore it and finish your sentence.
12:48So, at level three because you own this classification, you can start distinguishing between a real interruption and someone just, you know, going um.
12:59So, today most of these systems stop every time, but this is a piece that's improving.
13:05Like, we are seeing a lot of incremental improvement in this.
13:17So, by now that you've seen all three levels and if you look at these three Python files, pipeline is almost identical in all of them.
13:25The only thing that differs is how you answer all of these questions.
13:28When is the user done talking?
13:31In file one, you pass in Celero VAD analyzer and that says the user had been silent for 300 milliseconds, so you can assume they're done.
13:41In file two, you swap it with the Cadence turns STT service class and now the service is going to tell you when the turn is over.
13:50And in file three, you keep your Celero VAD analyzer, but you also add like a local smart turn analyzer version three, which is sort of a small model that runs during your silence and it's going to look at the prosody
14:07and it's going to look at prosody and intonation to decide whether that pause meant I'm done or I'm still thinking.
14:15So, you have the code-wise the same pipeline, but the configuration changes, which you know, leads to completely different behavior.
14:26So, to summarize your three levels, where our level one was Celero VAD, so you basically own the silence detection completely.
14:34Level two, you let the STT provider handle it for you.
14:38And in most cases, it it will speak smarter, but then you cannot see what happened inside it.
14:46Level three is your VAD plus smart turn, so where we basically own everything, like you have full portability.
14:54And now coming to the latency piece, because we've been throwing around these numbers, 755 milliseconds, 1.3 seconds, but let me show you where all the time actually goes in.
15:08So, this is a breakdown from Quintela, who was a co-founder of Daily.co and is the person who created Pipe Band.
15:15And these are some of the numbers based on their production measurement.
15:19So, you have the numbers for making an which is about 40 ms. Then you have network and jitter buffer of 52.
15:27So, these are physics.
15:28Like you can't really change them much.
15:30Then you have your transcription plus end pointing.
15:34That's about 300 ms. Again, that's your STT doing its thing.
15:40LLM time to first byte, and this is generally the dominant bottleneck because in a typical API setup, you're looking at 500 to 650 ms depending on what model you're calling and from which cloud provider, which region.
15:56And then you have your network out and playback TTS.
16:02That's it's going to take about 120 to 90 85 ms. So, you're totally looking at roughly 1,100 to 1,300 ms in a standard setup calling cloud APIs.
16:15Now, the Quindos team had demonstrated about 500 ms total voice-to-voice by co-locating all models in the same GPU cluster.
16:25Because they were able to eliminate the network hops within the service.
16:30And it's sort of an achievable floor because if you're willing to invest in infrastructure, but for most of the developers today calling APIs, you're somewhere in the range of 800 to 1,300
16:41ms. And the key insight to note here is that the STT, LLM, together eat about 2/3 of the is latency budget.
16:52So, these are the really the only two levers that we have to move this latency metric needle in a meaningful manner.
17:03So, coming to like which LLMs are actually fast enough for this?
17:08We benchmark current models specifically for voice, and the target we kept it as under 700 ms time to first token.
17:17Because anything slower will push our total response past where the user is going to start noticing.
17:23So, for now, like recording in June 2026, what we've observed Nemotron-3 Ultra gives us a 529 P50 latency.
17:36GPT-4.1 at about 536.
17:39But also, like what matters more in voice uh than P95.
17:56So, which LLMs are actually fast enough for this?
17:59We benchmark current models, specially uh for voice, and So, coming to the So, coming to the LLMs models, which are actually fast enough for the We benchmark current models
18:24well for voice.
18:25Uh so, these benchmarks are from June 26, and the target that we kept was under 700 ms for time to first open, because anything slower would push our total response time past the limit, like where users is going to start noticing.
18:43So, for now, the Nemotron-3 Ultra at 529 P50, we observed GPT-4.1 at 536, again P50.
18:55But what matters more in voice than maybe anywhere else is that the P95 tail.
19:02Because GPT-4.1 was great at P50, but it spikes to 1.7 at P95.
19:11And it was even worse for Claude 3 that it hit over 4 seconds and because in a conversation you cannot average this out because one slow response and your entire flow is gone.
19:23And there's one more dimension that people often miss, which is the multi-turn trip because after 15 or 20 turn, sometime model starts ignoring parts of the system prompt.
19:35They might get to verbose, they go off spirit.
19:39Because in that's fatal, you can't just dump a wall of text on someone if the instruction following degrades across turn.
19:49Then you have to do some sort of a context pruning or session resets.
19:55So a few production issues that we've seen that false interruptions significantly increase your escalation rate.
20:06Because when agents cut people off incorrectly, then users are more likely to request for a human in the loop uh support.
20:17And a 755 ms that's still I think the best measured voice-to-voice for a cascaded pipeline like the ones which we propose.
20:26We're nowhere near the human speed of interaction, which means the return taking also needs to do a lot of heavy lifting to make the experience feel better than the raw numbers it would suggest.
20:39And at 58.9% recall on smart turn, that's actually the best deployable turn detection you can get today and because although it still leaves like four out of 10 turns
20:53to be filled by a uh VAD timer, but this space is going to improve a lot over the next year few years and uh we'll have better model.
21:05And on infrastructure running all of this in production is also hard because five systems that all scale differently and can also fail differently.
21:16You can explore Picovoice Cloud for managed hosting where your focus is on agent logic or the AWS guidance reference architecture for enterprise deployments with all the enterprise guardrails.
21:30So yeah, like uh Uh that's it from me.
21:35Everything uh we've discussed or mentioned is linked on the screen that I pulled benchmark.
21:40Okay, over to you, Dan.
21:46Thanks, Jinson.
21:46Hi, everyone.
21:47Uh my name is Daniel Weijia.
21:49I'm also a solutions architect with the AWS startups team.
21:53So I think um Jinson walked you through a great overview of some of the key challenges when building voice agents.
21:59So what I'm going to do in this session is just to walk you through a hands-on demo um just to make the concepts a bit more concrete.
22:07Um hopefully um you can kind of um incorporate some of the learnings into your own voice agent pipelines as well.
22:16So what we'll be going through is just this Picovoice Cat 10 detection demo, which is just a simple repository that we've created.
22:24Um we'll talk through three examples.
22:27So one with um silence detection, another with um built-in 10 detection in the speech-to-text model.
22:35And then finally, um we'll go through the open smart 10 model as well.
22:40So this is really useful when you either want full control of your um voice pipeline just for from a compliance perspective or if you want to also fine-tune and customize your own 10 detection model based your own based on your own data.
22:58So yeah, let's get started.
23:00So let's just go through the source code of the first example.
23:05As you can see here, um using Silero is pretty easy in Pipe Cat.
23:10All you have to do is import the relevant libraries, and you can configure the parameters as well.
23:17Um I haven't actually uh configured any of the parameters, so these are just the default, but you can change the parameters like the confidence to trigger, the silence duration before the turn ends,
23:30and the minimum audio volume as well.
23:34And in terms of the voice voice pipeline, it's pretty standard.
23:37So, we've used um an STT model, the LLM, and also a TTS model here.
23:43We've just defined it as the Cartesia model for STT.
23:47Same with the Sonic model for TTS, and as you can see, we've used the voice that sounds like a a British lady.
23:54And for the LLM in the voice agent's brain, we've just used the Anthropic Claude like cool model.
24:00So, we'll see um what that looks like, and I'll spin up this example right now.
24:07So, So, if you haven't used Pipe Cat before, one of the nice things is that it can spin up a local prototyping environment.
24:17Really useful for playing around with your voice agent pipeline.
24:21So, let me just see if this now works.
24:24Okay.
24:24So, as you can see, it will come up with a um local uh development environment.
24:32So, what I might do is just connect to the voice agent, but before I do that, I might just mute just in case um I'm interrupting the voice agent.
24:42It should come up with like an opening greeting, and then I'll test it with some uh interactions to see how the turn detection is working.
25:01Hey there, I'm your travel assistant and I'm excited to help you plan an amazing trip.
25:05So, where are you thinking of going?
25:07Anywhere in the world you've got your eye on?
25:09As you can see, the assistant's gone ahead to do the opening greeting and on the right here in the terminal, I've just got um some real-time logs just so we can see what's happening.
25:23So, what I might do now is just I'm going to pretend that I'm uh thinking about my travel destination and see how the voice agent behaves.
25:33So, I'm thinking of going to um
25:41I'm all ears.
25:42Where are you thinking of heading?
25:44As you can see, I haven't really finished my uh sentence yet, but the voice agent is already responding.
25:52So, you can see how this works in the debug logs and as you can see, there's an end of turn complete here and the reason for that is due to the stop seconds
26:04parameter, which is um set to um 300 milliseconds, it looks like here.
26:10So, that's why it's gone ahead and ended the turn.
26:15Um what I might do is I might just exit this local server and try the second example.
26:24So, let's just clear that and I'll refresh that.
26:27Okay, so I'm going to try the second example here.
26:32And in this example, we'll use a um speech-to-text model which has turn detection.
26:39So, we'll see how the voice agent behaves.
26:43Okay.
26:46I'm going to try uh the same example, so let me just mute myself here.
26:56Hey there.
26:57I'm your travel assistant and I'm excited to help you plan an amazing trip.
27:01So, where are you thinking of heading?
27:03Are you dreaming of a beach getaway, a city adventure, mountains, or maybe somewhere completely different?
27:11All right, let's see how the model behaves this time.
27:16Hi, I'm thinking of going to um As you can see, the model is pretty smart.
27:31It now can detect the turn, so let's see if we can find out what's going on in the debug logs.
27:37So, um let's see if it emits uh some of those turn taking events here.
27:56So, as you can see, um the Katisha Inc. model does emit some turn events here.
28:10And as you can see, it hasn't really completed the turn.
28:13It hasn't sort of um responded back with any response.
28:19But, maybe I can unmute myself and then resume and get to the destination.
28:25So, Sydney?
28:31Oh, Sydney is fantastic.
28:33You're going to have such a great time there.
28:35Are you thinking about when you'd like to go?
28:37And how long are you planning to stay?
28:38That'll help me figure out the best flights and hotels for you.
28:42So, as you can see, once the model detects that it's a full complete sentence, is able to then respond.
28:49Now, in practice, you would also incorporate um some silence detection there because as you can see, I was quite silent.
28:56Um I was muted for a while and you'd probably want the voice agent to respond in practice.
29:02Um having like voice uh activity detection or silence detection is also useful for um interrupting the voice models um um or what you call barge in as well.
29:14So, definitely this is just a a quick demo, but in practice, you would probably combine both the turn detection model and the silence detection.
29:22Um just for the final demo, I might quickly show the smart turn model as well and see what that looks like.
29:30Um it will behave very similar to to this.
29:33It's just that um the model will be running on your uh either your local machine or um on a GPU uh environment as well if you want to deploy that into your into the cloud.
29:47So, let me just maybe clear this just to avoid confusion.
29:54And then I'll use um I'll refresh this as well.
29:57And I'll start now the smart turn example.
30:01I'll refresh this to see if we can Okay, there you go.
30:05It's starting it.
30:06And I'm just going to do the same example.
30:08So, I'll mute myself again.
30:18Hey there.
30:19I'm your travel assistant and I'm excited to help you plan an amazing trip.
30:22So, where are you thinking about going?
30:25Are you dreaming of a beach getaway, exploring a new city, or maybe something else entirely?
30:32I'm thinking of going to um
30:38I'm all ears.
30:39Where are you thinking of going?
30:41Now, let's take a look at the um both the silence detection, which is incorporated into this example, and also the turn detection model.
30:50Hopefully, there are some um debug logs that we can see.
30:55As you can see, um it's not detecting that I've completed the turn.
30:59The end of turn state is incomplete, but it did complete it um due to the silence that I've um I've um incurred there.
31:10Um so, that there was some silence, and therefore it sort of kept going with the with the um next turn.
31:18What I might do is see if I can do a complete sentence and see if, you know, um it can be detected through the local smart turn model.
31:30I'm actually thinking of um going to Sydney.
31:36Oh, Sydney's fantastic.
31:38I'd love to help you plan that trip.
31:39So, tell me, what time of year are you thinking about going, and how long are you planning to stay?
31:44Also, are you traveling solo, with a partner, or with family?
31:50Okay, fantastic.
31:50So, the voice agent was able to respond as soon as I uh completed the sentence, but let's see if that was actually intentional, or was that um something that
32:03um you know, can be detected through the smart turn model.
32:06So, let's have a look here to see if we can find the smart turn um debug logs.
32:21As you can see, this line here, I'm sorry about the legibility of the highlight, but effectively this here says that um the end of turn state in is complete,
32:32and the smart turn model is actually classifying that there is a high probability that that um the sentence is complete and therefore it's okay to go to the next turn.
32:43So that's in a nutshell how turn detection works.
32:48Hope that was helpful.