WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=hMlLw1LeIK8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/hMlLw1LeIK8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:03.440 --> 00:00:04.520
Hey everyone.

s2
00:00:04.520 --> 00:00:07.600
I'm Chintan and I have my colleague Daniel with me.

s3
00:00:07.600 --> 00:00:11.200
We are solutions architect on the AWS APJ startup team.

s4
00:00:11.200 --> 00:00:15.640
So today we are going to talk about something we've been working on for a while.

s5
00:00:15.640 --> 00:00:18.520
Which is turn taking in voice agents.

s6
00:00:18.520 --> 00:00:25.480
What I mean by that is how does the system actually know that you have finished speaking and it's time for the agent to respond?

s7
00:00:25.480 --> 00:00:27.720
These are all audio engineering problems.

s8
00:00:27.720 --> 00:00:38.800
They are not LLM problems because you can have the perfect model, perfect track but the experience still might feel broken if the turn taking is off.

s9
00:00:38.800 --> 00:00:43.880
So we are going to walk you through the concepts and three different approaches to solving this.

s10
00:00:43.880 --> 00:00:51.040
And Daniel is going to you know make it more concrete by running all the three demo for you.

s11
00:00:56.240 --> 00:01:06.160
So to kick things off uh we'll start with the 200 millisecond constraint which is basically the physics of why voice is hard.

s12
00:01:06.160 --> 00:01:12.320
Then we look into the pipeline architecture, what are the components, where turn taking actually lives.

s13
00:01:12.320 --> 00:01:15.800
We'll then look at three level of solving this problem.

s14
00:01:15.800 --> 00:01:21.480
Going from simple silence detection all the way up to running your own turn detection model.

s15
00:01:21.480 --> 00:01:29.160
And then we'll also talk about the latency, the budget and production issues that we've seen.

s16
00:01:30.040 --> 00:01:37.920
So but before any of that, I just want to show you the problem statement more clearly because you have the same user.

s17
00:01:37.920 --> 00:01:51.880
You have the same sentence on the both side but the user outcome is very different because on the left when the user says I want to fly and midway they try to correct themselves,

s18
00:01:51.880 --> 00:01:54.520
the agent just does not notice.

s19
00:01:54.520 --> 00:02:00.840
Because it keeps going on for almost 2 seconds while the person is sitting there trying to get a word in.

s20
00:02:00.840 --> 00:02:09.280
On the right side, the same exact interaction is happening, but the agent is able to capture the interruption in under 200 milliseconds.

s21
00:02:09.280 --> 00:02:13.360
And immediately backs off so the user can now speak.

s22
00:02:13.360 --> 00:02:15.640
Both the scenarios, the LLM was identical.

s23
00:02:15.640 --> 00:02:16.680
It was the same model.

s24
00:02:16.680 --> 00:02:21.600
It was the same prompt, but the difference in the user experience is so different.

s25
00:02:21.600 --> 00:02:25.840
And the difference is purely because of the audio pipeline.

s26
00:02:25.920 --> 00:02:31.880
And how fast does it notices like someone else is talking and it knows when to shut up.

s27
00:02:31.880 --> 00:02:36.280
Like that's the problem we're trying to solve today.

s28
00:02:37.600 --> 00:02:47.400
So, we'll cover this 200 millisecond constraint because 200 milliseconds is how fast humans switch turns with each other in a conversation.

s29
00:02:47.400 --> 00:02:53.440
And the implications are pretty brutal because at 800 milliseconds, things start to feel off.

s30
00:02:53.440 --> 00:03:05.040
While at 1.5 seconds, your user just hang up on you because you have like your HR chat agents might get 5 seconds to respond and nobody will care.

s31
00:03:05.040 --> 00:03:08.880
But with the voice agents, you don't get that luxury.

s32
00:03:08.880 --> 00:03:15.320
We've seen recently like Salesforce was able to pivot the pipeline and they published their results in March 26th.

s33
00:03:15.320 --> 00:03:19.480
But even there, best measured response time was 755 milliseconds.

s34
00:03:19.480 --> 00:03:25.760
So, that's like almost 4x slower than how humans naturally take turns when talking.

s35
00:03:26.120 --> 00:03:34.400
So, the question here becomes how do we close this gap and where we cannot close it like at least how do we get the turn taking right?

s36
00:03:34.400 --> 00:03:40.840
So, the user experience does not feel as bad as the raw numbers might suggest here.

s37
00:03:40.840 --> 00:03:48.920
So, if you're working with a budget of 755 milliseconds and every millisecond count, you also need to understand what's actually in the pipeline.

s38
00:03:48.920 --> 00:03:54.000
Like where does the time go and where does this turn taking fit?

s39
00:03:54.000 --> 00:04:01.360
So, the pipeline itself you probably might know the main components, the STT, the LLM, the TTS.

s40
00:04:01.360 --> 00:04:04.840
You have the audio in, text out, text in, audio out.

s41
00:04:04.840 --> 00:04:13.680
But the piece that's often missing like from people while people are thinking about this is this voice activity detection.

s42
00:04:13.680 --> 00:04:15.520
Commonly called as VAD.

s43
00:04:15.520 --> 00:04:21.840
It's a tiny component that sits right at the front and it is the one that controls turn taking.

s44
00:04:21.840 --> 00:04:26.320
Its job is to basically detect whether the user has stopped talking or not.

s45
00:04:26.320 --> 00:04:33.120
And the two other thing that you set alongside main pipeline are also very critical for turn taking.

s46
00:04:33.120 --> 00:04:37.280
So, you have the smart turn detection which watches the VAD signal.

s47
00:04:37.280 --> 00:04:47.160
Plus the audio features and then it has to make that decision that whether should it respond now or should it wait because the user might still be talking.

s48
00:04:47.160 --> 00:04:51.320
And the interruption handler is triggered when someone barges in.

s49
00:04:51.320 --> 00:05:01.640
So, what it'll do is it'll propagate a flush of downstream like the TTS would stop and the LLM generation need to be canceled so that the pipeline is ready for the new input

s50
00:05:01.640 --> 00:05:15.080
within about, you know, 50 ms. So, if you remember I was talking about like how there are different levels to, you know, building this turn taking in voice agents.

s51
00:05:15.080 --> 00:05:17.640
So, we'll start with a level one.

s52
00:05:17.640 --> 00:05:19.400
Level one is your Silero VAD.

s53
00:05:19.400 --> 00:05:30.000
So, this is a simple component which you fully own and like a lot of production system even today we see are running agents just by using this.

s54
00:05:30.000 --> 00:05:37.200
So, coming to Silero VAD, it's a small 300,000 parameter model.

s55
00:05:37.200 --> 00:05:40.400
It takes in like a short term Fourier transform.

s56
00:05:40.400 --> 00:05:46.040
It has a and it takes raw audio and it'll convert it into a spectral features.

s57
00:05:46.040 --> 00:05:57.560
It has four convolutional layers to pick up the pattern and an LSTM that gives it memory so across the frame so it's just not looking at one chunk in isolation.

s58
00:05:57.560 --> 00:06:05.200
And then it has a sigmoid that gives a probability of speech and it's like a very small 2 megabyte model.

s59
00:06:05.640 --> 00:06:12.320
And the one parameter that minimum silence millisecond is basically the entire user experience of level one.

s60
00:06:12.320 --> 00:06:19.280
If you keep it very low, the agent is going to be snapping like it will cut people off while they're still, you know, thinking.

s61
00:06:19.280 --> 00:06:32.160
But if you keep it very high, the agent might become super patient and it will never interrupt but you might get to uh get like a dead air where people will wonder if the agent is still connected or not.

s62
00:06:32.160 --> 00:06:34.320
So there's like no universally right answer.

s63
00:06:34.320 --> 00:06:35.880
It depends on the domain.

s64
00:06:35.880 --> 00:06:44.200
If you're working on it from a perspective of a sales agent, you probably want it to be like 200 milliseconds.

s65
00:06:44.200 --> 00:06:53.600
If you're in a domain where you need some to give some time to your user to respond, it could be like 1 uh thousand to 1,200 milliseconds.

s66
00:06:53.600 --> 00:06:57.360
It depends entirely on what you're building.

s67
00:06:57.360 --> 00:07:10.320
So VAD works well like that's why a lot of people are using it and it's generally good at what it does that is to return fast response to the basic question of whether there's someone talking right now.

s68
00:07:10.320 --> 00:07:22.240
But uh there are things like it was never designed to handle because there'll be situations where just knowing that is there a silence will not be enough because you also need to answer these questions

s69
00:07:22.240 --> 00:07:26.160
of how long to wait like uh the silence duration.

s70
00:07:26.160 --> 00:07:33.360
If someone is pausing for 300 milliseconds or 400 milliseconds, VAD is seeing the same exact thing.

s71
00:07:33.360 --> 00:07:41.640
It has no idea like whether the person is catching up their breath or whether they've completely finished their thought.

s72
00:07:41.640 --> 00:07:45.240
So like they cannot distinguish that.

s73
00:07:45.240 --> 00:07:52.080
And the second scenario is like, what if they, you know, the user talks over the agent, like, it barges in.

s74
00:07:52.080 --> 00:07:54.120
It does not know how to handle it.

s75
00:07:54.120 --> 00:07:59.160
And the same silence can have a different intent.

s76
00:07:59.160 --> 00:08:01.680
And it can be like completely different things.

s77
00:08:01.680 --> 00:08:07.880
It could be like a completed sentence, it would be an incomplete thought, it could be like a thinking pause.

s78
00:08:07.880 --> 00:08:12.720
Or maybe it could be something as basic as like a backchannel acknowledgement.

s79
00:08:12.720 --> 00:08:27.080
Uh the situation of the silence is same, but the intent again was very different in all the scenarios, but VAD considers all four equally, and it literally cannot tell them apart.

s80
00:08:27.240 --> 00:08:30.160
So, those are the limitation of level one.

s81
00:08:30.160 --> 00:08:39.159
And with the level two, basically what you get is that you let the STT service tell you like when the turn is over.

s82
00:08:39.240 --> 00:08:45.520
So, Cartesiant does turn detection right inside their STT websocket.

s83
00:08:45.520 --> 00:08:57.000
So, when they stream your audio in, the server would handle both the transcription and the turn detection together, and it will emit event when it thinks like the turn is done.

s84
00:08:57.000 --> 00:09:03.120
Similarly, Deepgram does the same thing with their what they call as their I think at the endpointing.

s85
00:09:03.120 --> 00:09:12.760
And their P50 latency for Cartesiant is about 300, and it's about 250 for Deepgram Nova 3. Both of these are also working really well.

s86
00:09:12.760 --> 00:09:16.240
Like, we see a lot of customers use that.

s87
00:09:16.240 --> 00:09:22.080
Because they are using like full audio signals plus some linguistic context to make these decisions.

s88
00:09:22.080 --> 00:09:27.320
So, which is like much more information than your VAD will ever have.

s89
00:09:27.320 --> 00:09:29.680
However, the trade-off is with the transparency.

s90
00:09:29.680 --> 00:09:32.760
Because when it is working, it works great.

s91
00:09:32.760 --> 00:09:41.480
But when it misfires, or when it cuts someone off at the wrong moment, then you like have no way to figure out because there's no law that will say

s92
00:09:41.480 --> 00:09:47.520
uh it was done because of this reason, because of the what was what it saw.

s93
00:09:47.520 --> 00:09:54.040
And that decision was made inside someone else's server, and you just got to live with that, basically.

s94
00:09:55.320 --> 00:09:57.600
So, that brings us to level three.

s95
00:09:57.600 --> 00:10:05.080
So, in level three, you keeps a little VAD running locally for basic signal that is there a speech.

s96
00:10:05.080 --> 00:10:11.680
And you add Smart Turn on top of it, which is a small model that runs during silence.

s97
00:10:12.000 --> 00:10:19.720
So, Smart Turn V3.2 is the latest model, like, and uh let me share some numbers with you.

s98
00:10:19.720 --> 00:10:26.800
And it has about 58.9% recall and 68.4 percent precision.

s99
00:10:26.800 --> 00:10:34.160
So, what this means is that about six out of 10 time when someone finishes a sentence, Smart Turn will be able to catch it quickly.

s100
00:10:34.160 --> 00:10:44.680
But the other four times, it might not be confident enough, but that's also fine because you still have your VAD timer, which is running underneath as a safety net.

s101
00:10:44.680 --> 00:10:50.839
If the Smart Turn doesn't fire, your stop second will kick in, like, at the configured latency anyway.

s102
00:10:50.839 --> 00:10:56.200
So, it could be like 300 ms. So, in that scenario, you are never stuck waiting.

s103
00:10:56.200 --> 00:10:59.800
You just, like, get the fast respond when the model is confident.

s104
00:10:59.800 --> 00:11:05.080
And you get a slightly slower but a safer respond when it's not.

s105
00:11:05.080 --> 00:11:12.720
The other number on this list is from, like, a Meta's published paper earlier this year, um around March.

s106
00:11:12.720 --> 00:11:21.880
There, they had reported a higher recall at 87.7%, but they haven't released the code, so you really cannot deploy it.

s107
00:11:22.480 --> 00:11:26.960
Smart Turn is BSD-2 licensed, and it's a 8-MB small model.

s108
00:11:26.960 --> 00:11:29.320
You can pip install it today as well.

s109
00:11:29.320 --> 00:11:33.360
So, yeah, like, that works out better for a lot of our customers.

s110
00:11:33.360 --> 00:11:35.760
We decide when to start talking.

s111
00:11:35.760 --> 00:11:37.280
But like, what about the other direction?

s112
00:11:37.280 --> 00:11:44.520
What happens when the user interrupts and while you are already, you know, the agent is talking?

s113
00:11:44.520 --> 00:11:47.320
So, on the left side, you can see what happens mechanically.

s114
00:11:47.320 --> 00:11:57.040
Like, user opens you mouth, the windy will pick it up in 32 milliseconds and then then like 15 milliseconds, the entire pipeline is flushed.

s115
00:11:57.040 --> 00:12:00.080
TTS stops, LLM cancels, everything is clean.

s116
00:12:00.080 --> 00:12:02.720
Like, the user never hears the agent.

s117
00:12:02.720 --> 00:12:06.840
You know, that part is taken care by Pipe Cat.

s118
00:12:06.920 --> 00:12:14.640
But, the important thing is like, should it have stopped there on because think about a normal conversation when someone says, "Yeah."

s119
00:12:14.640 --> 00:12:20.200
Like, if they send an acknowledgement while you're speaking, you don't stop, right?

s120
00:12:20.200 --> 00:12:25.880
You know, they're just agreeing, but if you they say like, "Okay, wait, no. Like, that was wrong."

s121
00:12:25.880 --> 00:12:27.720
You have to stop immediately.

s122
00:12:27.720 --> 00:12:28.360
Same thing here.

s123
00:12:28.360 --> 00:12:31.160
Like, the red is the it's a correction.

s124
00:12:31.160 --> 00:12:32.760
Stop on everything.

s125
00:12:32.760 --> 00:12:36.360
They amber could be just some sort of a filler background noise.

s126
00:12:36.360 --> 00:12:41.280
But, you want to go ahead and keep going, maybe in a bit quieter.

s127
00:12:41.360 --> 00:12:45.040
Blue means like a signal for cough for a background noise.

s128
00:12:45.040 --> 00:12:48.720
So, you can just ignore it and finish your sentence.

s129
00:12:48.720 --> 00:12:59.120
So, at level three because you own this classification, you can start distinguishing between a real interruption and someone just, you know, going um.

s130
00:12:59.120 --> 00:13:05.839
So, today most of these systems stop every time, but this is a piece that's improving.

s131
00:13:05.839 --> 00:13:10.839
Like, we are seeing a lot of incremental improvement in this.

s132
00:13:17.120 --> 00:13:25.160
So, by now that you've seen all three levels and if you look at these three Python files, pipeline is almost identical in all of them.

s133
00:13:25.160 --> 00:13:28.640
The only thing that differs is how you answer all of these questions.

s134
00:13:28.640 --> 00:13:31.280
When is the user done talking?

s135
00:13:31.280 --> 00:13:41.520
In file one, you pass in Celero VAD analyzer and that says the user had been silent for 300 milliseconds, so you can assume they're done.

s136
00:13:41.520 --> 00:13:50.880
In file two, you swap it with the Cadence turns STT service class and now the service is going to tell you when the turn is over.

s137
00:13:50.880 --> 00:14:06.280
And in file three, you keep your Celero VAD analyzer, but you also add like a local smart turn analyzer version three, which is sort of a small model that runs during your silence and it's going to look at the prosody

s138
00:14:07.480 --> 00:14:15.920
and it's going to look at prosody and intonation to decide whether that pause meant I'm done or I'm still thinking.

s139
00:14:15.920 --> 00:14:25.080
So, you have the code-wise the same pipeline, but the configuration changes, which you know, leads to completely different behavior.

s140
00:14:26.040 --> 00:14:34.560
So, to summarize your three levels, where our level one was Celero VAD, so you basically own the silence detection completely.

s141
00:14:34.560 --> 00:14:38.280
Level two, you let the STT provider handle it for you.

s142
00:14:38.280 --> 00:14:46.240
And in most cases, it it will speak smarter, but then you cannot see what happened inside it.

s143
00:14:46.240 --> 00:14:54.120
Level three is your VAD plus smart turn, so where we basically own everything, like you have full portability.

s144
00:14:54.960 --> 00:15:08.280
And now coming to the latency piece, because we've been throwing around these numbers, 755 milliseconds, 1.3 seconds, but let me show you where all the time actually goes in.

s145
00:15:08.720 --> 00:15:15.640
So, this is a breakdown from Quintela, who was a co-founder of Daily.co and is the person who created Pipe Band.

s146
00:15:15.640 --> 00:15:19.400
And these are some of the numbers based on their production measurement.

s147
00:15:19.400 --> 00:15:27.280
So, you have the numbers for making an which is about 40 ms. Then you have network and jitter buffer of 52.

s148
00:15:27.280 --> 00:15:28.720
So, these are physics.

s149
00:15:28.720 --> 00:15:30.520
Like you can't really change them much.

s150
00:15:30.520 --> 00:15:34.840
Then you have your transcription plus end pointing.

s151
00:15:34.840 --> 00:15:40.520
That's about 300 ms. Again, that's your STT doing its thing.

s152
00:15:40.840 --> 00:15:56.840
LLM time to first byte, and this is generally the dominant bottleneck because in a typical API setup, you're looking at 500 to 650 ms depending on what model you're calling and from which cloud provider, which region.

s153
00:15:56.840 --> 00:16:02.840
And then you have your network out and playback TTS.

s154
00:16:02.840 --> 00:16:15.960
That's it's going to take about 120 to 90 85 ms. So, you're totally looking at roughly 1,100 to 1,300 ms in a standard setup calling cloud APIs.

s155
00:16:15.960 --> 00:16:25.680
Now, the Quindos team had demonstrated about 500 ms total voice-to-voice by co-locating all models in the same GPU cluster.

s156
00:16:25.680 --> 00:16:30.160
Because they were able to eliminate the network hops within the service.

s157
00:16:30.160 --> 00:16:41.200
And it's sort of an achievable floor because if you're willing to invest in infrastructure, but for most of the developers today calling APIs, you're somewhere in the range of 800 to 1,300

s158
00:16:41.200 --> 00:16:52.360
ms. And the key insight to note here is that the STT, LLM, together eat about 2/3 of the is latency budget.

s159
00:16:52.360 --> 00:17:02.920
So, these are the really the only two levers that we have to move this latency metric needle in a meaningful manner.

s160
00:17:03.560 --> 00:17:08.199
So, coming to like which LLMs are actually fast enough for this?

s161
00:17:08.199 --> 00:17:17.079
We benchmark current models specifically for voice, and the target we kept it as under 700 ms time to first token.

s162
00:17:17.079 --> 00:17:23.800
Because anything slower will push our total response past where the user is going to start noticing.

s163
00:17:23.800 --> 00:17:36.320
So, for now, like recording in June 2026, what we've observed Nemotron-3 Ultra gives us a 529 P50 latency.

s164
00:17:36.320 --> 00:17:39.640
GPT-4.1 at about 536.

s165
00:17:39.640 --> 00:17:45.880
But also, like what matters more in voice uh than P95.

s166
00:17:56.000 --> 00:17:59.720
So, which LLMs are actually fast enough for this?

s167
00:17:59.720 --> 00:18:24.320
We benchmark current models, specially uh for voice, and So, coming to the So, coming to the LLMs models, which are actually fast enough for the We benchmark current models

s168
00:18:24.320 --> 00:18:25.600
well for voice.

s169
00:18:25.600 --> 00:18:43.000
Uh so, these benchmarks are from June 26, and the target that we kept was under 700 ms for time to first open, because anything slower would push our total response time past the limit, like where users is going to start noticing.

s170
00:18:43.000 --> 00:18:55.520
So, for now, the Nemotron-3 Ultra at 529 P50, we observed GPT-4.1 at 536, again P50.

s171
00:18:55.760 --> 00:19:02.480
But what matters more in voice than maybe anywhere else is that the P95 tail.

s172
00:19:02.480 --> 00:19:11.040
Because GPT-4.1 was great at P50, but it spikes to 1.7 at P95.

s173
00:19:11.040 --> 00:19:23.600
And it was even worse for Claude 3 that it hit over 4 seconds and because in a conversation you cannot average this out because one slow response and your entire flow is gone.

s174
00:19:23.600 --> 00:19:35.440
And there's one more dimension that people often miss, which is the multi-turn trip because after 15 or 20 turn, sometime model starts ignoring parts of the system prompt.

s175
00:19:35.440 --> 00:19:39.120
They might get to verbose, they go off spirit.

s176
00:19:39.120 --> 00:19:49.040
Because in that's fatal, you can't just dump a wall of text on someone if the instruction following degrades across turn.

s177
00:19:49.040 --> 00:19:54.000
Then you have to do some sort of a context pruning or session resets.

s178
00:19:55.960 --> 00:20:06.040
So a few production issues that we've seen that false interruptions significantly increase your escalation rate.

s179
00:20:06.040 --> 00:20:16.040
Because when agents cut people off incorrectly, then users are more likely to request for a human in the loop uh support.

s180
00:20:17.440 --> 00:20:26.600
And a 755 ms that's still I think the best measured voice-to-voice for a cascaded pipeline like the ones which we propose.

s181
00:20:26.600 --> 00:20:39.880
We're nowhere near the human speed of interaction, which means the return taking also needs to do a lot of heavy lifting to make the experience feel better than the raw numbers it would suggest.

s182
00:20:39.880 --> 00:20:53.120
And at 58.9% recall on smart turn, that's actually the best deployable turn detection you can get today and because although it still leaves like four out of 10 turns

s183
00:20:53.120 --> 00:21:05.440
to be filled by a uh VAD timer, but this space is going to improve a lot over the next year few years and uh we'll have better model.

s184
00:21:05.440 --> 00:21:16.640
And on infrastructure running all of this in production is also hard because five systems that all scale differently and can also fail differently.

s185
00:21:16.640 --> 00:21:30.160
You can explore Picovoice Cloud for managed hosting where your focus is on agent logic or the AWS guidance reference architecture for enterprise deployments with all the enterprise guardrails.

s186
00:21:30.680 --> 00:21:35.200
So yeah, like uh Uh that's it from me.

s187
00:21:35.200 --> 00:21:40.520
Everything uh we've discussed or mentioned is linked on the screen that I pulled benchmark.

s188
00:21:40.520 --> 00:21:43.520
Okay, over to you, Dan.

s189
00:21:46.080 --> 00:21:46.800
Thanks, Jinson.

s190
00:21:46.800 --> 00:21:47.800
Hi, everyone.

s191
00:21:47.800 --> 00:21:49.080
Uh my name is Daniel Weijia.

s192
00:21:49.080 --> 00:21:53.360
I'm also a solutions architect with the AWS startups team.

s193
00:21:53.360 --> 00:21:59.840
So I think um Jinson walked you through a great overview of some of the key challenges when building voice agents.

s194
00:21:59.840 --> 00:22:07.800
So what I'm going to do in this session is just to walk you through a hands-on demo um just to make the concepts a bit more concrete.

s195
00:22:07.800 --> 00:22:16.160
Um hopefully um you can kind of um incorporate some of the learnings into your own voice agent pipelines as well.

s196
00:22:16.160 --> 00:22:24.760
So what we'll be going through is just this Picovoice Cat 10 detection demo, which is just a simple repository that we've created.

s197
00:22:24.760 --> 00:22:27.160
Um we'll talk through three examples.

s198
00:22:27.160 --> 00:22:35.920
So one with um silence detection, another with um built-in 10 detection in the speech-to-text model.

s199
00:22:35.920 --> 00:22:40.840
And then finally, um we'll go through the open smart 10 model as well.

s200
00:22:40.840 --> 00:22:58.000
So this is really useful when you either want full control of your um voice pipeline just for from a compliance perspective or if you want to also fine-tune and customize your own 10 detection model based your own based on your own data.

s201
00:22:58.000 --> 00:23:00.000
So yeah, let's get started.

s202
00:23:00.000 --> 00:23:05.680
So let's just go through the source code of the first example.

s203
00:23:05.680 --> 00:23:10.000
As you can see here, um using Silero is pretty easy in Pipe Cat.

s204
00:23:10.000 --> 00:23:17.680
All you have to do is import the relevant libraries, and you can configure the parameters as well.

s205
00:23:17.680 --> 00:23:30.960
Um I haven't actually uh configured any of the parameters, so these are just the default, but you can change the parameters like the confidence to trigger, the silence duration before the turn ends,

s206
00:23:30.960 --> 00:23:34.120
and the minimum audio volume as well.

s207
00:23:34.120 --> 00:23:37.320
And in terms of the voice voice pipeline, it's pretty standard.

s208
00:23:37.320 --> 00:23:43.240
So, we've used um an STT model, the LLM, and also a TTS model here.

s209
00:23:43.240 --> 00:23:47.600
We've just defined it as the Cartesia model for STT.

s210
00:23:47.600 --> 00:23:54.600
Same with the Sonic model for TTS, and as you can see, we've used the voice that sounds like a a British lady.

s211
00:23:54.600 --> 00:24:00.720
And for the LLM in the voice agent's brain, we've just used the Anthropic Claude like cool model.

s212
00:24:00.720 --> 00:24:07.080
So, we'll see um what that looks like, and I'll spin up this example right now.

s213
00:24:07.080 --> 00:24:17.360
So, So, if you haven't used Pipe Cat before, one of the nice things is that it can spin up a local prototyping environment.

s214
00:24:17.360 --> 00:24:21.080
Really useful for playing around with your voice agent pipeline.

s215
00:24:21.080 --> 00:24:24.280
So, let me just see if this now works.

s216
00:24:24.280 --> 00:24:24.680
Okay.

s217
00:24:24.680 --> 00:24:32.480
So, as you can see, it will come up with a um local uh development environment.

s218
00:24:32.480 --> 00:24:42.440
So, what I might do is just connect to the voice agent, but before I do that, I might just mute just in case um I'm interrupting the voice agent.

s219
00:24:42.440 --> 00:24:54.080
It should come up with like an opening greeting, and then I'll test it with some uh interactions to see how the turn detection is working.

s220
00:25:01.280 --> 00:25:05.680
Hey there, I'm your travel assistant and I'm excited to help you plan an amazing trip.

s221
00:25:05.680 --> 00:25:07.720
So, where are you thinking of going?

s222
00:25:07.720 --> 00:25:09.960
Anywhere in the world you've got your eye on?

s223
00:25:09.960 --> 00:25:23.080
As you can see, the assistant's gone ahead to do the opening greeting and on the right here in the terminal, I've just got um some real-time logs just so we can see what's happening.

s224
00:25:23.080 --> 00:25:32.640
So, what I might do now is just I'm going to pretend that I'm uh thinking about my travel destination and see how the voice agent behaves.

s225
00:25:33.120 --> 00:25:37.560
So, I'm thinking of going to um

s226
00:25:41.200 --> 00:25:42.280
I'm all ears.

s227
00:25:42.280 --> 00:25:44.640
Where are you thinking of heading?

s228
00:25:44.640 --> 00:25:52.600
As you can see, I haven't really finished my uh sentence yet, but the voice agent is already responding.

s229
00:25:52.600 --> 00:26:04.840
So, you can see how this works in the debug logs and as you can see, there's an end of turn complete here and the reason for that is due to the stop seconds

s230
00:26:04.840 --> 00:26:10.720
parameter, which is um set to um 300 milliseconds, it looks like here.

s231
00:26:10.720 --> 00:26:15.400
So, that's why it's gone ahead and ended the turn.

s232
00:26:15.400 --> 00:26:24.520
Um what I might do is I might just exit this local server and try the second example.

s233
00:26:24.520 --> 00:26:27.480
So, let's just clear that and I'll refresh that.

s234
00:26:27.480 --> 00:26:32.080
Okay, so I'm going to try the second example here.

s235
00:26:32.840 --> 00:26:39.760
And in this example, we'll use a um speech-to-text model which has turn detection.

s236
00:26:39.760 --> 00:26:43.120
So, we'll see how the voice agent behaves.

s237
00:26:43.120 --> 00:26:45.120
Okay.

s238
00:26:46.000 --> 00:26:51.680
I'm going to try uh the same example, so let me just mute myself here.

s239
00:26:56.920 --> 00:26:57.720
Hey there.

s240
00:26:57.720 --> 00:27:01.360
I'm your travel assistant and I'm excited to help you plan an amazing trip.

s241
00:27:01.360 --> 00:27:03.440
So, where are you thinking of heading?

s242
00:27:03.440 --> 00:27:10.320
Are you dreaming of a beach getaway, a city adventure, mountains, or maybe somewhere completely different?

s243
00:27:11.880 --> 00:27:15.840
All right, let's see how the model behaves this time.

s244
00:27:16.080 --> 00:27:31.120
Hi, I'm thinking of going to um As you can see, the model is pretty smart.

s245
00:27:31.120 --> 00:27:37.160
It now can detect the turn, so let's see if we can find out what's going on in the debug logs.

s246
00:27:37.160 --> 00:27:44.960
So, um let's see if it emits uh some of those turn taking events here.

s247
00:27:56.680 --> 00:28:03.600
So, as you can see, um the Katisha Inc. model does emit some turn events here.

s248
00:28:10.520 --> 00:28:13.160
And as you can see, it hasn't really completed the turn.

s249
00:28:13.160 --> 00:28:19.760
It hasn't sort of um responded back with any response.

s250
00:28:19.760 --> 00:28:25.480
But, maybe I can unmute myself and then resume and get to the destination.

s251
00:28:25.480 --> 00:28:29.560
So, Sydney?

s252
00:28:31.000 --> 00:28:33.200
Oh, Sydney is fantastic.

s253
00:28:33.200 --> 00:28:35.160
You're going to have such a great time there.

s254
00:28:35.160 --> 00:28:37.000
Are you thinking about when you'd like to go?

s255
00:28:37.000 --> 00:28:38.800
And how long are you planning to stay?

s256
00:28:38.800 --> 00:28:42.800
That'll help me figure out the best flights and hotels for you.

s257
00:28:42.800 --> 00:28:49.080
So, as you can see, once the model detects that it's a full complete sentence, is able to then respond.

s258
00:28:49.080 --> 00:28:56.800
Now, in practice, you would also incorporate um some silence detection there because as you can see, I was quite silent.

s259
00:28:56.800 --> 00:29:02.800
Um I was muted for a while and you'd probably want the voice agent to respond in practice.

s260
00:29:02.800 --> 00:29:14.120
Um having like voice uh activity detection or silence detection is also useful for um interrupting the voice models um um or what you call barge in as well.

s261
00:29:14.120 --> 00:29:22.720
So, definitely this is just a a quick demo, but in practice, you would probably combine both the turn detection model and the silence detection.

s262
00:29:22.720 --> 00:29:30.360
Um just for the final demo, I might quickly show the smart turn model as well and see what that looks like.

s263
00:29:30.360 --> 00:29:33.120
Um it will behave very similar to to this.

s264
00:29:33.120 --> 00:29:47.160
It's just that um the model will be running on your uh either your local machine or um on a GPU uh environment as well if you want to deploy that into your into the cloud.

s265
00:29:47.600 --> 00:29:52.960
So, let me just maybe clear this just to avoid confusion.

s266
00:29:54.280 --> 00:29:57.520
And then I'll use um I'll refresh this as well.

s267
00:29:57.520 --> 00:30:01.440
And I'll start now the smart turn example.

s268
00:30:01.440 --> 00:30:05.360
I'll refresh this to see if we can Okay, there you go.

s269
00:30:05.360 --> 00:30:06.320
It's starting it.

s270
00:30:06.320 --> 00:30:08.200
And I'm just going to do the same example.

s271
00:30:08.200 --> 00:30:11.080
So, I'll mute myself again.

s272
00:30:18.440 --> 00:30:19.240
Hey there.

s273
00:30:19.240 --> 00:30:22.800
I'm your travel assistant and I'm excited to help you plan an amazing trip.

s274
00:30:22.800 --> 00:30:25.080
So, where are you thinking about going?

s275
00:30:25.080 --> 00:30:31.080
Are you dreaming of a beach getaway, exploring a new city, or maybe something else entirely?

s276
00:30:32.960 --> 00:30:36.640
I'm thinking of going to um

s277
00:30:38.560 --> 00:30:39.520
I'm all ears.

s278
00:30:39.520 --> 00:30:41.800
Where are you thinking of going?

s279
00:30:41.800 --> 00:30:50.720
Now, let's take a look at the um both the silence detection, which is incorporated into this example, and also the turn detection model.

s280
00:30:50.720 --> 00:30:55.040
Hopefully, there are some um debug logs that we can see.

s281
00:30:55.040 --> 00:30:59.440
As you can see, um it's not detecting that I've completed the turn.

s282
00:30:59.440 --> 00:31:10.360
The end of turn state is incomplete, but it did complete it um due to the silence that I've um I've um incurred there.

s283
00:31:10.360 --> 00:31:18.680
Um so, that there was some silence, and therefore it sort of kept going with the with the um next turn.

s284
00:31:18.680 --> 00:31:29.040
What I might do is see if I can do a complete sentence and see if, you know, um it can be detected through the local smart turn model.

s285
00:31:30.720 --> 00:31:34.880
I'm actually thinking of um going to Sydney.

s286
00:31:36.480 --> 00:31:38.280
Oh, Sydney's fantastic.

s287
00:31:38.280 --> 00:31:39.920
I'd love to help you plan that trip.

s288
00:31:39.920 --> 00:31:44.800
So, tell me, what time of year are you thinking about going, and how long are you planning to stay?

s289
00:31:44.800 --> 00:31:50.080
Also, are you traveling solo, with a partner, or with family?

s290
00:31:50.080 --> 00:31:50.920
Okay, fantastic.

s291
00:31:50.920 --> 00:32:03.320
So, the voice agent was able to respond as soon as I uh completed the sentence, but let's see if that was actually intentional, or was that um something that

s292
00:32:03.320 --> 00:32:06.320
um you know, can be detected through the smart turn model.

s293
00:32:06.320 --> 00:32:13.760
So, let's have a look here to see if we can find the smart turn um debug logs.

s294
00:32:21.000 --> 00:32:32.040
As you can see, this line here, I'm sorry about the legibility of the highlight, but effectively this here says that um the end of turn state in is complete,

s295
00:32:32.040 --> 00:32:43.400
and the smart turn model is actually classifying that there is a high probability that that um the sentence is complete and therefore it's okay to go to the next turn.

s296
00:32:43.400 --> 00:32:48.640
So that's in a nutshell how turn detection works.

s297
00:32:48.640 --> 00:32:51.240
Hope that was helpful.