WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=OmyqelTwpOo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/OmyqelTwpOo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.880 --> 00:00:16.240
Um all right, we're going to talk about voice agents.

s3
00:00:16.240 --> 00:00:24.400
And most of the agents that we deal with on a day-to-day basis, um they they do things in the background, right?

s4
00:00:24.400 --> 00:00:38.320
They they write code for us, they do research, we give them something to do, a task, and we walk away, we come back, and we don't really care if they take an extra 10 seconds, an extra 10 minutes, maybe sometimes 10 hours,

s5
00:00:38.320 --> 00:00:39.080
right?

s6
00:00:39.080 --> 00:00:40.600
Um and we expect that.

s7
00:00:40.600 --> 00:00:48.800
But at some point in the last year or so, you have probably interacted with another kind of agent, a real-time agent.

s8
00:00:48.800 --> 00:00:53.080
Maybe when you called your pharmacy, maybe uh when you called the airline.

s9
00:00:53.080 --> 00:00:56.520
I talked to one just the other day when I called a car dealership.

s10
00:00:56.520 --> 00:01:01.760
And at some point in that conversation, you did the the most human-like thing possible.

s11
00:01:01.760 --> 00:01:03.600
You spoke over it.

s12
00:01:03.600 --> 00:01:04.839
You interrupted it.

s13
00:01:04.839 --> 00:01:06.040
You confused it.

s14
00:01:06.040 --> 00:01:09.320
And the entire conversation fell apart, it just didn't feel natural.

s15
00:01:09.320 --> 00:01:16.080
And then you did what I do every single time, operator, hit zero, like get me to a human, right?

s16
00:01:16.080 --> 00:01:17.120
So, how do we fix that?

s17
00:01:17.120 --> 00:01:21.800
How do we make these agents as human-like as possible?

s18
00:01:21.800 --> 00:01:35.560
Um and so, like in in a um a finance space, what if your caller was calling to schedule a payment, and they changed the the payment amount midway through, they changed the date,

s19
00:01:35.560 --> 00:01:38.640
um and and the agent didn't keep up with it, right?

s20
00:01:38.640 --> 00:01:42.600
That is where we start having issues, especially in the finance space.

s21
00:01:42.600 --> 00:01:45.160
The the the the risk is much higher, right?

s22
00:01:45.160 --> 00:01:47.760
So, how do we fix that?

s23
00:01:47.760 --> 00:01:53.440
So, we're going to talk about how can we build human-like voice agents for financial services.

s24
00:01:53.440 --> 00:01:59.160
And when you think about building a good voice agent, one of the first things that comes to mind is latency.

s25
00:01:59.160 --> 00:02:01.560
We need it to be as fast as possible, right?

s26
00:02:01.560 --> 00:02:03.280
Well, latency is part of the equation.

s27
00:02:03.280 --> 00:02:07.120
Latency has a finite budget, and we'll talk about what that is.

s28
00:02:07.120 --> 00:02:10.520
Um but the keyword here is human-like.

s29
00:02:10.520 --> 00:02:14.080
How do we make these voice agents as human-like as we can?

s30
00:02:14.080 --> 00:02:15.440
That's the goal.

s31
00:02:15.440 --> 00:02:19.640
So, there's quite a few people that are talking about voice lately.

s32
00:02:19.640 --> 00:02:21.360
Um some of these might look familiar.

s33
00:02:21.360 --> 00:02:23.880
We've got this this guy Sam.

s34
00:02:23.880 --> 00:02:26.880
Um he's excited about voice models.

s35
00:02:26.880 --> 00:02:33.040
He says uh that they're starting to change the way they that we interface with AI.

s36
00:02:33.040 --> 00:02:33.959
Isn't that true?

s37
00:02:33.959 --> 00:02:38.680
And then this this um weird name here in the bottom right, Swyx.

s38
00:02:38.680 --> 00:02:40.160
I don't know if you've heard of him.

s39
00:02:40.160 --> 00:02:44.640
Um he says oh uh that this is the OG modality.

s40
00:02:44.640 --> 00:02:51.400
Um the correct modality for aligning AGI to our intents is very good voice AI engineering.

s41
00:02:51.400 --> 00:02:54.120
That's that's that's a pretty bold statement there.

s42
00:02:54.120 --> 00:02:59.040
And then like get here says we learn to speak before we learn to read.

s43
00:02:59.040 --> 00:02:59.440
Right?

s44
00:02:59.440 --> 00:03:01.760
Voice is the most natural interface that we have.

s45
00:03:01.760 --> 00:03:06.440
So, how can we make it as natural as possible and human-like as possible?

s46
00:03:06.440 --> 00:03:12.360
Quick show of hands, who who here has ever built a voice agent and deployed it to production?

s47
00:03:12.360 --> 00:03:13.760
All right, I see a few hands.

s48
00:03:13.760 --> 00:03:19.120
All right, so some of you know the challenges that you might face when doing this.

s49
00:03:19.120 --> 00:03:25.440
Um others here who have not built one, maybe you have um seen a very good example of one.

s50
00:03:25.440 --> 00:03:31.400
Have you ever used ChatGPT's advanced voice mode to have a conversation with ChatGPT?

s51
00:03:31.400 --> 00:03:33.440
Okay, so that's that's cool.

s52
00:03:33.440 --> 00:03:41.360
Segue here that OpenAI actually chose LiveKit to power that that functionality there.

s53
00:03:41.360 --> 00:03:47.400
So, it's a very good example of a good use case or a good uh good example of a good voice agent.

s54
00:03:47.400 --> 00:03:55.800
Um LiveKit is the open-source real-time framework and infrastructure for these human-like agents.

s55
00:03:55.800 --> 00:04:00.920
Um agents that can hear, see, and speak to a person on the other end.

s56
00:04:00.920 --> 00:04:03.440
And so, we're not just talking about voice, actually.

s57
00:04:03.440 --> 00:04:11.280
Um there's there's multiple ways to interact with with agents, voice, video, data streams, uh even physical AI.

s58
00:04:11.280 --> 00:04:18.280
We have a robotics division as well, but voice is the entry point voice is is the most natural modality, and voice is where there's a lot of challenges.

s59
00:04:18.280 --> 00:04:21.280
So, we're going to focus just on voice today.

s60
00:04:21.280 --> 00:04:26.919
So, in a traditional voice uh voice agent, you have what's called a cascaded pipeline.

s61
00:04:26.919 --> 00:04:29.400
This is three models that work together.

s62
00:04:29.400 --> 00:04:33.760
You've got your speech-to-text model, your LLM, and your text-to-speech model.

s63
00:04:33.760 --> 00:04:36.480
And these all have to work in unison.

s64
00:04:36.480 --> 00:04:41.480
And there are also these other models, these real-time models, speech-to-speech models, right?

s65
00:04:41.480 --> 00:04:43.280
And they replace that pipeline.

s66
00:04:43.280 --> 00:04:44.880
But that's a newer concept.

s67
00:04:44.880 --> 00:04:49.760
Um it's it's getting better, but it's not quite there for every use case yet, right?

s68
00:04:49.760 --> 00:04:54.960
But even then, we've got the traditional pipeline with three models, we've got the speech-to-speech models.

s69
00:04:54.960 --> 00:04:58.040
There's so many things to choose from, so much complexity.

s70
00:04:58.040 --> 00:04:59.800
Where do we even get started?

s71
00:04:59.800 --> 00:05:01.800
And here's what we have to choose from.

s72
00:05:01.800 --> 00:05:09.000
Hundreds of models, dozens of providers, and quick show of hands here, who has ever picked a model based on a leaderboard?

s73
00:05:09.000 --> 00:05:13.040
Like you pick them whatever model's at the top of the leaderboard, that's what we're going to use, right?

s74
00:05:13.040 --> 00:05:16.640
Well, we're going to talk about why that does not work for real-time agents.

s75
00:05:16.640 --> 00:05:19.760
We can't just pick models off the top of a leaderboard.

s76
00:05:19.760 --> 00:05:25.880
Um because leaderboards, they score these models, these components, but we're shipping a conversation.

s77
00:05:25.880 --> 00:05:27.200
We need to be able to score them together.

s78
00:05:27.200 --> 00:05:29.160
So, we're going to talk about how we can do that.

s79
00:05:29.160 --> 00:05:30.320
So, let's go through a scenario here.

s80
00:05:30.320 --> 00:05:33.919
Let's let's suppose that a caller uh wants to pay their bill.

s81
00:05:33.919 --> 00:05:36.919
And they say, "I want to pay on the 3rd."

s82
00:05:36.919 --> 00:05:40.880
But the speech-to-text model hears the 30th.

s83
00:05:40.880 --> 00:05:45.360
And then the LLM is planning now for the 30th, and that goes into the system.

s84
00:05:45.360 --> 00:05:46.760
The LLM doesn't know any better.

s85
00:05:46.760 --> 00:05:49.080
It doesn't know that the speech detects model got it wrong.

s86
00:05:49.080 --> 00:05:53.000
And so all of those errors flow downhill from there.

s87
00:05:53.000 --> 00:05:56.720
And so the the the leaderboard, it can't show us this.

s88
00:05:56.720 --> 00:06:01.480
It can't judge this because it just was looking at it individual models, right?

s89
00:06:01.480 --> 00:06:07.400
And so also these leaderboards, they score these models based on clean audio.

s90
00:06:07.400 --> 00:06:13.320
But your callers, they're calling on the speakerphone in traffic with kids in the backseat, right?

s91
00:06:13.320 --> 00:06:20.480
So we need to be able to score our own agents based on real-world use cases and examples.

s92
00:06:20.720 --> 00:06:23.200
So let's let's say that we fall into this leaderboard trap.

s93
00:06:23.200 --> 00:06:24.880
We just pick a model.

s94
00:06:24.880 --> 00:06:29.880
Um then the the first question that people ask is always, you know, which model should I pick?

s95
00:06:29.880 --> 00:06:39.680
So they use a leaderboard, they pick a model, but that the actual question that we should be asking is which stack should I ship for my use case?

s96
00:06:39.680 --> 00:06:42.280
And so that is what we're going to figure out.

s97
00:06:42.280 --> 00:06:45.720
How do we answer that question?

s98
00:06:45.840 --> 00:06:53.680
So before we answer that question, let's say that you have picked some models and you start measuring.

s99
00:06:53.680 --> 00:06:55.800
How are these models performing?

s100
00:06:55.800 --> 00:06:59.080
Let's uh say you're you're testing your TTS provider.

s101
00:06:59.080 --> 00:07:04.160
And TTS provider on the dashboard, it's it's uh reporting time to first byte.

s102
00:07:04.160 --> 00:07:07.200
And that looks really fast, 210 milliseconds time to first byte.

s103
00:07:07.200 --> 00:07:08.360
That's super fast.

s104
00:07:08.360 --> 00:07:18.720
Um but we've actually tested lots of popular providers that are shipping up to 3/4 of a second of silence after that first byte.

s105
00:07:18.720 --> 00:07:21.960
So a byte does not equal audio.

s106
00:07:21.960 --> 00:07:26.120
So what we're actually looking for is time to first audio.

s107
00:07:26.120 --> 00:07:30.520
How much time does it take for our user to actually start hearing something?

s108
00:07:30.520 --> 00:07:34.960
So time to first audio is what we should be measuring, not time to first byte.

s109
00:07:34.960 --> 00:07:42.840
Um so imagine, you know, the payment went through and and and you you're asking, you know, um you know, did my payment go through?"

s110
00:07:42.840 --> 00:07:44.680
And then you hear 2 seconds of silence.

s111
00:07:44.680 --> 00:07:46.400
That's not very reassuring, right?

s112
00:07:46.400 --> 00:07:51.960
So, we need to be able to measure this this gap that the caller hears.

s113
00:07:52.480 --> 00:07:55.680
Now, the same story for lang- uh large language models.

s114
00:07:55.680 --> 00:08:01.000
So, time to first token is what we get in text.

s115
00:08:01.000 --> 00:08:07.280
It's it's a text-based measurement, but tokens do not equate to audio, either, right?

s116
00:08:07.280 --> 00:08:09.920
Your TTS can't start speaking on a token.

s117
00:08:09.920 --> 00:08:15.560
A TTS model text-to-speech model actually needs an entire sentence to start speaking.

s118
00:08:15.560 --> 00:08:20.160
Um and so, that's what we need to measure is actually time to first sentence.

s119
00:08:20.160 --> 00:08:29.120
Um because a an LLM that is very fast at producing tokens is not always the same LLM that produces the first sentence the fastest.

s120
00:08:29.120 --> 00:08:32.039
So, time to first sentence is what we should be looking at.

s121
00:08:32.039 --> 00:08:38.560
And then another thing that most leaderboards don't measure is these these agents, they need to do things.

s122
00:08:38.560 --> 00:08:39.880
They're not just there to chat with us.

s123
00:08:39.880 --> 00:08:42.320
They need to be able to do things on the back end.

s124
00:08:42.320 --> 00:08:49.120
They have to look up a transaction, uh check for a payment status, submit a dispute, right?

s125
00:08:49.120 --> 00:08:52.839
And so, these take time, 2 10 40 milliseconds.

s126
00:08:52.839 --> 00:08:54.520
And and so, how do we handle that?

s127
00:08:54.520 --> 00:09:00.240
How how do we hide 40 seconds or more of latency?

s128
00:09:00.240 --> 00:09:01.000
Well, we can't.

s129
00:09:01.000 --> 00:09:02.320
That The answer is we can't.

s130
00:09:02.320 --> 00:09:06.240
But ask yourself, what would a human do in the situation?

s131
00:09:06.240 --> 00:09:11.360
So, is a human just going to go silent while the system is doing something?

s132
00:09:11.360 --> 00:09:13.960
No, they're going to say, you know, "Give give me a second.

s133
00:09:13.960 --> 00:09:15.240
I'm looking that up.

s134
00:09:15.240 --> 00:09:17.360
Um system's running a little slow, almost there."

s135
00:09:17.360 --> 00:09:21.960
Or they're going to move on and they're going to ask another question while the system is doing something in the back end.

s136
00:09:21.960 --> 00:09:25.600
So, that's that that's the human-like behavior, multitasking, right?

s137
00:09:25.600 --> 00:09:26.560
We can do that.

s138
00:09:26.560 --> 00:09:30.160
So, why can't agents multitask and do that?

s139
00:09:30.160 --> 00:09:32.480
Well, that was a trick question because they can.

s140
00:09:32.480 --> 00:09:32.960
They should.

s141
00:09:32.960 --> 00:09:34.280
They should be able to.

s142
00:09:34.280 --> 00:09:36.080
And this is the shape of that.

s143
00:09:36.080 --> 00:09:46.360
Um this is the only code that I'm going to show you uh in this in this talk, but this is a normal tool call and the the one line is what makes the difference here.

s144
00:09:46.360 --> 00:09:56.360
Um, this one line, this update, this async await, this awaiting this tool call is is the whole trick here and this is what's handing the mic back to the agent.

s145
00:09:56.360 --> 00:10:02.520
This is allowing the conversation and the tool to stop sharing a thread.

s146
00:10:02.520 --> 00:10:04.240
They're just they're separate now.

s147
00:10:04.240 --> 00:10:13.640
And so um, this is what again makes the agent more human like because it's able to keep the conversation going as something is going on in the back end.

s148
00:10:13.640 --> 00:10:14.920
So, don't don't take my word for it.

s149
00:10:14.920 --> 00:10:20.040
I've got a quick demo here and we'll listen and see what it sounds like.

s150
00:10:20.040 --> 00:10:22.160
Thank you for calling The Gilded Rose Hotel.

s151
00:10:22.160 --> 00:10:23.760
How can I help you?

s152
00:10:23.760 --> 00:10:26.360
Yeah, I'd like to book a room.

s153
00:10:26.600 --> 00:10:28.520
What dates are you looking at?

s154
00:10:28.520 --> 00:10:32.320
Uh, next week, Wednesday through Friday.

s155
00:10:32.320 --> 00:10:36.680
Next Wednesday, July 30th to Friday, August 1st?

s156
00:10:36.680 --> 00:10:38.520
That's right.

s157
00:10:38.520 --> 00:10:40.640
How many guests will be staying?

s158
00:10:40.640 --> 00:10:42.640
Two.

s159
00:10:42.760 --> 00:10:48.760
I have queen two beds, king double queen, a suite or the penthouse?

s160
00:10:48.960 --> 00:10:51.800
Would you prefer a city or ocean view?

s161
00:10:51.800 --> 00:10:53.800
Ocean.

s162
00:10:54.280 --> 00:10:56.440
May I have your name for the booking?

s163
00:10:56.440 --> 00:10:58.760
Jesse Hall.

s164
00:10:58.920 --> 00:11:00.520
Is that Jesse Hall?

s165
00:11:00.520 --> 00:11:02.520
Yes.

s166
00:11:03.120 --> 00:11:04.640
What's your email?

s167
00:11:04.640 --> 00:11:06.640
me@me.com

s168
00:11:07.880 --> 00:11:10.600
Is me@me.com correct?

s169
00:11:10.600 --> 00:11:12.800
That's right.

s170
00:11:13.320 --> 00:11:15.040
And your phone number?

s171
00:11:15.040 --> 00:11:17.040
555-123-4567

s172
00:11:20.760 --> 00:11:23.240
Is that 555-123-4567?

s173
00:11:25.080 --> 00:11:27.240
That's right.

s174
00:11:27.600 --> 00:11:29.600
I'll need a credit card to hold the room.

s175
00:11:29.600 --> 00:11:30.800
What's the the number?

s176
00:11:30.800 --> 00:11:32.640
We'll stop here.

s177
00:11:32.640 --> 00:11:36.680
Okay, so you notice here that the conversation kept going.

s178
00:11:36.680 --> 00:11:38.920
There were no awkward pauses.

s179
00:11:38.920 --> 00:11:42.760
Um I even interrupted it and it just kept rolling with it, right?

s180
00:11:42.760 --> 00:11:46.160
It It didn't ever fall apart or feel unnatural.

s181
00:11:46.160 --> 00:11:51.080
And we see here, you can see the the tool calls that are happening along the way and the things that it's doing.

s182
00:11:51.080 --> 00:11:57.520
It's actually communicating with a back end and it's saving every uh piece of information that I'm giving it.

s183
00:11:57.520 --> 00:12:01.160
But what it's doing is it's not waiting for the back end to say, "Okay, that's saved.

s184
00:12:01.160 --> 00:12:02.680
Now you can move on to the next one."

s185
00:12:02.680 --> 00:12:03.520
"Okay, that's saved.

s186
00:12:03.520 --> 00:12:04.800
Move on to the next one."

s187
00:12:04.800 --> 00:12:07.720
It's moving on to the next thing immediately.

s188
00:12:07.720 --> 00:12:09.400
It's keeping the conversation going.

s189
00:12:09.400 --> 00:12:14.080
And if for some reason a tool call would have come back with an error or something, it would have then corrected itself.

s190
00:12:14.080 --> 00:12:17.600
It would have said, "Let me Let me verify this or that, whatever happened, right?"

s191
00:12:17.600 --> 00:12:29.120
And worst case scenario, if it was having issues, it would uh it would actually uh um pass me over to a human in that in that uh scenario then, right?

s192
00:12:29.120 --> 00:12:33.120
So, this is how our agents should operate.

s193
00:12:33.120 --> 00:12:42.640
Now, there's one more um axis of this, and that is knowing the agent knowing when to speak and when to to listen, when to keep listening.

s194
00:12:42.640 --> 00:12:44.760
And so, this is called turn detection.

s195
00:12:44.760 --> 00:12:53.040
And so, LiveKit has a um a built-in turn detector that doesn't only listen to the transcript like most turn detectors do.

s196
00:12:53.040 --> 00:12:54.880
It also listens to the audio.

s197
00:12:54.880 --> 00:12:58.560
It listens to the rhythm, the pitch, the intonation.

s198
00:12:58.560 --> 00:13:00.120
And we published the numbers on this.

s199
00:13:00.120 --> 00:13:08.440
The the leading um um alternative still interrupts and cuts users off 9.9% of the time.

s200
00:13:08.440 --> 00:13:12.080
And ours, 4.5% in our use case.

s201
00:13:12.080 --> 00:13:14.760
We ran it against the our hotel agent and others.

s202
00:13:14.760 --> 00:13:18.560
And that hap And that is what the numbers that we got for our use cases.

s203
00:13:18.560 --> 00:13:19.680
This is all open source.

s204
00:13:19.680 --> 00:13:24.160
You can actually run these e-vals and run this against your data set and see what numbers you get.

s205
00:13:24.160 --> 00:13:27.240
See if you get a um uh better uh increased numbers here.

s206
00:13:27.240 --> 00:13:33.200
And this is just one small piece of the puzzle again to make these voice agents more human-like.

s207
00:13:33.200 --> 00:13:37.160
Them knowing when to stop talking or when to listen.

s208
00:13:37.160 --> 00:13:46.120
For instance, if if I were trying to make a payment and I said I'd I'd like to pay 500 and $55.23, right?

s209
00:13:46.120 --> 00:13:55.280
Most of the turn detectors would have stopped at 500 because they saw 500 on the transcript and they saw 2 second pause, I must be done, right?

s210
00:13:55.280 --> 00:14:02.080
But ours would have heard my inflection, knew I wasn't done talking, and it would have waited for me.

s211
00:14:02.640 --> 00:14:05.960
So, let's look now at that latency budget, that the gap.

s212
00:14:05.960 --> 00:14:13.440
This is the gap between the caller's last syllable and the agent's first audible response.

s213
00:14:13.440 --> 00:14:30.160
So, the turn detector decides when the caller is done, the language model begins preparing a response based on a provisional transcript, and then the TTS starts generating that audio as soon as it can.

s214
00:14:30.160 --> 00:14:41.120
And so, every model that you choose, every function, every handoff, all of that is right here within this window, and this window has a finite number, a fixed window,

s215
00:14:41.120 --> 00:14:42.240
a fixed size.

s216
00:14:42.240 --> 00:14:49.880
And so, if you can stay roughly inside 1 and 1/2 seconds, the call still holds, the conversation holds.

s217
00:14:49.880 --> 00:14:53.760
It's starting to feel a little off at 1 and 1/2 seconds.

s218
00:14:53.760 --> 00:15:02.240
But if you can get down to 5 or 600 milliseconds between this the two turns, that's when it really starts to feel human-like.

s219
00:15:02.240 --> 00:15:05.040
And so, that question before, how fast can I go?

s220
00:15:05.040 --> 00:15:06.960
We just needed to get to go faster and faster.

s221
00:15:06.960 --> 00:15:08.240
That's not the answer.

s222
00:15:08.240 --> 00:15:15.520
The answer is how human-like of an agent can I fit within this latency window.

s223
00:15:15.520 --> 00:15:23.520
So, again, latency, that's a budget, and and we know how how to fit into that, but how can we be as human-like as possible?

s224
00:15:23.520 --> 00:15:24.960
That's the goal.

s225
00:15:24.960 --> 00:15:28.200
So, if we add all this up, that that question, which model should I pick?

s226
00:15:28.200 --> 00:15:39.000
We That was never something that we could answer because these scores, again, these leaderboards, they can't answer that because they're basing it off the models, but we are shipping a conversation.

s227
00:15:39.000 --> 00:15:40.680
So, let's answer that actual question.

s228
00:15:40.680 --> 00:15:41.920
How can we pick the models?

s229
00:15:41.920 --> 00:15:44.200
Which models should we pick?

s230
00:15:44.200 --> 00:15:46.080
So, this is what we have to do.

s231
00:15:46.080 --> 00:15:47.760
We test the conversation.

s232
00:15:47.760 --> 00:15:53.720
We take a complete working agent So, that hotel receptionist that you heard just a minute ago, we took that.

s233
00:15:53.720 --> 00:15:55.640
It has a real back end.

s234
00:15:55.640 --> 00:15:57.280
Um it's an actual It's not just a demo.

s235
00:15:57.280 --> 00:15:58.920
It actually works.

s236
00:15:58.920 --> 00:15:59.640
Uh and it's open source.

s237
00:15:59.640 --> 00:16:00.280
You can check it out.

s238
00:16:00.280 --> 00:16:04.600
So, we take that agent and we put simulated callers against it.

s239
00:16:04.600 --> 00:16:06.960
And these callers can pursue goals.

s240
00:16:06.960 --> 00:16:08.080
They can change their mind.

s241
00:16:08.080 --> 00:16:08.880
They can interrupt.

s242
00:16:08.880 --> 00:16:09.800
They can mumble.

s243
00:16:09.800 --> 00:16:11.160
We can add background noise.

s244
00:16:11.160 --> 00:16:18.720
We can make sure that this is a real-world scenario, and we run all of these simulated users against that agent.

s245
00:16:18.720 --> 00:16:23.800
Um we can run text-based simulations, which just tests our business logic.

s246
00:16:23.800 --> 00:16:29.880
Or we can also run audio-based simulations that tests the speech recognition capabilities.

s247
00:16:29.880 --> 00:16:31.480
It tests turn-taking.

s248
00:16:31.480 --> 00:16:36.839
The entire um the entire voice pipeline is what's tested there.

s249
00:16:36.839 --> 00:16:45.600
And so, we take the scenarios, and the scenario should describe what the caller wants and what actions the agent should take, what the outcome should be.

s250
00:16:45.600 --> 00:16:46.480
And then we score that.

s251
00:16:46.480 --> 00:16:48.120
We score those outcomes.

s252
00:16:48.120 --> 00:16:52.240
And so, in the hotel benchmark, you know, did the correct reservation hit the database?

s253
00:16:52.240 --> 00:16:57.839
In a financial benchmark, you know, did the correct authorized amount get charged, right?

s254
00:16:57.839 --> 00:17:00.040
And so, we we check these agents.

s255
00:17:00.040 --> 00:17:03.360
We make sure that every all the outcomes are met.

s256
00:17:03.360 --> 00:17:07.680
And then we don't score it based on just the completion alone, right?

s257
00:17:07.680 --> 00:17:11.240
Because we know that agents can game completions, right?

s258
00:17:11.240 --> 00:17:13.040
We have to make sure that they took the right steps.

s259
00:17:13.040 --> 00:17:17.480
Did they take 10 steps when they only should have taken two steps, right?

s260
00:17:17.480 --> 00:17:19.959
Did they hallucinate a function call?

s261
00:17:19.959 --> 00:17:23.079
These are the things that we have to test, the entire conversation there.

s262
00:17:23.079 --> 00:17:25.920
And then, we swap the stack out.

s263
00:17:25.920 --> 00:17:31.160
We We have the same agent, the same scenarios, but we change the models.

s264
00:17:31.160 --> 00:17:33.280
And then, we run that loop again.

s265
00:17:33.280 --> 00:17:38.520
The identical input Every model sees identical input byte for byte.

s266
00:17:38.520 --> 00:17:40.600
We swap the models, we run the loop.

s267
00:17:40.600 --> 00:17:51.280
We swap the models again, we run the loop continuously over and over, and we are building at that point a benchmark, a personalized benchmark, for our use case, for our agent.

s268
00:17:51.280 --> 00:17:53.920
So, there are as many benchmarks now as there are agents.

s269
00:17:53.920 --> 00:17:58.880
You run this against yours, and you find out which models work best for your situation.

s270
00:17:58.880 --> 00:18:04.360
That is the only way to find out which model is best for you.

s271
00:18:05.080 --> 00:18:07.640
Now, this is how you catch surprises.

s272
00:18:07.640 --> 00:18:15.120
Um a model that that gets the answer for you the quickest is the right model for you, right?

s273
00:18:15.120 --> 00:18:20.400
And surprisingly, that model may not be the model that that's at the top of the leaderboard, right?

s274
00:18:20.400 --> 00:18:22.880
But it's best for your use case.

s275
00:18:22.880 --> 00:18:28.040
And again, these benchmarks, the simulations, all this this methodology, it's all open source.

s276
00:18:28.040 --> 00:18:35.360
So, you can You can check it out for yourselves, you can run it against your agent, and see how yours performs.

s277
00:18:35.560 --> 00:18:39.840
And so, here is the results of that benchmark on our hotel agent.

s278
00:18:39.840 --> 00:18:44.960
This is the LLM that ranked the highest for our hotel receptionist agent.

s279
00:18:44.960 --> 00:18:47.880
That's Gemma 4 31B on LiveKit inference.

s280
00:18:47.880 --> 00:18:53.720
You notice on the second line, the same model, Gemma 4 31B on another provider, was slower.

s281
00:18:53.720 --> 00:18:59.160
So, LiveKit optimizes its inference serving for voice workloads.

s282
00:18:59.160 --> 00:19:02.440
And LiveKit is model agnostic, so you can use any model that you want.

s283
00:19:02.440 --> 00:19:10.040
So, again, you plug those models in, and you figure out which one is work it will work best for your use case.

s284
00:19:10.120 --> 00:19:18.920
Now, if you remember one thing from this this entire talk, remember this that a voice agent isn't just a chatbot with a speaker attached.

s285
00:19:18.920 --> 00:19:21.840
It's There's so much more complexity there.

s286
00:19:21.840 --> 00:19:26.160
It's a real-time system and that needs to respond like a person.

s287
00:19:26.160 --> 00:19:28.760
We need to make it as human-like as possible.

s288
00:19:28.760 --> 00:19:35.400
And and and when we evaluate it from that viewpoint, we will be able to to build that.

s289
00:19:35.400 --> 00:19:36.400
And so, here's a checklist.

s290
00:19:36.400 --> 00:19:37.880
You can take a take a photo here.

s291
00:19:37.880 --> 00:19:40.720
We need to set our latency budget that the caller hears.

s292
00:19:40.720 --> 00:19:44.360
Measure time to first audio, time to first sentence.

s293
00:19:44.360 --> 00:19:47.240
Confirm critical dates.

s294
00:19:47.240 --> 00:19:52.400
Anything that's financial money, we need to make sure that we have those those checks in place.

s295
00:19:52.400 --> 00:19:55.760
Verify identity before an account is accessed.

s296
00:19:55.760 --> 00:19:58.680
Keep listening while slow tools run.

s297
00:19:58.680 --> 00:20:00.640
Keep the conversation going.

s298
00:20:00.640 --> 00:20:04.560
And then run the complete conversation repeatedly through simulations.

s299
00:20:04.560 --> 00:20:09.240
Test the final state, privacy safeguards, unnecessary tool calls, turn-taking.

s300
00:20:09.240 --> 00:20:16.680
All these things and then when you find these failures, you turn those into regression scenarios and you keep testing over and over again.

s301
00:20:16.680 --> 00:20:21.840
So, start with one one of those workflows or failures that you care about.

s302
00:20:21.840 --> 00:20:23.880
Keep Keep iterating then.

s303
00:20:23.880 --> 00:20:24.320
Run it.

s304
00:20:24.320 --> 00:20:26.280
Listen to the conversation.

s305
00:20:26.280 --> 00:20:27.720
Check what happened on the back end.

s306
00:20:27.720 --> 00:20:35.040
At some point in the very near future, I hope, that we all call one of these voice agents and we talk over it.

s307
00:20:35.040 --> 00:20:41.040
We confuse it and it just rolls with it and it's and it's a very human-like experience.

s308
00:20:41.040 --> 00:20:44.960
That's That's our goal, but here's what I need from each and every one of you.

s309
00:20:44.960 --> 00:20:47.160
I need you to go build that.

s310
00:20:47.160 --> 00:20:49.400
Thank you.

s311
00:20:54.560 --> 00:20:56.560
Woo!