WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=j4AmIK6OvHc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/j4AmIK6OvHc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.480 --> 00:00:14.800
All right, let's get started.

s2
00:00:14.800 --> 00:00:22.680
So, most of the agents that we build today, they do their work in the background, right?

s3
00:00:22.680 --> 00:00:24.280
They write code for us.

s4
00:00:24.280 --> 00:00:25.720
They do research.

s5
00:00:25.720 --> 00:00:35.880
Uh you give them a task, you walk away, you come back, and so nobody really cares if uh a background agent takes an extra 10 seconds or even 10 minutes, right?

s6
00:00:35.880 --> 00:00:37.080
We're used to that.

s7
00:00:37.080 --> 00:00:43.320
But, in at some point in this past year, you've probably talked to another kind of agent.

s8
00:00:43.320 --> 00:00:45.360
A real-time agent.

s9
00:00:45.360 --> 00:00:50.120
Uh maybe uh it was a pharmacy refill, maybe at an airline.

s10
00:00:50.120 --> 00:00:53.920
Um I talked to one the other day when I called a car dealership.

s11
00:00:53.920 --> 00:00:56.800
And and so something happened on that call.

s12
00:00:56.800 --> 00:01:00.080
You probably did the most human thing possible.

s13
00:01:00.080 --> 00:01:03.520
You talked over that AI.

s14
00:01:03.520 --> 00:01:08.320
And that voice agent fell apart, and it just didn't feel natural, did it?

s15
00:01:08.320 --> 00:01:15.360
And so um if you ask uh anybody in this industry, you're probably going to get a very similar answer of how to fix that.

s16
00:01:15.360 --> 00:01:17.000
It's going to be speed.

s17
00:01:17.000 --> 00:01:21.560
Uh faster models, lower latency, shave off another 100 milliseconds.

s18
00:01:21.560 --> 00:01:24.120
Um but that is the wrong answer.

s19
00:01:24.120 --> 00:01:29.680
Uh and so latency it is a budget, and I'll I'll tell you exactly what that budget is.

s20
00:01:29.680 --> 00:01:31.520
But, the goal is something else entirely.

s21
00:01:31.520 --> 00:01:36.920
The goal is being uh the voice AI being as human-like as possible.

s22
00:01:36.920 --> 00:01:40.520
Uh an agent that listens while it talks.

s23
00:01:40.520 --> 00:01:45.800
Uh one that knows the difference between you finishing your thought and you still thinking.

s24
00:01:45.800 --> 00:01:49.320
Uh one that rolls with it when you change your mind.

s25
00:01:49.320 --> 00:01:51.880
And so speed alone can get you none of that.

s26
00:01:51.880 --> 00:01:57.200
What gets you there is which models you pick and how you configure your pipeline.

s27
00:01:57.200 --> 00:02:00.360
And so that's the first problem that we're going to encounter.

s28
00:02:00.360 --> 00:02:04.560
No one tells you how to pick the right models.

s29
00:02:04.560 --> 00:02:06.600
The leaderboards can't help you.

s30
00:02:06.600 --> 00:02:08.560
The vendors definitely can't help you.

s31
00:02:08.560 --> 00:02:12.040
So, let's talk about how we're going to fix that issue.

s32
00:02:12.040 --> 00:02:19.680
So, a show of hands quickly, who here has ever built a voice agent and shipped it to real users?

s33
00:02:19.680 --> 00:02:20.240
Okay.

s34
00:02:20.240 --> 00:02:21.280
All right.

s35
00:02:21.280 --> 00:02:28.760
Some of you have experience and you know the challenges that you'll face when deploying a real-time agent.

s36
00:02:28.760 --> 00:02:35.920
Another show of hands, who here has used ChatGPT's advanced voice mode to talk with ChatGPT?

s37
00:02:35.920 --> 00:02:38.120
Yeah, many of us have have used that, right?

s38
00:02:38.120 --> 00:02:42.680
So, OpenAI actually chose LiveKit to power that.

s39
00:02:42.680 --> 00:02:51.480
LiveKit is the open-source real-time framework and infrastructure for the most human-like voice agents possible.

s40
00:02:51.480 --> 00:02:57.160
Uh agents that can hear, see, and speak with the person on the other end.

s41
00:02:57.160 --> 00:03:02.560
Uh and so, not just voice, but also video, uh data, and even physical AI.

s42
00:03:02.560 --> 00:03:08.920
But, because voice is is challenging, uh we're going to actually focus on voice during this talk.

s43
00:03:08.920 --> 00:03:14.360
And so, um what we're going to uh look at is is voice agents.

s44
00:03:14.360 --> 00:03:20.440
They basically have um three models uh that work together as one system.

s45
00:03:20.440 --> 00:03:25.200
So, your speech-to-text, your LLM, and then your speech uh text-to-speech.

s46
00:03:25.200 --> 00:03:26.640
So, three slots.

s47
00:03:26.640 --> 00:03:30.040
Now, I do have to acknowledge that there are real-time models as well.

s48
00:03:30.040 --> 00:03:34.000
And and real-time models, uh they are a newer concept.

s49
00:03:34.000 --> 00:03:37.800
They're getting better and better, uh but they're not right for every use case.

s50
00:03:37.800 --> 00:03:48.239
And just the fact that there are real-time models even adds more complexity to our problem of trying to figure out which model is best for our voice pipeline.

s51
00:03:48.239 --> 00:03:51.000
So, here is what you have to choose from.

s52
00:03:51.000 --> 00:04:00.520
Hundreds of models, dozens of providers, uh and show of hands here, who has picked a model just because they saw it at the top of a leaderboard?

s53
00:04:00.520 --> 00:04:02.440
Yeah, most of us.

s54
00:04:02.440 --> 00:04:05.960
Um but here's why that doesn't work for real-time agents.

s55
00:04:05.960 --> 00:04:14.240
Leaderboards score components, but you are shipping a cascade, a pipeline.

s56
00:04:14.360 --> 00:04:19.040
So, let's say that your caller wants to book an appointment on Tuesday the 3rd.

s57
00:04:19.040 --> 00:04:23.200
Your speech-to-text hears Tuesday the 30th.

s58
00:04:23.200 --> 00:04:36.600
And so, from that moment on, it doesn't matter how smart your language model is, it's now confidently booking the wrong day, and every error flows downhill from there, and the layers below it have no idea.

s59
00:04:36.600 --> 00:04:42.400
And so, those leaderboard scores, they came from clean audio in a quiet room.

s60
00:04:42.400 --> 00:04:49.120
Uh but your callers, they're calling on a speakerphone, in traffic, with kids in the backseat.

s61
00:04:49.120 --> 00:04:52.360
So, those leaderboards never met your users.

s62
00:04:52.360 --> 00:04:55.360
Uh and there's a name for this, the leaderboard trap.

s63
00:04:55.360 --> 00:05:01.720
So, this the score measures the model, but again, you're shipping a conversation.

s64
00:05:01.720 --> 00:05:09.200
But everyone building voice agents, they're asking, you know, what is the best speech-to-text model, LLM, TTS?

s65
00:05:09.200 --> 00:05:14.240
But the real question is, which stack should I ship?

s66
00:05:14.240 --> 00:05:21.600
So, the these are two completely different questions, and the only only one of them has a leaderboard, until today.

s67
00:05:21.600 --> 00:05:24.240
Uh we're going to talk about that more in a bit.

s68
00:05:24.240 --> 00:05:30.280
So, let's say you fall into this trap, you pick your models based on the leaderboard, and now you start measuring things.

s69
00:05:30.280 --> 00:05:32.000
This is the next problem.

s70
00:05:32.000 --> 00:05:34.840
You're probably measuring the wrong thing.

s71
00:05:34.840 --> 00:05:45.400
You see, your TTS provider uh reports time to first byte, and that looks great on the dashboard, uh but bytes are not sound.

s72
00:05:45.400 --> 00:05:53.200
And so, we've measured popular providers that are actually shipping hundreds of milliseconds of silence at the beginning of their audio.

s73
00:05:53.200 --> 00:06:01.040
And in the worst case, up to 3/4 of a second of nothing but silence before that first audio comes in.

s74
00:06:01.040 --> 00:06:05.720
So, time to first audio is actually what your caller hears.

s75
00:06:05.720 --> 00:06:09.360
And so, this is the distinction that this entire talk hangs on.

s76
00:06:09.360 --> 00:06:13.480
This measured latency versus perceived latency.

s77
00:06:13.480 --> 00:06:17.040
So, what your dashboard says versus what your user hears.

s78
00:06:17.040 --> 00:06:21.520
The dashboard says it's really fast, but your ears say it's slow.

s79
00:06:21.520 --> 00:06:23.760
Same story for language models.

s80
00:06:23.760 --> 00:06:27.640
Time to first token is a text metric.

s81
00:06:27.640 --> 00:06:32.240
So, your TTS can't start speaking though on a token, right?

s82
00:06:32.240 --> 00:06:34.480
So, it needs an entire sentence.

s83
00:06:34.480 --> 00:06:45.320
So, that that number matters for for voice because it needs the entire first sentence, but the model ranks differently on that.

s84
00:06:45.320 --> 00:06:55.640
So, a model that streams back back a model that streams tokens the fastest is not necessarily the model that finishes a sentence first.

s85
00:06:55.919 --> 00:07:05.400
Now, if if you let's say you do measure all of this perfectly, there's another part of the conversation that the benchmarking that gets completely ignored.

s86
00:07:05.400 --> 00:07:09.000
And that is that a real agent doesn't just chat.

s87
00:07:09.000 --> 00:07:10.600
It needs to do things, right?

s88
00:07:10.600 --> 00:07:15.280
It has to look up orders, check availability, book rooms, right?

s89
00:07:15.280 --> 00:07:18.480
And so, a tool is going to take time.

s90
00:07:18.480 --> 00:07:21.240
2 seconds, 10, 40, or more seconds.

s91
00:07:21.240 --> 00:07:28.080
And no model choice is going to be able to hide a 40 or more second latency.

s92
00:07:28.080 --> 00:07:31.960
So, think about what a human receptionist would do here.

s93
00:07:31.960 --> 00:07:35.680
They they don't freeze while the the booking system loads, right?

s94
00:07:35.680 --> 00:07:37.400
They say, "Give me a sec.

s95
00:07:37.400 --> 00:07:39.360
I'm looking up your your account, right?"

s96
00:07:39.360 --> 00:07:40.960
They they buy time.

s97
00:07:40.960 --> 00:07:43.920
And they listen while they keep working.

s98
00:07:43.920 --> 00:07:47.440
And that is that human-like behavior that we're looking for.

s99
00:07:47.440 --> 00:07:53.120
So, why can't an agent just keep talking and do things asynchronously?

s100
00:07:53.120 --> 00:07:54.480
Well, it can.

s101
00:07:54.480 --> 00:07:57.480
And here's the shape of that, and this is the only code that you're going to see.

s102
00:07:57.480 --> 00:07:59.200
Uh but this is a normal tool.

s103
00:07:59.200 --> 00:08:02.640
Uh there's just one thing different, and that is this line right here.

s104
00:08:02.640 --> 00:08:10.720
Uh the first update hands the mic back to the agent while it keeps uh working in the background.

s105
00:08:10.720 --> 00:08:12.280
And that's the whole trick.

s106
00:08:12.280 --> 00:08:17.160
The conversation and the tool stop sharing a thread.

s107
00:08:17.160 --> 00:08:19.200
Uh and so, don't take my word for it.

s108
00:08:19.200 --> 00:08:21.480
It Listen for yourselves.

s109
00:08:21.480 --> 00:08:23.080
Thanks for calling the Live Oak Hotel.

s110
00:08:23.080 --> 00:08:24.040
How can I help you?

s111
00:08:24.040 --> 00:08:26.360
Can I book a room?

s112
00:08:26.360 --> 00:08:27.880
Of course.

s113
00:08:27.880 --> 00:08:29.280
What dates are you looking at?

s114
00:08:29.280 --> 00:08:32.039
October the 10th through 12th.

s115
00:08:32.039 --> 00:08:33.960
October 10th to 12th?

s116
00:08:33.960 --> 00:08:35.479
And for how many guests?

s117
00:08:35.479 --> 00:08:37.479
Two.

s118
00:08:38.200 --> 00:08:42.320
I have queen with two beds, king, double, queen, or suite.

s119
00:08:42.320 --> 00:08:45.839
Which Would you prefer a city or ocean view?

s120
00:08:45.839 --> 00:08:47.839
Ocean.

s121
00:08:47.960 --> 00:08:49.760
Who should I put the booking under?

s122
00:08:49.760 --> 00:08:52.200
Jesse Hall.

s123
00:08:55.400 --> 00:09:04.800
You're all set for October 10th to 12th in a king room with an ocean view for $582.40 on the card ending in 1111.

s124
00:09:04.800 --> 00:09:06.280
Shall I go ahead?

s125
00:09:06.280 --> 00:09:07.040
Actually, you know what?

s126
00:09:07.040 --> 00:09:12.680
I need to change the date to December the 20th through the 24th.

s127
00:09:12.680 --> 00:09:15.800
December 20th to 24th?

s128
00:09:15.880 --> 00:09:22.400
That brings the total to $1,164.80 for December 20th to 24th.

s129
00:09:22.400 --> 00:09:23.600
Shall I go ahead?

s130
00:09:23.600 --> 00:09:25.600
Yes.

s131
00:09:25.839 --> 00:09:27.360
You're all set.

s132
00:09:27.360 --> 00:09:35.040
Okay, so we had to speed up some of the parts there because of time, but the the the agent kept the conversation going.

s133
00:09:35.040 --> 00:09:36.839
There were no awkward pauses.

s134
00:09:36.839 --> 00:09:40.160
The caller even interrupted it and it kept going.

s135
00:09:40.160 --> 00:09:45.760
And the caller also changed his mind in the middle of things and the agent just rolled with it, right?

s136
00:09:45.760 --> 00:09:49.080
This is the human-like behavior that we're looking for.

s137
00:09:49.080 --> 00:09:54.640
So this is again not one model being good, this is an entire system being good.

s138
00:09:54.640 --> 00:09:59.480
This is the infrastructure, the framework and the models all working together.

s139
00:09:59.480 --> 00:10:05.440
And that's exactly what no component or model benchmark can see.

s140
00:10:05.440 --> 00:10:13.640
Now there's one more axis to this that the leaderboards miss entirely and this is knowing when to speak and when not to speak.

s141
00:10:13.640 --> 00:10:22.200
So humans negotiate turn taking in a matter of just a a couple of a hundred milliseconds, mostly from tone.

s142
00:10:22.200 --> 00:10:27.240
And a pipeline that it waits for the entire transcript to come back, it's already late, right?

s143
00:10:27.240 --> 00:10:31.520
So we built a turn detector that is based on audio itself.

s144
00:10:31.520 --> 00:10:40.680
So it listens to the rhyme, the pitch, the intonation of the speech of the audio and then we published the numbers on this.

s145
00:10:40.680 --> 00:10:49.040
So at the same latency budget, the best alternative cuts users off 9.9% of the time.

s146
00:10:49.040 --> 00:10:54.120
Ours is 4.5% and the eval harness is open source.

s147
00:10:54.120 --> 00:11:00.160
You don't have to trust me, you can run your own data through this and see what you get.

s148
00:11:00.160 --> 00:11:02.080
So let's go one level deeper for a minute.

s149
00:11:02.080 --> 00:11:08.520
The gap between your caller's last syllable and your agent's first audible bite.

s150
00:11:08.520 --> 00:11:22.640
So end of turn detection fires from audio before the transcript lands, the language model is already running on a provisional transcript and if the final words match then you just saved a half a second.

s151
00:11:22.640 --> 00:11:28.160
And then the TTS starts on the first complete sentence, not the last token.

s152
00:11:28.160 --> 00:11:34.560
And so the the the first audible bite is when the human stops waiting.

s153
00:11:34.560 --> 00:11:40.680
And so every model that you choose, it shows up here in this window, and this window has a fixed size.

s154
00:11:40.680 --> 00:11:48.160
If you can stay within a second and a half end to end, the conversation still holds, but you're you're right on the edge.

s155
00:11:48.160 --> 00:11:53.960
If you can get closer to 600 milliseconds, then it starts to feel really human.

s156
00:11:53.960 --> 00:12:02.880
And so the question is never how fast can I go, but what is the most human-like agent that I can fit inside this window?

s157
00:12:02.880 --> 00:12:08.680
So back to latency as a budget, but human-like is the goal.

s158
00:12:08.720 --> 00:12:14.520
So if we add all of this up, the question that we asked at the beginning, what models should I pick?

s159
00:12:14.520 --> 00:12:16.840
That was never actually answerable.

s160
00:12:16.840 --> 00:12:20.680
And so we built the thing that we wish existed.

s161
00:12:20.680 --> 00:12:23.160
Here's how we built the benchmark.

s162
00:12:23.160 --> 00:12:24.960
We take a complete working agent.

s163
00:12:24.960 --> 00:12:28.960
The first one is that hotel receptionist that you just heard.

s164
00:12:28.960 --> 00:12:31.080
And it it takes real bookings.

s165
00:12:31.080 --> 00:12:32.880
It has a real back end.

s166
00:12:32.880 --> 00:12:34.800
And it's also open source.

s167
00:12:34.800 --> 00:12:36.080
You can look at the repo.

s168
00:12:36.080 --> 00:12:40.680
And we run hundreds of simulated callers against it.

s169
00:12:40.680 --> 00:12:42.960
The the simulator interrupts.

s170
00:12:42.960 --> 00:12:44.040
It mumbles.

s171
00:12:44.040 --> 00:12:47.440
It changes its mind, just like a real caller would.

s172
00:12:47.440 --> 00:12:55.680
And then we don't score it on vibes, but we score outcomes defined the way that we would define them for a human.

s173
00:12:55.680 --> 00:12:58.839
And so did the right reservation land in the database?

s174
00:12:58.839 --> 00:13:00.760
Did a caller get cut off?

s175
00:13:00.760 --> 00:13:03.080
Are the names and dates correct?

s176
00:13:03.080 --> 00:13:08.120
Are there were there any invented tool calls or any leaked personal data, right?

s177
00:13:08.120 --> 00:13:12.880
So we measure all of those things against human equivalent behaviors.

s178
00:13:12.880 --> 00:13:14.760
That's the bar.

s179
00:13:14.760 --> 00:13:21.240
And then completion alone isn't enough, because completion can actually be gameable, right?

s180
00:13:21.240 --> 00:13:25.040
So a model can kind of guess its way to complete, right?

s181
00:13:25.040 --> 00:13:29.640
And so we penalize what a completion a completion score can't see.

s182
00:13:29.640 --> 00:13:38.520
If like if it asks questions that didn't need to be asked or if it takes 10 steps to do something that should have only taken two steps.

s183
00:13:38.520 --> 00:13:40.640
And then we swap the stack out.

s184
00:13:40.640 --> 00:13:44.760
The same agent, the same scenarios, but different models.

s185
00:13:44.760 --> 00:13:49.280
Every model sees the exact identical input byte for byte.

s186
00:13:49.280 --> 00:13:55.560
But this creates benchmarks against our agent example.

s187
00:13:55.560 --> 00:14:00.040
There are as many benchmarks as there are agents.

s188
00:14:00.040 --> 00:14:02.560
So the benchmark is actually yours.

s189
00:14:02.560 --> 00:14:08.520
You can run this exact thing run your agent through this exact benchmark and get your scores.

s190
00:14:08.520 --> 00:14:12.200
And so this is how you catch surprises.

s191
00:14:12.200 --> 00:14:16.840
The whole thing is open source, the methodology, the eval sets, the agent itself.

s192
00:14:16.840 --> 00:14:21.840
You can point the same thing at your agent and let us know how it performs for you.

s193
00:14:21.840 --> 00:14:27.920
Now here's an important fact about LiveKit is that we don't make STT, LLM, or TTS models.

s194
00:14:27.920 --> 00:14:31.000
Every provider plugs into our stack.

s195
00:14:31.000 --> 00:14:32.839
So we don't have a horse in this race.

s196
00:14:32.839 --> 00:14:35.839
We just watch all of them run a lot.

s197
00:14:35.839 --> 00:14:38.960
And so we can confidently pick the winners.

s198
00:14:38.960 --> 00:14:47.480
And so what you're looking at right here is a sneak peek of the Gemini 4 31B model on LiveKit inference.

s199
00:14:47.480 --> 00:14:51.360
But this isn't just an off-the-shelf Gemma 4 31B.

s200
00:14:51.360 --> 00:15:00.240
It's tuned specifically for production voice agents with long prompts, many tools, and tight latency budgets.

s201
00:15:00.240 --> 00:15:07.480
It passes voice evals at 88% and it starts speaking in about 380 milliseconds.

s202
00:15:07.480 --> 00:15:11.240
So that's over two times faster than the next model.

s203
00:15:11.240 --> 00:15:18.560
Making this the actual the fastest high-quality voice LLM available today.

s204
00:15:18.560 --> 00:15:23.400
And so this is actually going to going to launch tomorrow on LiveKit inference.

s205
00:15:23.400 --> 00:15:30.320
So again, our goal is to have the most human-like agents possible while sticking to that latency budget.

s206
00:15:30.320 --> 00:15:33.040
So if you remember one thing from this talk, make it this one.

s207
00:15:33.040 --> 00:15:38.320
A voice agent isn't a chatbot with a speaker stacked onto it.

s208
00:15:38.320 --> 00:15:40.320
It's human-like infrastructure.

s209
00:15:40.320 --> 00:15:46.480
A real-time system that senses and responds the way that a human, a person, would.

s210
00:15:46.480 --> 00:15:49.600
The moment that you treat it that way, every decision is going to get clearer.

s211
00:15:49.600 --> 00:15:54.720
The budget, the metrics, the models, and the bar that you test against.

s212
00:15:54.720 --> 00:15:55.680
So here's the checklist.

s213
00:15:55.680 --> 00:15:57.200
You can take a photo of this.

s214
00:15:57.200 --> 00:15:59.360
Set your latency budget first.

s215
00:15:59.360 --> 00:16:02.160
Measure perceived latency.

s216
00:16:02.160 --> 00:16:07.120
Spend whatever budget is left on model quality, not on speed that doesn't matter.

s217
00:16:07.120 --> 00:16:12.960
And then anything over a second it should go asynchronously so that the conversation never stops.

s218
00:16:12.960 --> 00:16:17.720
And then just test the conversation, not the components.

s219
00:16:17.720 --> 00:16:20.280
And then keep testing in production.

s220
00:16:20.280 --> 00:16:22.280
So here's the benchmark.

s221
00:16:22.280 --> 00:16:23.320
Uh scan the QR code.

s222
00:16:23.320 --> 00:16:24.520
You can check out the benchmark.

s223
00:16:24.520 --> 00:16:27.200
You can run this against your own agents.

s224
00:16:27.200 --> 00:16:37.640
And at uh remember at some point in the near future, you are going to call a voice agent, and you're going to talk over it, and it's not going to fall apart.

s225
00:16:37.640 --> 00:16:40.720
It's going to act like a human would in that scenario.

s226
00:16:40.720 --> 00:16:42.720
So go build that.

s227
00:16:42.720 --> 00:16:44.960
Thank you.

s228
00:16:58.536 --> 00:17:00.536
[music]