Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit

AI Engineer · 17 min · 228 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12All right, let's get started.
00:14So, most of the agents that we build today, they do their work in the background, right?
00:22They write code for us.
00:24They do research.
00:25Uh you give them a task, you walk away, you come back, and so nobody really cares if uh a background agent takes an extra 10 seconds or even 10 minutes, right?
00:35We're used to that.
00:37But, in at some point in this past year, you've probably talked to another kind of agent.
00:43A real-time agent.
00:45Uh maybe uh it was a pharmacy refill, maybe at an airline.
00:50Um I talked to one the other day when I called a car dealership.
00:53And and so something happened on that call.
00:56You probably did the most human thing possible.
01:00You talked over that AI.
01:03And that voice agent fell apart, and it just didn't feel natural, did it?
01:08And so um if you ask uh anybody in this industry, you're probably going to get a very similar answer of how to fix that.
01:15It's going to be speed.
01:17Uh faster models, lower latency, shave off another 100 milliseconds.
01:21Um but that is the wrong answer.
01:24Uh and so latency it is a budget, and I'll I'll tell you exactly what that budget is.
01:29But, the goal is something else entirely.
01:31The goal is being uh the voice AI being as human-like as possible.
01:36Uh an agent that listens while it talks.
01:40Uh one that knows the difference between you finishing your thought and you still thinking.
01:45Uh one that rolls with it when you change your mind.
01:49And so speed alone can get you none of that.
01:51What gets you there is which models you pick and how you configure your pipeline.
01:57And so that's the first problem that we're going to encounter.
02:00No one tells you how to pick the right models.
02:04The leaderboards can't help you.
02:06The vendors definitely can't help you.
02:08So, let's talk about how we're going to fix that issue.
02:12So, a show of hands quickly, who here has ever built a voice agent and shipped it to real users?
02:19Okay.
02:20All right.
02:21Some of you have experience and you know the challenges that you'll face when deploying a real-time agent.
02:28Another show of hands, who here has used ChatGPT's advanced voice mode to talk with ChatGPT?
02:35Yeah, many of us have have used that, right?
02:38So, OpenAI actually chose LiveKit to power that.
02:42LiveKit is the open-source real-time framework and infrastructure for the most human-like voice agents possible.
02:51Uh agents that can hear, see, and speak with the person on the other end.
02:57Uh and so, not just voice, but also video, uh data, and even physical AI.
03:02But, because voice is is challenging, uh we're going to actually focus on voice during this talk.
03:08And so, um what we're going to uh look at is is voice agents.
03:14They basically have um three models uh that work together as one system.
03:20So, your speech-to-text, your LLM, and then your speech uh text-to-speech.
03:25So, three slots.
03:26Now, I do have to acknowledge that there are real-time models as well.
03:30And and real-time models, uh they are a newer concept.
03:34They're getting better and better, uh but they're not right for every use case.
03:37And just the fact that there are real-time models even adds more complexity to our problem of trying to figure out which model is best for our voice pipeline.
03:48So, here is what you have to choose from.
03:51Hundreds of models, dozens of providers, uh and show of hands here, who has picked a model just because they saw it at the top of a leaderboard?
04:00Yeah, most of us.
04:02Um but here's why that doesn't work for real-time agents.
04:05Leaderboards score components, but you are shipping a cascade, a pipeline.
04:14So, let's say that your caller wants to book an appointment on Tuesday the 3rd.
04:19Your speech-to-text hears Tuesday the 30th.
04:23And so, from that moment on, it doesn't matter how smart your language model is, it's now confidently booking the wrong day, and every error flows downhill from there, and the layers below it have no idea.
04:36And so, those leaderboard scores, they came from clean audio in a quiet room.
04:42Uh but your callers, they're calling on a speakerphone, in traffic, with kids in the backseat.
04:49So, those leaderboards never met your users.
04:52Uh and there's a name for this, the leaderboard trap.
04:55So, this the score measures the model, but again, you're shipping a conversation.
05:01But everyone building voice agents, they're asking, you know, what is the best speech-to-text model, LLM, TTS?
05:09But the real question is, which stack should I ship?
05:14So, the these are two completely different questions, and the only only one of them has a leaderboard, until today.
05:21Uh we're going to talk about that more in a bit.
05:24So, let's say you fall into this trap, you pick your models based on the leaderboard, and now you start measuring things.
05:30This is the next problem.
05:32You're probably measuring the wrong thing.
05:34You see, your TTS provider uh reports time to first byte, and that looks great on the dashboard, uh but bytes are not sound.
05:45And so, we've measured popular providers that are actually shipping hundreds of milliseconds of silence at the beginning of their audio.
05:53And in the worst case, up to 3/4 of a second of nothing but silence before that first audio comes in.
06:01So, time to first audio is actually what your caller hears.
06:05And so, this is the distinction that this entire talk hangs on.
06:09This measured latency versus perceived latency.
06:13So, what your dashboard says versus what your user hears.
06:17The dashboard says it's really fast, but your ears say it's slow.
06:21Same story for language models.
06:23Time to first token is a text metric.
06:27So, your TTS can't start speaking though on a token, right?
06:32So, it needs an entire sentence.
06:34So, that that number matters for for voice because it needs the entire first sentence, but the model ranks differently on that.
06:45So, a model that streams back back a model that streams tokens the fastest is not necessarily the model that finishes a sentence first.
06:55Now, if if you let's say you do measure all of this perfectly, there's another part of the conversation that the benchmarking that gets completely ignored.
07:05And that is that a real agent doesn't just chat.
07:09It needs to do things, right?
07:10It has to look up orders, check availability, book rooms, right?
07:15And so, a tool is going to take time.
07:182 seconds, 10, 40, or more seconds.
07:21And no model choice is going to be able to hide a 40 or more second latency.
07:28So, think about what a human receptionist would do here.
07:31They they don't freeze while the the booking system loads, right?
07:35They say, "Give me a sec.
07:37I'm looking up your your account, right?"
07:39They they buy time.
07:40And they listen while they keep working.
07:43And that is that human-like behavior that we're looking for.
07:47So, why can't an agent just keep talking and do things asynchronously?
07:53Well, it can.
07:54And here's the shape of that, and this is the only code that you're going to see.
07:57Uh but this is a normal tool.
07:59Uh there's just one thing different, and that is this line right here.
08:02Uh the first update hands the mic back to the agent while it keeps uh working in the background.
08:10And that's the whole trick.
08:12The conversation and the tool stop sharing a thread.
08:17Uh and so, don't take my word for it.
08:19It Listen for yourselves.
08:21Thanks for calling the Live Oak Hotel.
08:23How can I help you?
08:24Can I book a room?
08:26Of course.
08:27What dates are you looking at?
08:29October the 10th through 12th.
08:32October 10th to 12th?
08:33And for how many guests?
08:35Two.
08:38I have queen with two beds, king, double, queen, or suite.
08:42Which Would you prefer a city or ocean view?
08:45Ocean.
08:47Who should I put the booking under?
08:49Jesse Hall.
08:55You're all set for October 10th to 12th in a king room with an ocean view for $582.40 on the card ending in 1111.
09:04Shall I go ahead?
09:06Actually, you know what?
09:07I need to change the date to December the 20th through the 24th.
09:12December 20th to 24th?
09:15That brings the total to $1,164.80 for December 20th to 24th.
09:22Shall I go ahead?
09:23Yes.
09:25You're all set.
09:27Okay, so we had to speed up some of the parts there because of time, but the the the agent kept the conversation going.
09:35There were no awkward pauses.
09:36The caller even interrupted it and it kept going.
09:40And the caller also changed his mind in the middle of things and the agent just rolled with it, right?
09:45This is the human-like behavior that we're looking for.
09:49So this is again not one model being good, this is an entire system being good.
09:54This is the infrastructure, the framework and the models all working together.
09:59And that's exactly what no component or model benchmark can see.
10:05Now there's one more axis to this that the leaderboards miss entirely and this is knowing when to speak and when not to speak.
10:13So humans negotiate turn taking in a matter of just a a couple of a hundred milliseconds, mostly from tone.
10:22And a pipeline that it waits for the entire transcript to come back, it's already late, right?
10:27So we built a turn detector that is based on audio itself.
10:31So it listens to the rhyme, the pitch, the intonation of the speech of the audio and then we published the numbers on this.
10:40So at the same latency budget, the best alternative cuts users off 9.9% of the time.
10:49Ours is 4.5% and the eval harness is open source.
10:54You don't have to trust me, you can run your own data through this and see what you get.
11:00So let's go one level deeper for a minute.
11:02The gap between your caller's last syllable and your agent's first audible bite.
11:08So end of turn detection fires from audio before the transcript lands, the language model is already running on a provisional transcript and if the final words match then you just saved a half a second.
11:22And then the TTS starts on the first complete sentence, not the last token.
11:28And so the the the first audible bite is when the human stops waiting.
11:34And so every model that you choose, it shows up here in this window, and this window has a fixed size.
11:40If you can stay within a second and a half end to end, the conversation still holds, but you're you're right on the edge.
11:48If you can get closer to 600 milliseconds, then it starts to feel really human.
11:53And so the question is never how fast can I go, but what is the most human-like agent that I can fit inside this window?
12:02So back to latency as a budget, but human-like is the goal.
12:08So if we add all of this up, the question that we asked at the beginning, what models should I pick?
12:14That was never actually answerable.
12:16And so we built the thing that we wish existed.
12:20Here's how we built the benchmark.
12:23We take a complete working agent.
12:24The first one is that hotel receptionist that you just heard.
12:28And it it takes real bookings.
12:31It has a real back end.
12:32And it's also open source.
12:34You can look at the repo.
12:36And we run hundreds of simulated callers against it.
12:40The the simulator interrupts.
12:42It mumbles.
12:44It changes its mind, just like a real caller would.
12:47And then we don't score it on vibes, but we score outcomes defined the way that we would define them for a human.
12:55And so did the right reservation land in the database?
12:58Did a caller get cut off?
13:00Are the names and dates correct?
13:03Are there were there any invented tool calls or any leaked personal data, right?
13:08So we measure all of those things against human equivalent behaviors.
13:12That's the bar.
13:14And then completion alone isn't enough, because completion can actually be gameable, right?
13:21So a model can kind of guess its way to complete, right?
13:25And so we penalize what a completion a completion score can't see.
13:29If like if it asks questions that didn't need to be asked or if it takes 10 steps to do something that should have only taken two steps.
13:38And then we swap the stack out.
13:40The same agent, the same scenarios, but different models.
13:44Every model sees the exact identical input byte for byte.
13:49But this creates benchmarks against our agent example.
13:55There are as many benchmarks as there are agents.
14:00So the benchmark is actually yours.
14:02You can run this exact thing run your agent through this exact benchmark and get your scores.
14:08And so this is how you catch surprises.
14:12The whole thing is open source, the methodology, the eval sets, the agent itself.
14:16You can point the same thing at your agent and let us know how it performs for you.
14:21Now here's an important fact about LiveKit is that we don't make STT, LLM, or TTS models.
14:27Every provider plugs into our stack.
14:31So we don't have a horse in this race.
14:32We just watch all of them run a lot.
14:35And so we can confidently pick the winners.
14:38And so what you're looking at right here is a sneak peek of the Gemini 4 31B model on LiveKit inference.
14:47But this isn't just an off-the-shelf Gemma 4 31B.
14:51It's tuned specifically for production voice agents with long prompts, many tools, and tight latency budgets.
15:00It passes voice evals at 88% and it starts speaking in about 380 milliseconds.
15:07So that's over two times faster than the next model.
15:11Making this the actual the fastest high-quality voice LLM available today.
15:18And so this is actually going to going to launch tomorrow on LiveKit inference.
15:23So again, our goal is to have the most human-like agents possible while sticking to that latency budget.
15:30So if you remember one thing from this talk, make it this one.
15:33A voice agent isn't a chatbot with a speaker stacked onto it.
15:38It's human-like infrastructure.
15:40A real-time system that senses and responds the way that a human, a person, would.
15:46The moment that you treat it that way, every decision is going to get clearer.
15:49The budget, the metrics, the models, and the bar that you test against.
15:54So here's the checklist.
15:55You can take a photo of this.
15:57Set your latency budget first.
15:59Measure perceived latency.
16:02Spend whatever budget is left on model quality, not on speed that doesn't matter.
16:07And then anything over a second it should go asynchronously so that the conversation never stops.
16:12And then just test the conversation, not the components.
16:17And then keep testing in production.
16:20So here's the benchmark.
16:22Uh scan the QR code.
16:23You can check out the benchmark.
16:24You can run this against your own agents.
16:27And at uh remember at some point in the near future, you are going to call a voice agent, and you're going to talk over it, and it's not going to fall apart.
16:37It's going to act like a human would in that scenario.
16:40So go build that.
16:42Thank you.
16:58[music]