How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
https://www.youtube.com/watch?v=OmyqelTwpOo
AI Engineer · 20 min · 311 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] Um all right, we're going to talk about voice agents.
[00:16] And most of the agents that we deal with on a day-to-day basis, um they they do things in the background, right?
[00:24] They they write code for us, they do research, we give them something to do, a task, and we walk away, we come back, and we don't really care if they take an extra 10 seconds, an extra 10 minutes, maybe sometimes 10 hours,
[00:38] right?
[00:39] Um and we expect that.
[00:40] But at some point in the last year or so, you have probably interacted with another kind of agent, a real-time agent.
[00:48] Maybe when you called your pharmacy, maybe uh when you called the airline.
[00:53] I talked to one just the other day when I called a car dealership.
[00:56] And at some point in that conversation, you did the the most human-like thing possible.
[01:01] You spoke over it.
[01:03] You interrupted it.
[01:04] You confused it.
[01:06] And the entire conversation fell apart, it just didn't feel natural.
[01:09] And then you did what I do every single time, operator, hit zero, like get me to a human, right?
[01:16] So, how do we fix that?
[01:17] How do we make these agents as human-like as possible?
[01:21] Um and so, like in in a um a finance space, what if your caller was calling to schedule a payment, and they changed the the payment amount midway through, they changed the date,
[01:35] um and and the agent didn't keep up with it, right?
[01:38] That is where we start having issues, especially in the finance space.
[01:42] The the the the risk is much higher, right?
[01:45] So, how do we fix that?
[01:47] So, we're going to talk about how can we build human-like voice agents for financial services.
[01:53] And when you think about building a good voice agent, one of the first things that comes to mind is latency.
[01:59] We need it to be as fast as possible, right?
[02:01] Well, latency is part of the equation.
[02:03] Latency has a finite budget, and we'll talk about what that is.
[02:07] Um but the keyword here is human-like.
[02:10] How do we make these voice agents as human-like as we can?
[02:14] That's the goal.
[02:15] So, there's quite a few people that are talking about voice lately.
[02:19] Um some of these might look familiar.
[02:21] We've got this this guy Sam.
[02:23] Um he's excited about voice models.
[02:26] He says uh that they're starting to change the way they that we interface with AI.
[02:33] Isn't that true?
[02:33] And then this this um weird name here in the bottom right, Swyx.
[02:38] I don't know if you've heard of him.
[02:40] Um he says oh uh that this is the OG modality.
[02:44] Um the correct modality for aligning AGI to our intents is very good voice AI engineering.
[02:51] That's that's that's a pretty bold statement there.
[02:54] And then like get here says we learn to speak before we learn to read.
[02:59] Right?
[02:59] Voice is the most natural interface that we have.
[03:01] So, how can we make it as natural as possible and human-like as possible?
[03:06] Quick show of hands, who who here has ever built a voice agent and deployed it to production?
[03:12] All right, I see a few hands.
[03:13] All right, so some of you know the challenges that you might face when doing this.
[03:19] Um others here who have not built one, maybe you have um seen a very good example of one.
[03:25] Have you ever used ChatGPT's advanced voice mode to have a conversation with ChatGPT?
[03:31] Okay, so that's that's cool.
[03:33] Segue here that OpenAI actually chose LiveKit to power that that functionality there.
[03:41] So, it's a very good example of a good use case or a good uh good example of a good voice agent.
[03:47] Um LiveKit is the open-source real-time framework and infrastructure for these human-like agents.
[03:55] Um agents that can hear, see, and speak to a person on the other end.
[04:00] And so, we're not just talking about voice, actually.
[04:03] Um there's there's multiple ways to interact with with agents, voice, video, data streams, uh even physical AI.
[04:11] We have a robotics division as well, but voice is the entry point voice is is the most natural modality, and voice is where there's a lot of challenges.
[04:18] So, we're going to focus just on voice today.
[04:21] So, in a traditional voice uh voice agent, you have what's called a cascaded pipeline.
[04:26] This is three models that work together.
[04:29] You've got your speech-to-text model, your LLM, and your text-to-speech model.
[04:33] And these all have to work in unison.
[04:36] And there are also these other models, these real-time models, speech-to-speech models, right?
[04:41] And they replace that pipeline.
[04:43] But that's a newer concept.
[04:44] Um it's it's getting better, but it's not quite there for every use case yet, right?
[04:49] But even then, we've got the traditional pipeline with three models, we've got the speech-to-speech models.
[04:54] There's so many things to choose from, so much complexity.
[04:58] Where do we even get started?
[04:59] And here's what we have to choose from.
[05:01] Hundreds of models, dozens of providers, and quick show of hands here, who has ever picked a model based on a leaderboard?
[05:09] Like you pick them whatever model's at the top of the leaderboard, that's what we're going to use, right?
[05:13] Well, we're going to talk about why that does not work for real-time agents.
[05:16] We can't just pick models off the top of a leaderboard.
[05:19] Um because leaderboards, they score these models, these components, but we're shipping a conversation.
[05:25] We need to be able to score them together.
[05:27] So, we're going to talk about how we can do that.
[05:29] So, let's go through a scenario here.
[05:30] Let's let's suppose that a caller uh wants to pay their bill.
[05:33] And they say, "I want to pay on the 3rd."
[05:36] But the speech-to-text model hears the 30th.
[05:40] And then the LLM is planning now for the 30th, and that goes into the system.
[05:45] The LLM doesn't know any better.
[05:46] It doesn't know that the speech detects model got it wrong.
[05:49] And so all of those errors flow downhill from there.
[05:53] And so the the the leaderboard, it can't show us this.
[05:56] It can't judge this because it just was looking at it individual models, right?
[06:01] And so also these leaderboards, they score these models based on clean audio.
[06:07] But your callers, they're calling on the speakerphone in traffic with kids in the backseat, right?
[06:13] So we need to be able to score our own agents based on real-world use cases and examples.
[06:20] So let's let's say that we fall into this leaderboard trap.
[06:23] We just pick a model.
[06:24] Um then the the first question that people ask is always, you know, which model should I pick?
[06:29] So they use a leaderboard, they pick a model, but that the actual question that we should be asking is which stack should I ship for my use case?
[06:39] And so that is what we're going to figure out.
[06:42] How do we answer that question?
[06:45] So before we answer that question, let's say that you have picked some models and you start measuring.
[06:53] How are these models performing?
[06:55] Let's uh say you're you're testing your TTS provider.
[06:59] And TTS provider on the dashboard, it's it's uh reporting time to first byte.
[07:04] And that looks really fast, 210 milliseconds time to first byte.
[07:07] That's super fast.
[07:08] Um but we've actually tested lots of popular providers that are shipping up to 3/4 of a second of silence after that first byte.
[07:18] So a byte does not equal audio.
[07:21] So what we're actually looking for is time to first audio.
[07:26] How much time does it take for our user to actually start hearing something?
[07:30] So time to first audio is what we should be measuring, not time to first byte.
[07:34] Um so imagine, you know, the payment went through and and and you you're asking, you know, um you know, did my payment go through?"
[07:42] And then you hear 2 seconds of silence.
[07:44] That's not very reassuring, right?
[07:46] So, we need to be able to measure this this gap that the caller hears.
[07:52] Now, the same story for lang- uh large language models.
[07:55] So, time to first token is what we get in text.
[08:01] It's it's a text-based measurement, but tokens do not equate to audio, either, right?
[08:07] Your TTS can't start speaking on a token.
[08:09] A TTS model text-to-speech model actually needs an entire sentence to start speaking.
[08:15] Um and so, that's what we need to measure is actually time to first sentence.
[08:20] Um because a an LLM that is very fast at producing tokens is not always the same LLM that produces the first sentence the fastest.
[08:29] So, time to first sentence is what we should be looking at.
[08:32] And then another thing that most leaderboards don't measure is these these agents, they need to do things.
[08:38] They're not just there to chat with us.
[08:39] They need to be able to do things on the back end.
[08:42] They have to look up a transaction, uh check for a payment status, submit a dispute, right?
[08:49] And so, these take time, 2 10 40 milliseconds.
[08:52] And and so, how do we handle that?
[08:54] How how do we hide 40 seconds or more of latency?
[09:00] Well, we can't.
[09:01] That The answer is we can't.
[09:02] But ask yourself, what would a human do in the situation?
[09:06] So, is a human just going to go silent while the system is doing something?
[09:11] No, they're going to say, you know, "Give give me a second.
[09:13] I'm looking that up.
[09:15] Um system's running a little slow, almost there."
[09:17] Or they're going to move on and they're going to ask another question while the system is doing something in the back end.
[09:21] So, that's that that's the human-like behavior, multitasking, right?
[09:25] We can do that.
[09:26] So, why can't agents multitask and do that?
[09:30] Well, that was a trick question because they can.
[09:32] They should.
[09:32] They should be able to.
[09:34] And this is the shape of that.
[09:36] Um this is the only code that I'm going to show you uh in this in this talk, but this is a normal tool call and the the one line is what makes the difference here.
[09:46] Um, this one line, this update, this async await, this awaiting this tool call is is the whole trick here and this is what's handing the mic back to the agent.
[09:56] This is allowing the conversation and the tool to stop sharing a thread.
[10:02] They're just they're separate now.
[10:04] And so um, this is what again makes the agent more human like because it's able to keep the conversation going as something is going on in the back end.
[10:13] So, don't don't take my word for it.
[10:14] I've got a quick demo here and we'll listen and see what it sounds like.
[10:20] Thank you for calling The Gilded Rose Hotel.
[10:22] How can I help you?
[10:23] Yeah, I'd like to book a room.
[10:26] What dates are you looking at?
[10:28] Uh, next week, Wednesday through Friday.
[10:32] Next Wednesday, July 30th to Friday, August 1st?
[10:36] That's right.
[10:38] How many guests will be staying?
[10:40] Two.
[10:42] I have queen two beds, king double queen, a suite or the penthouse?
[10:48] Would you prefer a city or ocean view?
[10:51] Ocean.
[10:54] May I have your name for the booking?
[10:56] Jesse Hall.
[10:58] Is that Jesse Hall?
[11:00] Yes.
[11:03] What's your email?
[11:04] me@me.com
[11:07] Is me@me.com correct?
[11:10] That's right.
[11:13] And your phone number?
[11:15] 555-123-4567
[11:20] Is that 555-123-4567?
[11:25] That's right.
[11:27] I'll need a credit card to hold the room.
[11:29] What's the the number?
[11:30] We'll stop here.
[11:32] Okay, so you notice here that the conversation kept going.
[11:36] There were no awkward pauses.
[11:38] Um I even interrupted it and it just kept rolling with it, right?
[11:42] It It didn't ever fall apart or feel unnatural.
[11:46] And we see here, you can see the the tool calls that are happening along the way and the things that it's doing.
[11:51] It's actually communicating with a back end and it's saving every uh piece of information that I'm giving it.
[11:57] But what it's doing is it's not waiting for the back end to say, "Okay, that's saved.
[12:01] Now you can move on to the next one."
[12:02] "Okay, that's saved.
[12:03] Move on to the next one."
[12:04] It's moving on to the next thing immediately.
[12:07] It's keeping the conversation going.
[12:09] And if for some reason a tool call would have come back with an error or something, it would have then corrected itself.
[12:14] It would have said, "Let me Let me verify this or that, whatever happened, right?"
[12:17] And worst case scenario, if it was having issues, it would uh it would actually uh um pass me over to a human in that in that uh scenario then, right?
[12:29] So, this is how our agents should operate.
[12:33] Now, there's one more um axis of this, and that is knowing the agent knowing when to speak and when to to listen, when to keep listening.
[12:42] And so, this is called turn detection.
[12:44] And so, LiveKit has a um a built-in turn detector that doesn't only listen to the transcript like most turn detectors do.
[12:53] It also listens to the audio.
[12:54] It listens to the rhythm, the pitch, the intonation.
[12:58] And we published the numbers on this.
[13:00] The the leading um um alternative still interrupts and cuts users off 9.9% of the time.
[13:08] And ours, 4.5% in our use case.
[13:12] We ran it against the our hotel agent and others.
[13:14] And that hap And that is what the numbers that we got for our use cases.
[13:18] This is all open source.
[13:19] You can actually run these e-vals and run this against your data set and see what numbers you get.
[13:24] See if you get a um uh better uh increased numbers here.
[13:27] And this is just one small piece of the puzzle again to make these voice agents more human-like.
[13:33] Them knowing when to stop talking or when to listen.
[13:37] For instance, if if I were trying to make a payment and I said I'd I'd like to pay 500 and $55.23, right?
[13:46] Most of the turn detectors would have stopped at 500 because they saw 500 on the transcript and they saw 2 second pause, I must be done, right?
[13:55] But ours would have heard my inflection, knew I wasn't done talking, and it would have waited for me.
[14:02] So, let's look now at that latency budget, that the gap.
[14:05] This is the gap between the caller's last syllable and the agent's first audible response.
[14:13] So, the turn detector decides when the caller is done, the language model begins preparing a response based on a provisional transcript, and then the TTS starts generating that audio as soon as it can.
[14:30] And so, every model that you choose, every function, every handoff, all of that is right here within this window, and this window has a finite number, a fixed window,
[14:41] a fixed size.
[14:42] And so, if you can stay roughly inside 1 and 1/2 seconds, the call still holds, the conversation holds.
[14:49] It's starting to feel a little off at 1 and 1/2 seconds.
[14:53] But if you can get down to 5 or 600 milliseconds between this the two turns, that's when it really starts to feel human-like.
[15:02] And so, that question before, how fast can I go?
[15:05] We just needed to get to go faster and faster.
[15:06] That's not the answer.
[15:08] The answer is how human-like of an agent can I fit within this latency window.
[15:15] So, again, latency, that's a budget, and and we know how how to fit into that, but how can we be as human-like as possible?
[15:23] That's the goal.
[15:24] So, if we add all this up, that that question, which model should I pick?
[15:28] We That was never something that we could answer because these scores, again, these leaderboards, they can't answer that because they're basing it off the models, but we are shipping a conversation.
[15:39] So, let's answer that actual question.
[15:40] How can we pick the models?
[15:41] Which models should we pick?
[15:44] So, this is what we have to do.
[15:46] We test the conversation.
[15:47] We take a complete working agent So, that hotel receptionist that you heard just a minute ago, we took that.
[15:53] It has a real back end.
[15:55] Um it's an actual It's not just a demo.
[15:57] It actually works.
[15:58] Uh and it's open source.
[15:59] You can check it out.
[16:00] So, we take that agent and we put simulated callers against it.
[16:04] And these callers can pursue goals.
[16:06] They can change their mind.
[16:08] They can interrupt.
[16:08] They can mumble.
[16:09] We can add background noise.
[16:11] We can make sure that this is a real-world scenario, and we run all of these simulated users against that agent.
[16:18] Um we can run text-based simulations, which just tests our business logic.
[16:23] Or we can also run audio-based simulations that tests the speech recognition capabilities.
[16:29] It tests turn-taking.
[16:31] The entire um the entire voice pipeline is what's tested there.
[16:36] And so, we take the scenarios, and the scenario should describe what the caller wants and what actions the agent should take, what the outcome should be.
[16:45] And then we score that.
[16:46] We score those outcomes.
[16:48] And so, in the hotel benchmark, you know, did the correct reservation hit the database?
[16:52] In a financial benchmark, you know, did the correct authorized amount get charged, right?
[16:57] And so, we we check these agents.
[17:00] We make sure that every all the outcomes are met.
[17:03] And then we don't score it based on just the completion alone, right?
[17:07] Because we know that agents can game completions, right?
[17:11] We have to make sure that they took the right steps.
[17:13] Did they take 10 steps when they only should have taken two steps, right?
[17:17] Did they hallucinate a function call?
[17:19] These are the things that we have to test, the entire conversation there.
[17:23] And then, we swap the stack out.
[17:25] We We have the same agent, the same scenarios, but we change the models.
[17:31] And then, we run that loop again.
[17:33] The identical input Every model sees identical input byte for byte.
[17:38] We swap the models, we run the loop.
[17:40] We swap the models again, we run the loop continuously over and over, and we are building at that point a benchmark, a personalized benchmark, for our use case, for our agent.
[17:51] So, there are as many benchmarks now as there are agents.
[17:53] You run this against yours, and you find out which models work best for your situation.
[17:58] That is the only way to find out which model is best for you.
[18:05] Now, this is how you catch surprises.
[18:07] Um a model that that gets the answer for you the quickest is the right model for you, right?
[18:15] And surprisingly, that model may not be the model that that's at the top of the leaderboard, right?
[18:20] But it's best for your use case.
[18:22] And again, these benchmarks, the simulations, all this this methodology, it's all open source.
[18:28] So, you can You can check it out for yourselves, you can run it against your agent, and see how yours performs.
[18:35] And so, here is the results of that benchmark on our hotel agent.
[18:39] This is the LLM that ranked the highest for our hotel receptionist agent.
[18:44] That's Gemma 4 31B on LiveKit inference.
[18:47] You notice on the second line, the same model, Gemma 4 31B on another provider, was slower.
[18:53] So, LiveKit optimizes its inference serving for voice workloads.
[18:59] And LiveKit is model agnostic, so you can use any model that you want.
[19:02] So, again, you plug those models in, and you figure out which one is work it will work best for your use case.
[19:10] Now, if you remember one thing from this this entire talk, remember this that a voice agent isn't just a chatbot with a speaker attached.
[19:18] It's There's so much more complexity there.
[19:21] It's a real-time system and that needs to respond like a person.
[19:26] We need to make it as human-like as possible.
[19:28] And and and when we evaluate it from that viewpoint, we will be able to to build that.
[19:35] And so, here's a checklist.
[19:36] You can take a take a photo here.
[19:37] We need to set our latency budget that the caller hears.
[19:40] Measure time to first audio, time to first sentence.
[19:44] Confirm critical dates.
[19:47] Anything that's financial money, we need to make sure that we have those those checks in place.
[19:52] Verify identity before an account is accessed.
[19:55] Keep listening while slow tools run.
[19:58] Keep the conversation going.
[20:00] And then run the complete conversation repeatedly through simulations.
[20:04] Test the final state, privacy safeguards, unnecessary tool calls, turn-taking.
[20:09] All these things and then when you find these failures, you turn those into regression scenarios and you keep testing over and over again.
[20:16] So, start with one one of those workflows or failures that you care about.
[20:21] Keep Keep iterating then.
[20:23] Run it.
[20:24] Listen to the conversation.
[20:26] Check what happened on the back end.
[20:27] At some point in the very near future, I hope, that we all call one of these voice agents and we talk over it.
[20:35] We confuse it and it just rolls with it and it's and it's a very human-like experience.
[20:41] That's That's our goal, but here's what I need from each and every one of you.
[20:44] I need you to go build that.
[20:47] Thank you.
[20:54] Woo!