How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit

AI Engineer · 20 min · 311 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Um all right, we're going to talk about voice agents.
00:16And most of the agents that we deal with on a day-to-day basis, um they they do things in the background, right?
00:24They they write code for us, they do research, we give them something to do, a task, and we walk away, we come back, and we don't really care if they take an extra 10 seconds, an extra 10 minutes, maybe sometimes 10 hours,
00:38right?
00:39Um and we expect that.
00:40But at some point in the last year or so, you have probably interacted with another kind of agent, a real-time agent.
00:48Maybe when you called your pharmacy, maybe uh when you called the airline.
00:53I talked to one just the other day when I called a car dealership.
00:56And at some point in that conversation, you did the the most human-like thing possible.
01:01You spoke over it.
01:03You interrupted it.
01:04You confused it.
01:06And the entire conversation fell apart, it just didn't feel natural.
01:09And then you did what I do every single time, operator, hit zero, like get me to a human, right?
01:16So, how do we fix that?
01:17How do we make these agents as human-like as possible?
01:21Um and so, like in in a um a finance space, what if your caller was calling to schedule a payment, and they changed the the payment amount midway through, they changed the date,
01:35um and and the agent didn't keep up with it, right?
01:38That is where we start having issues, especially in the finance space.
01:42The the the the risk is much higher, right?
01:45So, how do we fix that?
01:47So, we're going to talk about how can we build human-like voice agents for financial services.
01:53And when you think about building a good voice agent, one of the first things that comes to mind is latency.
01:59We need it to be as fast as possible, right?
02:01Well, latency is part of the equation.
02:03Latency has a finite budget, and we'll talk about what that is.
02:07Um but the keyword here is human-like.
02:10How do we make these voice agents as human-like as we can?
02:14That's the goal.
02:15So, there's quite a few people that are talking about voice lately.
02:19Um some of these might look familiar.
02:21We've got this this guy Sam.
02:23Um he's excited about voice models.
02:26He says uh that they're starting to change the way they that we interface with AI.
02:33Isn't that true?
02:33And then this this um weird name here in the bottom right, Swyx.
02:38I don't know if you've heard of him.
02:40Um he says oh uh that this is the OG modality.
02:44Um the correct modality for aligning AGI to our intents is very good voice AI engineering.
02:51That's that's that's a pretty bold statement there.
02:54And then like get here says we learn to speak before we learn to read.
02:59Right?
02:59Voice is the most natural interface that we have.
03:01So, how can we make it as natural as possible and human-like as possible?
03:06Quick show of hands, who who here has ever built a voice agent and deployed it to production?
03:12All right, I see a few hands.
03:13All right, so some of you know the challenges that you might face when doing this.
03:19Um others here who have not built one, maybe you have um seen a very good example of one.
03:25Have you ever used ChatGPT's advanced voice mode to have a conversation with ChatGPT?
03:31Okay, so that's that's cool.
03:33Segue here that OpenAI actually chose LiveKit to power that that functionality there.
03:41So, it's a very good example of a good use case or a good uh good example of a good voice agent.
03:47Um LiveKit is the open-source real-time framework and infrastructure for these human-like agents.
03:55Um agents that can hear, see, and speak to a person on the other end.
04:00And so, we're not just talking about voice, actually.
04:03Um there's there's multiple ways to interact with with agents, voice, video, data streams, uh even physical AI.
04:11We have a robotics division as well, but voice is the entry point voice is is the most natural modality, and voice is where there's a lot of challenges.
04:18So, we're going to focus just on voice today.
04:21So, in a traditional voice uh voice agent, you have what's called a cascaded pipeline.
04:26This is three models that work together.
04:29You've got your speech-to-text model, your LLM, and your text-to-speech model.
04:33And these all have to work in unison.
04:36And there are also these other models, these real-time models, speech-to-speech models, right?
04:41And they replace that pipeline.
04:43But that's a newer concept.
04:44Um it's it's getting better, but it's not quite there for every use case yet, right?
04:49But even then, we've got the traditional pipeline with three models, we've got the speech-to-speech models.
04:54There's so many things to choose from, so much complexity.
04:58Where do we even get started?
04:59And here's what we have to choose from.
05:01Hundreds of models, dozens of providers, and quick show of hands here, who has ever picked a model based on a leaderboard?
05:09Like you pick them whatever model's at the top of the leaderboard, that's what we're going to use, right?
05:13Well, we're going to talk about why that does not work for real-time agents.
05:16We can't just pick models off the top of a leaderboard.
05:19Um because leaderboards, they score these models, these components, but we're shipping a conversation.
05:25We need to be able to score them together.
05:27So, we're going to talk about how we can do that.
05:29So, let's go through a scenario here.
05:30Let's let's suppose that a caller uh wants to pay their bill.
05:33And they say, "I want to pay on the 3rd."
05:36But the speech-to-text model hears the 30th.
05:40And then the LLM is planning now for the 30th, and that goes into the system.
05:45The LLM doesn't know any better.
05:46It doesn't know that the speech detects model got it wrong.
05:49And so all of those errors flow downhill from there.
05:53And so the the the leaderboard, it can't show us this.
05:56It can't judge this because it just was looking at it individual models, right?
06:01And so also these leaderboards, they score these models based on clean audio.
06:07But your callers, they're calling on the speakerphone in traffic with kids in the backseat, right?
06:13So we need to be able to score our own agents based on real-world use cases and examples.
06:20So let's let's say that we fall into this leaderboard trap.
06:23We just pick a model.
06:24Um then the the first question that people ask is always, you know, which model should I pick?
06:29So they use a leaderboard, they pick a model, but that the actual question that we should be asking is which stack should I ship for my use case?
06:39And so that is what we're going to figure out.
06:42How do we answer that question?
06:45So before we answer that question, let's say that you have picked some models and you start measuring.
06:53How are these models performing?
06:55Let's uh say you're you're testing your TTS provider.
06:59And TTS provider on the dashboard, it's it's uh reporting time to first byte.
07:04And that looks really fast, 210 milliseconds time to first byte.
07:07That's super fast.
07:08Um but we've actually tested lots of popular providers that are shipping up to 3/4 of a second of silence after that first byte.
07:18So a byte does not equal audio.
07:21So what we're actually looking for is time to first audio.
07:26How much time does it take for our user to actually start hearing something?
07:30So time to first audio is what we should be measuring, not time to first byte.
07:34Um so imagine, you know, the payment went through and and and you you're asking, you know, um you know, did my payment go through?"
07:42And then you hear 2 seconds of silence.
07:44That's not very reassuring, right?
07:46So, we need to be able to measure this this gap that the caller hears.
07:52Now, the same story for lang- uh large language models.
07:55So, time to first token is what we get in text.
08:01It's it's a text-based measurement, but tokens do not equate to audio, either, right?
08:07Your TTS can't start speaking on a token.
08:09A TTS model text-to-speech model actually needs an entire sentence to start speaking.
08:15Um and so, that's what we need to measure is actually time to first sentence.
08:20Um because a an LLM that is very fast at producing tokens is not always the same LLM that produces the first sentence the fastest.
08:29So, time to first sentence is what we should be looking at.
08:32And then another thing that most leaderboards don't measure is these these agents, they need to do things.
08:38They're not just there to chat with us.
08:39They need to be able to do things on the back end.
08:42They have to look up a transaction, uh check for a payment status, submit a dispute, right?
08:49And so, these take time, 2 10 40 milliseconds.
08:52And and so, how do we handle that?
08:54How how do we hide 40 seconds or more of latency?
09:00Well, we can't.
09:01That The answer is we can't.
09:02But ask yourself, what would a human do in the situation?
09:06So, is a human just going to go silent while the system is doing something?
09:11No, they're going to say, you know, "Give give me a second.
09:13I'm looking that up.
09:15Um system's running a little slow, almost there."
09:17Or they're going to move on and they're going to ask another question while the system is doing something in the back end.
09:21So, that's that that's the human-like behavior, multitasking, right?
09:25We can do that.
09:26So, why can't agents multitask and do that?
09:30Well, that was a trick question because they can.
09:32They should.
09:32They should be able to.
09:34And this is the shape of that.
09:36Um this is the only code that I'm going to show you uh in this in this talk, but this is a normal tool call and the the one line is what makes the difference here.
09:46Um, this one line, this update, this async await, this awaiting this tool call is is the whole trick here and this is what's handing the mic back to the agent.
09:56This is allowing the conversation and the tool to stop sharing a thread.
10:02They're just they're separate now.
10:04And so um, this is what again makes the agent more human like because it's able to keep the conversation going as something is going on in the back end.
10:13So, don't don't take my word for it.
10:14I've got a quick demo here and we'll listen and see what it sounds like.
10:20Thank you for calling The Gilded Rose Hotel.
10:22How can I help you?
10:23Yeah, I'd like to book a room.
10:26What dates are you looking at?
10:28Uh, next week, Wednesday through Friday.
10:32Next Wednesday, July 30th to Friday, August 1st?
10:36That's right.
10:38How many guests will be staying?
10:40Two.
10:42I have queen two beds, king double queen, a suite or the penthouse?
10:48Would you prefer a city or ocean view?
10:51Ocean.
10:54May I have your name for the booking?
10:56Jesse Hall.
10:58Is that Jesse Hall?
11:00Yes.
11:03What's your email?
11:04me@me.com
11:07Is me@me.com correct?
11:10That's right.
11:13And your phone number?
11:15555-123-4567
11:20Is that 555-123-4567?
11:25That's right.
11:27I'll need a credit card to hold the room.
11:29What's the the number?
11:30We'll stop here.
11:32Okay, so you notice here that the conversation kept going.
11:36There were no awkward pauses.
11:38Um I even interrupted it and it just kept rolling with it, right?
11:42It It didn't ever fall apart or feel unnatural.
11:46And we see here, you can see the the tool calls that are happening along the way and the things that it's doing.
11:51It's actually communicating with a back end and it's saving every uh piece of information that I'm giving it.
11:57But what it's doing is it's not waiting for the back end to say, "Okay, that's saved.
12:01Now you can move on to the next one."
12:02"Okay, that's saved.
12:03Move on to the next one."
12:04It's moving on to the next thing immediately.
12:07It's keeping the conversation going.
12:09And if for some reason a tool call would have come back with an error or something, it would have then corrected itself.
12:14It would have said, "Let me Let me verify this or that, whatever happened, right?"
12:17And worst case scenario, if it was having issues, it would uh it would actually uh um pass me over to a human in that in that uh scenario then, right?
12:29So, this is how our agents should operate.
12:33Now, there's one more um axis of this, and that is knowing the agent knowing when to speak and when to to listen, when to keep listening.
12:42And so, this is called turn detection.
12:44And so, LiveKit has a um a built-in turn detector that doesn't only listen to the transcript like most turn detectors do.
12:53It also listens to the audio.
12:54It listens to the rhythm, the pitch, the intonation.
12:58And we published the numbers on this.
13:00The the leading um um alternative still interrupts and cuts users off 9.9% of the time.
13:08And ours, 4.5% in our use case.
13:12We ran it against the our hotel agent and others.
13:14And that hap And that is what the numbers that we got for our use cases.
13:18This is all open source.
13:19You can actually run these e-vals and run this against your data set and see what numbers you get.
13:24See if you get a um uh better uh increased numbers here.
13:27And this is just one small piece of the puzzle again to make these voice agents more human-like.
13:33Them knowing when to stop talking or when to listen.
13:37For instance, if if I were trying to make a payment and I said I'd I'd like to pay 500 and $55.23, right?
13:46Most of the turn detectors would have stopped at 500 because they saw 500 on the transcript and they saw 2 second pause, I must be done, right?
13:55But ours would have heard my inflection, knew I wasn't done talking, and it would have waited for me.
14:02So, let's look now at that latency budget, that the gap.
14:05This is the gap between the caller's last syllable and the agent's first audible response.
14:13So, the turn detector decides when the caller is done, the language model begins preparing a response based on a provisional transcript, and then the TTS starts generating that audio as soon as it can.
14:30And so, every model that you choose, every function, every handoff, all of that is right here within this window, and this window has a finite number, a fixed window,
14:41a fixed size.
14:42And so, if you can stay roughly inside 1 and 1/2 seconds, the call still holds, the conversation holds.
14:49It's starting to feel a little off at 1 and 1/2 seconds.
14:53But if you can get down to 5 or 600 milliseconds between this the two turns, that's when it really starts to feel human-like.
15:02And so, that question before, how fast can I go?
15:05We just needed to get to go faster and faster.
15:06That's not the answer.
15:08The answer is how human-like of an agent can I fit within this latency window.
15:15So, again, latency, that's a budget, and and we know how how to fit into that, but how can we be as human-like as possible?
15:23That's the goal.
15:24So, if we add all this up, that that question, which model should I pick?
15:28We That was never something that we could answer because these scores, again, these leaderboards, they can't answer that because they're basing it off the models, but we are shipping a conversation.
15:39So, let's answer that actual question.
15:40How can we pick the models?
15:41Which models should we pick?
15:44So, this is what we have to do.
15:46We test the conversation.
15:47We take a complete working agent So, that hotel receptionist that you heard just a minute ago, we took that.
15:53It has a real back end.
15:55Um it's an actual It's not just a demo.
15:57It actually works.
15:58Uh and it's open source.
15:59You can check it out.
16:00So, we take that agent and we put simulated callers against it.
16:04And these callers can pursue goals.
16:06They can change their mind.
16:08They can interrupt.
16:08They can mumble.
16:09We can add background noise.
16:11We can make sure that this is a real-world scenario, and we run all of these simulated users against that agent.
16:18Um we can run text-based simulations, which just tests our business logic.
16:23Or we can also run audio-based simulations that tests the speech recognition capabilities.
16:29It tests turn-taking.
16:31The entire um the entire voice pipeline is what's tested there.
16:36And so, we take the scenarios, and the scenario should describe what the caller wants and what actions the agent should take, what the outcome should be.
16:45And then we score that.
16:46We score those outcomes.
16:48And so, in the hotel benchmark, you know, did the correct reservation hit the database?
16:52In a financial benchmark, you know, did the correct authorized amount get charged, right?
16:57And so, we we check these agents.
17:00We make sure that every all the outcomes are met.
17:03And then we don't score it based on just the completion alone, right?
17:07Because we know that agents can game completions, right?
17:11We have to make sure that they took the right steps.
17:13Did they take 10 steps when they only should have taken two steps, right?
17:17Did they hallucinate a function call?
17:19These are the things that we have to test, the entire conversation there.
17:23And then, we swap the stack out.
17:25We We have the same agent, the same scenarios, but we change the models.
17:31And then, we run that loop again.
17:33The identical input Every model sees identical input byte for byte.
17:38We swap the models, we run the loop.
17:40We swap the models again, we run the loop continuously over and over, and we are building at that point a benchmark, a personalized benchmark, for our use case, for our agent.
17:51So, there are as many benchmarks now as there are agents.
17:53You run this against yours, and you find out which models work best for your situation.
17:58That is the only way to find out which model is best for you.
18:05Now, this is how you catch surprises.
18:07Um a model that that gets the answer for you the quickest is the right model for you, right?
18:15And surprisingly, that model may not be the model that that's at the top of the leaderboard, right?
18:20But it's best for your use case.
18:22And again, these benchmarks, the simulations, all this this methodology, it's all open source.
18:28So, you can You can check it out for yourselves, you can run it against your agent, and see how yours performs.
18:35And so, here is the results of that benchmark on our hotel agent.
18:39This is the LLM that ranked the highest for our hotel receptionist agent.
18:44That's Gemma 4 31B on LiveKit inference.
18:47You notice on the second line, the same model, Gemma 4 31B on another provider, was slower.
18:53So, LiveKit optimizes its inference serving for voice workloads.
18:59And LiveKit is model agnostic, so you can use any model that you want.
19:02So, again, you plug those models in, and you figure out which one is work it will work best for your use case.
19:10Now, if you remember one thing from this this entire talk, remember this that a voice agent isn't just a chatbot with a speaker attached.
19:18It's There's so much more complexity there.
19:21It's a real-time system and that needs to respond like a person.
19:26We need to make it as human-like as possible.
19:28And and and when we evaluate it from that viewpoint, we will be able to to build that.
19:35And so, here's a checklist.
19:36You can take a take a photo here.
19:37We need to set our latency budget that the caller hears.
19:40Measure time to first audio, time to first sentence.
19:44Confirm critical dates.
19:47Anything that's financial money, we need to make sure that we have those those checks in place.
19:52Verify identity before an account is accessed.
19:55Keep listening while slow tools run.
19:58Keep the conversation going.
20:00And then run the complete conversation repeatedly through simulations.
20:04Test the final state, privacy safeguards, unnecessary tool calls, turn-taking.
20:09All these things and then when you find these failures, you turn those into regression scenarios and you keep testing over and over again.
20:16So, start with one one of those workflows or failures that you care about.
20:21Keep Keep iterating then.
20:23Run it.
20:24Listen to the conversation.
20:26Check what happened on the back end.
20:27At some point in the very near future, I hope, that we all call one of these voice agents and we talk over it.
20:35We confuse it and it just rolls with it and it's and it's a very human-like experience.
20:41That's That's our goal, but here's what I need from each and every one of you.
20:44I need you to go build that.
20:47Thank you.
20:54Woo!