Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

AI Engineer · 13 min · 136 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12My name is Bo.
00:13I'm going to be here presenting real-time voice agents with Frontier Intelligence.
00:19Effectively, going to be talking a little bit about how we at Xnor.ai architected our voice agent harness to get real-time voice with the Frontier level of intelligence that we need.
00:32Okay.
00:33So, before I start, I think I wanted to kind of draw some parallels about why we decided to go with cascaded voice agents and especially kind of comparing that to self-driving cars which I was working in before.
00:49So, to me, cascaded voice agents makes a lot of sense when you view it in lens of kind of breaking it down into perception which is for self-driving cars, it's you know,
01:00the bounding boxes, the camera, the lidar.
01:03For voice, it's going to be the transcription.
01:06Basically, effectively turning these like signals from the real world into elements of data that the language model or whatever brain you're working on can process.
01:19Second one is the planning step which is pretty straightforward.
01:23This is where the language model will take in the outputs from the perception stage and produce the outputs that you want to produce out back and out into the real world.
01:34And finally, there's the controls layer where on self-driving, you'd be taking the trajectory that the planner would output and kind of turn it into the real controls to kind of build like drive the car.
01:48Here, we're turning the text into audio that we use to express our voice agent's thoughts.
01:56And um yeah, so here I'm going to be like going to diving into each one of these elements and we've made a few kind of interesting tricks on each of these areas to
02:08improve the speed of our voice agents without sacrificing the intelligence.
02:13So, the first one is going to be uh the transcriber layer.
02:17So, we came up with this concept called like the streaming speculative transcriber where effectively we are layering a fast streaming transcriber like Flux on top of or kind of below a
02:29uh scribe V2 or a accurate batch transcription which kind of takes in more context.
02:35It's a little bit slower, but it will give you more accurate detections.
02:39So, we're going to walk through a scenario.
02:41So, in this in this case the agent just asked, you know, providing can you provide your name and date of birth and the user is going to say this and we'll see how that plays out
02:50um timing-wise.
02:51So, first we're going to get, you know, the short detection.
02:55Um we'll get it from we'll get it from the streaming layer.
02:58The accurate layer uh the corrective layer is not going to fire because it's the same text.
03:04Um we're going to get some more streaming text detections and in this case the corrective layer is actually canceled because we got new um new text.
03:14So, you know, more context, more audio is going to beat the old accurate one.
03:21And here's where kind of the first correction comes in.
03:23So, because the scribe V2 layer understands, you know, the the context of the question, it's able to understand that this is talking about name and this is a date of birth.
03:34Then a couple more detections, these are just punctuation, we don't care.
03:37And so, in the end we kind of release this text over to the agent.
03:44And moving on um to the language model layer.
03:47So, here since we're kind of using these slow but intelligent LLMs, we really want to reduce the number of round trips and the thing that causes us to do a lot of inferences is tool calling.
04:00So, one way to get rid of that is by having background agents do the tool calling for you and kind of um push the tools back into the context of the main agent so that it thinks it made the tool call, but
04:15um but it it it really didn't.
04:18So, uh so, we remember from like detections from before.
04:22So, well, what happened is each one of these detections is going to trigger a um an early kind of generation of the agent and we but we won't actually
04:35emit this out until we're confirming that the user has finished speaking.
04:39So, in this case, the user says, "Sure."
04:42The agent kind of knows that the user is about to say something else.
04:45Our background tool calling here, which is going to be helping us find figure out the name and the date of birth from the user detection, is not firing.
04:53So, nothing much there.
04:55Um the next instant detection comes in.
04:58It says that, you know, still not really a name.
05:02Um our agent kind of plays along and continues there.
05:06Now, kind of a more more context come comes back.
05:09The agent kind of feels like there should be a name.
05:11It's going to ask to spell it out because it's probably thinking there's some transcription error here.
05:16Still no name or date of birth.
05:19And then finally, this you remember this is kind of our corrected um final instant detection from the transcriber from the Scribe V2.
05:27Um here, our eager kind of agent generation that was made without any tool calls is going to get canceled because the background agent finally is able to find the name and date of birth it's looking for.
05:39So, it's going to retrigger and now the the agent actually has the context it needs.
05:44Um and you see here, it's kind of we're doing it the tool call here is a little bit um um some intelligence there.
05:50We're going to be like, you know, correcting mis-transcriptions of name, and doing some like phonetic matching here.
05:57Um and yeah, and then we'll kind of once we've understood that this is the end of the user utterance, we'll kind of emit it out.
06:04So, pretty standard.
06:06Okay, and then the next layer here is going to be text-to-speech.
06:09So, with text-to-speech the goal is to kind of take what the agent said, and the agent's going to be emitting this in a streaming fashion.
06:17So, we're going to need to um produce audio as quickly as possible.
06:21And ideally, what you can do is before the agent has even finished generating the full text you can have the audio play, so it's kind of hiding the latency of finishing the generation.
06:34So, um I'm going to kind of play the streaming um the stream the streaming uh agent output now.
06:41So, starts with you.
06:43And yeah, actually before I uh further, there's this new concept that we're introducing here called the prefix cache.
06:49So, the prefix cache is going to be looking at the um agent stream, and seeing if we already have generated audio for that sequence of words um from like a prior generation, or maybe like the same generation
07:05um in this uh in this call as as well.
07:10So, um it sees the word you.
07:12Uh we for this prefix cache, we're going to be, you know, we don't want to like immediately hit on every single word.
07:19We're going to be waiting for a little bit more words.
07:22Um so, after three words, the prefix cache gets our first hit.
07:27And um over here on the right, this is kind of our text-to-speech standard provider, you know, Cartesia is a text-to-speech engine with web socket support.
07:36So, we're we're piping the agent through the the cache, and also piping it through web socket.
07:44Um more tokens come in, more cache, more sending through web socket.
07:49Not much to say here.
07:51And okay, so now we get our first uh kind of first unique thing, which is we found the token that actually causes a cache miss.
08:01And it makes sense.
08:02If we're kind of caching previous generations, um you said your name is is a pretty common thing, but once you we add in the name, suddenly we're that's that's going to result in the cache miss.
08:12At this point, we're actually going to yield out our cached audio.
08:15So, you said your name is is going to be um emitted as the rest of the streaming text is coming back.
08:22So, at this point, the user hears the agent and user doesn't really know what's going on.
08:27They just looks like really fast response times to them.
08:31Um and now the kind of remaining text flows through.
08:35And at this point, we've already emitted from the cache.
08:38The cache has done its job.
08:40Um the rest we can kind of throw into Cartesia.
08:43And here's kind of the trick where Cartesia has seen the entire transcript up to this point.
08:53It to to Cartesia, like it doesn't know about the existence of this prefix cache.
08:58It's just going to generate this full sentence with, you know, standard natural prosody.
09:03But, what we do is when the generation comes back, since we've already played the audio here, we can actually suppress the audio from Cartesia here and just play out the remaining stuff.
09:14So, the user, there's might be a tiny bit of a hiccup.
09:19You know, I'll play some audio later and you'll know that you probably won't be able to notice.
09:24And effectively, we're kind of taking this and or taking this part and just emitting it directly after this audio has finished playing.
09:34So, it looks seamless to the user.
09:37And yeah, so I guess at this point kind of this is a lot of technical details.
09:42I'll I mean the best way is just kind of play a call and see see this in action.
09:49So here I recorded a call and let's just hear what happens.
09:55This is Bo OBGYN Elise speaking.
10:00Hi, I think I might be pregnant and I want to schedule an ultrasound to confirm.
10:06That's exciting news.
10:08Can you provide your name and date of birth?
10:12Sure, it's Elise Trial.
10:14Date of birth 2303.
10:21Thank you.
10:22Are you a new patient with us?
10:26Yes.
10:29Welcome to Bo OBGYN.
10:31Can I text you a link to upload your insurance information?
10:36Sure.
10:39Okay, sent.
10:40Let me know when you've received it.
10:48I've received your insurance information.
10:51The earliest appointment I have is Thursday, July 2nd at 10:00 a.m.
10:55Actually, give me a second.
10:57Let me check my calendar.
11:01Sure, take your time.
11:04Um do you have something for next week?
11:09For next week, I can offer an ultrasound with Dr. Avery Stone at North Clinic on Tuesday, July 7th at 2:00 p.m. or 3:00 p.m. Do either of those work for you?
11:20Yeah, 2:00 p.m. works.
11:23Great.
11:24Your appointment has been booked.
11:26We look forward to seeing you then.
11:29Thanks.
11:30Bye-bye.
11:32All right.
11:33Yeah, that's pretty much it.
11:34Um Yeah, you can kind of see our all this like uh streaming and you know, a lot of things are happening in the background and and you know, this is what really makes like voice agents interesting.
11:48And um there's a lot of effort that can be done in the harness to really kind of get a um natural conversation, which is what we're after.
11:59Uh okay.
12:00Yeah, so I guess briefly, you know, in the last part, I want to just talk a little bit about Elise.
12:05So I think Elise, you know, our headquarters are in New York and kind of we're trying to expand our presence here in the Bay Area.
12:12Um we I think it's maybe like a different style of company that I think people are like uh think of when they think about AI startups in San Francisco.
12:23Where we're actually very focused on um just like helping people and helping people where they need it, like kind of the life's most critical areas.
12:33We work on housing, health care and uh we're doing really well and you know, here's there's a link here to um you kind of join our team and there's going to we're going to be
12:46uh posting a lot on Twitter, so you can follow us at EliseAI as well.
12:50Um yeah, that's that's it.
12:53[applause]