WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=MBHOH1NmDqc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/MBHOH1NmDqc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.720 --> 00:00:13.920
My name is Bo.

s3
00:00:13.920 --> 00:00:19.040
I'm going to be here presenting real-time voice agents with Frontier Intelligence.

s4
00:00:19.040 --> 00:00:32.640
Effectively, going to be talking a little bit about how we at Xnor.ai architected our voice agent harness to get real-time voice with the Frontier level of intelligence that we need.

s5
00:00:32.960 --> 00:00:33.320
Okay.

s6
00:00:33.320 --> 00:00:49.040
So, before I start, I think I wanted to kind of draw some parallels about why we decided to go with cascaded voice agents and especially kind of comparing that to self-driving cars which I was working in before.

s7
00:00:49.040 --> 00:01:00.480
So, to me, cascaded voice agents makes a lot of sense when you view it in lens of kind of breaking it down into perception which is for self-driving cars, it's you know,

s8
00:01:00.480 --> 00:01:03.720
the bounding boxes, the camera, the lidar.

s9
00:01:03.720 --> 00:01:06.920
For voice, it's going to be the transcription.

s10
00:01:06.920 --> 00:01:19.200
Basically, effectively turning these like signals from the real world into elements of data that the language model or whatever brain you're working on can process.

s11
00:01:19.200 --> 00:01:23.360
Second one is the planning step which is pretty straightforward.

s12
00:01:23.360 --> 00:01:34.480
This is where the language model will take in the outputs from the perception stage and produce the outputs that you want to produce out back and out into the real world.

s13
00:01:34.480 --> 00:01:48.360
And finally, there's the controls layer where on self-driving, you'd be taking the trajectory that the planner would output and kind of turn it into the real controls to kind of build like drive the car.

s14
00:01:48.360 --> 00:01:56.200
Here, we're turning the text into audio that we use to express our voice agent's thoughts.

s15
00:01:56.760 --> 00:02:08.119
And um yeah, so here I'm going to be like going to diving into each one of these elements and we've made a few kind of interesting tricks on each of these areas to

s16
00:02:08.119 --> 00:02:13.320
improve the speed of our voice agents without sacrificing the intelligence.

s17
00:02:13.520 --> 00:02:17.360
So, the first one is going to be uh the transcriber layer.

s18
00:02:17.360 --> 00:02:29.520
So, we came up with this concept called like the streaming speculative transcriber where effectively we are layering a fast streaming transcriber like Flux on top of or kind of below a

s19
00:02:29.520 --> 00:02:35.160
uh scribe V2 or a accurate batch transcription which kind of takes in more context.

s20
00:02:35.160 --> 00:02:39.080
It's a little bit slower, but it will give you more accurate detections.

s21
00:02:39.080 --> 00:02:41.040
So, we're going to walk through a scenario.

s22
00:02:41.040 --> 00:02:50.640
So, in this in this case the agent just asked, you know, providing can you provide your name and date of birth and the user is going to say this and we'll see how that plays out

s23
00:02:50.640 --> 00:02:51.800
um timing-wise.

s24
00:02:51.800 --> 00:02:55.440
So, first we're going to get, you know, the short detection.

s25
00:02:55.440 --> 00:02:58.959
Um we'll get it from we'll get it from the streaming layer.

s26
00:02:58.959 --> 00:03:04.920
The accurate layer uh the corrective layer is not going to fire because it's the same text.

s27
00:03:04.920 --> 00:03:14.959
Um we're going to get some more streaming text detections and in this case the corrective layer is actually canceled because we got new um new text.

s28
00:03:14.959 --> 00:03:21.480
So, you know, more context, more audio is going to beat the old accurate one.

s29
00:03:21.480 --> 00:03:23.920
And here's where kind of the first correction comes in.

s30
00:03:23.920 --> 00:03:34.320
So, because the scribe V2 layer understands, you know, the the context of the question, it's able to understand that this is talking about name and this is a date of birth.

s31
00:03:34.320 --> 00:03:37.800
Then a couple more detections, these are just punctuation, we don't care.

s32
00:03:37.800 --> 00:03:44.360
And so, in the end we kind of release this text over to the agent.

s33
00:03:44.360 --> 00:03:47.800
And moving on um to the language model layer.

s34
00:03:47.800 --> 00:04:00.480
So, here since we're kind of using these slow but intelligent LLMs, we really want to reduce the number of round trips and the thing that causes us to do a lot of inferences is tool calling.

s35
00:04:00.480 --> 00:04:15.400
So, one way to get rid of that is by having background agents do the tool calling for you and kind of um push the tools back into the context of the main agent so that it thinks it made the tool call, but

s36
00:04:15.400 --> 00:04:18.000
um but it it it really didn't.

s37
00:04:18.000 --> 00:04:22.520
So, uh so, we remember from like detections from before.

s38
00:04:22.520 --> 00:04:35.200
So, well, what happened is each one of these detections is going to trigger a um an early kind of generation of the agent and we but we won't actually

s39
00:04:35.200 --> 00:04:39.680
emit this out until we're confirming that the user has finished speaking.

s40
00:04:39.680 --> 00:04:42.480
So, in this case, the user says, "Sure."

s41
00:04:42.480 --> 00:04:45.280
The agent kind of knows that the user is about to say something else.

s42
00:04:45.280 --> 00:04:53.000
Our background tool calling here, which is going to be helping us find figure out the name and the date of birth from the user detection, is not firing.

s43
00:04:53.000 --> 00:04:55.360
So, nothing much there.

s44
00:04:55.360 --> 00:04:58.120
Um the next instant detection comes in.

s45
00:04:58.120 --> 00:05:02.000
It says that, you know, still not really a name.

s46
00:05:02.000 --> 00:05:06.240
Um our agent kind of plays along and continues there.

s47
00:05:06.240 --> 00:05:09.240
Now, kind of a more more context come comes back.

s48
00:05:09.240 --> 00:05:11.680
The agent kind of feels like there should be a name.

s49
00:05:11.680 --> 00:05:16.760
It's going to ask to spell it out because it's probably thinking there's some transcription error here.

s50
00:05:16.760 --> 00:05:19.120
Still no name or date of birth.

s51
00:05:19.120 --> 00:05:27.000
And then finally, this you remember this is kind of our corrected um final instant detection from the transcriber from the Scribe V2.

s52
00:05:27.000 --> 00:05:39.080
Um here, our eager kind of agent generation that was made without any tool calls is going to get canceled because the background agent finally is able to find the name and date of birth it's looking for.

s53
00:05:39.080 --> 00:05:44.800
So, it's going to retrigger and now the the agent actually has the context it needs.

s54
00:05:44.800 --> 00:05:50.800
Um and you see here, it's kind of we're doing it the tool call here is a little bit um um some intelligence there.

s55
00:05:50.800 --> 00:05:57.520
We're going to be like, you know, correcting mis-transcriptions of name, and doing some like phonetic matching here.

s56
00:05:57.520 --> 00:06:04.040
Um and yeah, and then we'll kind of once we've understood that this is the end of the user utterance, we'll kind of emit it out.

s57
00:06:04.040 --> 00:06:06.440
So, pretty standard.

s58
00:06:06.440 --> 00:06:09.800
Okay, and then the next layer here is going to be text-to-speech.

s59
00:06:09.800 --> 00:06:17.000
So, with text-to-speech the goal is to kind of take what the agent said, and the agent's going to be emitting this in a streaming fashion.

s60
00:06:17.000 --> 00:06:21.840
So, we're going to need to um produce audio as quickly as possible.

s61
00:06:21.840 --> 00:06:34.760
And ideally, what you can do is before the agent has even finished generating the full text you can have the audio play, so it's kind of hiding the latency of finishing the generation.

s62
00:06:34.760 --> 00:06:41.240
So, um I'm going to kind of play the streaming um the stream the streaming uh agent output now.

s63
00:06:41.240 --> 00:06:43.320
So, starts with you.

s64
00:06:43.320 --> 00:06:49.600
And yeah, actually before I uh further, there's this new concept that we're introducing here called the prefix cache.

s65
00:06:49.600 --> 00:07:05.919
So, the prefix cache is going to be looking at the um agent stream, and seeing if we already have generated audio for that sequence of words um from like a prior generation, or maybe like the same generation

s66
00:07:05.919 --> 00:07:10.000
um in this uh in this call as as well.

s67
00:07:10.000 --> 00:07:12.520
So, um it sees the word you.

s68
00:07:12.520 --> 00:07:19.320
Uh we for this prefix cache, we're going to be, you know, we don't want to like immediately hit on every single word.

s69
00:07:19.320 --> 00:07:22.560
We're going to be waiting for a little bit more words.

s70
00:07:22.560 --> 00:07:27.600
Um so, after three words, the prefix cache gets our first hit.

s71
00:07:27.600 --> 00:07:36.760
And um over here on the right, this is kind of our text-to-speech standard provider, you know, Cartesia is a text-to-speech engine with web socket support.

s72
00:07:36.760 --> 00:07:44.400
So, we're we're piping the agent through the the cache, and also piping it through web socket.

s73
00:07:44.840 --> 00:07:49.480
Um more tokens come in, more cache, more sending through web socket.

s74
00:07:49.480 --> 00:07:51.480
Not much to say here.

s75
00:07:51.480 --> 00:08:01.320
And okay, so now we get our first uh kind of first unique thing, which is we found the token that actually causes a cache miss.

s76
00:08:01.320 --> 00:08:02.280
And it makes sense.

s77
00:08:02.280 --> 00:08:12.720
If we're kind of caching previous generations, um you said your name is is a pretty common thing, but once you we add in the name, suddenly we're that's that's going to result in the cache miss.

s78
00:08:12.720 --> 00:08:15.520
At this point, we're actually going to yield out our cached audio.

s79
00:08:15.520 --> 00:08:22.400
So, you said your name is is going to be um emitted as the rest of the streaming text is coming back.

s80
00:08:22.400 --> 00:08:27.080
So, at this point, the user hears the agent and user doesn't really know what's going on.

s81
00:08:27.080 --> 00:08:31.560
They just looks like really fast response times to them.

s82
00:08:31.760 --> 00:08:35.360
Um and now the kind of remaining text flows through.

s83
00:08:35.360 --> 00:08:38.440
And at this point, we've already emitted from the cache.

s84
00:08:38.440 --> 00:08:40.080
The cache has done its job.

s85
00:08:40.080 --> 00:08:43.919
Um the rest we can kind of throw into Cartesia.

s86
00:08:43.919 --> 00:08:53.040
And here's kind of the trick where Cartesia has seen the entire transcript up to this point.

s87
00:08:53.040 --> 00:08:58.120
It to to Cartesia, like it doesn't know about the existence of this prefix cache.

s88
00:08:58.120 --> 00:09:03.480
It's just going to generate this full sentence with, you know, standard natural prosody.

s89
00:09:03.480 --> 00:09:14.240
But, what we do is when the generation comes back, since we've already played the audio here, we can actually suppress the audio from Cartesia here and just play out the remaining stuff.

s90
00:09:14.240 --> 00:09:19.240
So, the user, there's might be a tiny bit of a hiccup.

s91
00:09:19.240 --> 00:09:24.320
You know, I'll play some audio later and you'll know that you probably won't be able to notice.

s92
00:09:24.320 --> 00:09:34.320
And effectively, we're kind of taking this and or taking this part and just emitting it directly after this audio has finished playing.

s93
00:09:34.320 --> 00:09:37.080
So, it looks seamless to the user.

s94
00:09:37.080 --> 00:09:42.760
And yeah, so I guess at this point kind of this is a lot of technical details.

s95
00:09:42.760 --> 00:09:49.160
I'll I mean the best way is just kind of play a call and see see this in action.

s96
00:09:49.160 --> 00:09:55.120
So here I recorded a call and let's just hear what happens.

s97
00:09:55.120 --> 00:09:59.600
This is Bo OBGYN Elise speaking.

s98
00:10:00.080 --> 00:10:05.880
Hi, I think I might be pregnant and I want to schedule an ultrasound to confirm.

s99
00:10:06.160 --> 00:10:08.200
That's exciting news.

s100
00:10:08.200 --> 00:10:12.000
Can you provide your name and date of birth?

s101
00:10:12.040 --> 00:10:14.440
Sure, it's Elise Trial.

s102
00:10:14.440 --> 00:10:17.880
Date of birth 2303.

s103
00:10:21.680 --> 00:10:22.640
Thank you.

s104
00:10:22.640 --> 00:10:25.880
Are you a new patient with us?

s105
00:10:26.280 --> 00:10:28.280
Yes.

s106
00:10:29.240 --> 00:10:31.640
Welcome to Bo OBGYN.

s107
00:10:31.640 --> 00:10:35.760
Can I text you a link to upload your insurance information?

s108
00:10:36.920 --> 00:10:38.920
Sure.

s109
00:10:39.040 --> 00:10:40.480
Okay, sent.

s110
00:10:40.480 --> 00:10:43.560
Let me know when you've received it.

s111
00:10:48.680 --> 00:10:51.080
I've received your insurance information.

s112
00:10:51.080 --> 00:10:55.280
The earliest appointment I have is Thursday, July 2nd at 10:00 a.m.

s113
00:10:55.280 --> 00:10:57.040
Actually, give me a second.

s114
00:10:57.040 --> 00:10:59.680
Let me check my calendar.

s115
00:11:01.400 --> 00:11:04.360
Sure, take your time.

s116
00:11:04.480 --> 00:11:08.480
Um do you have something for next week?

s117
00:11:09.160 --> 00:11:20.880
For next week, I can offer an ultrasound with Dr. Avery Stone at North Clinic on Tuesday, July 7th at 2:00 p.m. or 3:00 p.m. Do either of those work for you?

s118
00:11:20.880 --> 00:11:23.600
Yeah, 2:00 p.m. works.

s119
00:11:23.800 --> 00:11:24.800
Great.

s120
00:11:24.800 --> 00:11:26.560
Your appointment has been booked.

s121
00:11:26.560 --> 00:11:29.800
We look forward to seeing you then.

s122
00:11:29.800 --> 00:11:30.280
Thanks.

s123
00:11:30.280 --> 00:11:32.280
Bye-bye.

s124
00:11:32.839 --> 00:11:33.280
All right.

s125
00:11:33.280 --> 00:11:34.880
Yeah, that's pretty much it.

s126
00:11:34.880 --> 00:11:48.160
Um Yeah, you can kind of see our all this like uh streaming and you know, a lot of things are happening in the background and and you know, this is what really makes like voice agents interesting.

s127
00:11:48.160 --> 00:11:59.440
And um there's a lot of effort that can be done in the harness to really kind of get a um natural conversation, which is what we're after.

s128
00:11:59.440 --> 00:12:00.200
Uh okay.

s129
00:12:00.200 --> 00:12:05.480
Yeah, so I guess briefly, you know, in the last part, I want to just talk a little bit about Elise.

s130
00:12:05.480 --> 00:12:12.680
So I think Elise, you know, our headquarters are in New York and kind of we're trying to expand our presence here in the Bay Area.

s131
00:12:12.680 --> 00:12:23.040
Um we I think it's maybe like a different style of company that I think people are like uh think of when they think about AI startups in San Francisco.

s132
00:12:23.040 --> 00:12:33.600
Where we're actually very focused on um just like helping people and helping people where they need it, like kind of the life's most critical areas.

s133
00:12:33.600 --> 00:12:46.120
We work on housing, health care and uh we're doing really well and you know, here's there's a link here to um you kind of join our team and there's going to we're going to be

s134
00:12:46.120 --> 00:12:50.600
uh posting a lot on Twitter, so you can follow us at EliseAI as well.

s135
00:12:50.600 --> 00:12:53.563
Um yeah, that's that's it.

s136
00:12:53.563 --> 00:12:55.563
[applause]