WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=vblnYHzBgS4

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/vblnYHzBgS4.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.120 --> 00:00:16.960
Let's just do a couple of quick questions and then we'll jump right in.

s3
00:00:16.960 --> 00:00:24.240
Uh, how many of us in the room here have built voice AI agents?

s4
00:00:24.720 --> 00:00:28.080
Okay, that's a that's a pretty good audience here.

s5
00:00:28.080 --> 00:00:33.279
And how many of you guys have built AI agents that have been deployed in production?

s6
00:00:34.640 --> 00:00:35.440
Not bad.

s7
00:00:35.440 --> 00:00:36.239
Okay, cool.

s8
00:00:36.239 --> 00:00:40.399
So, uh we'll talk about what typically happens, right?

s9
00:00:40.399 --> 00:00:44.160
Like everyone's talking about wise AI agents.

s10
00:00:44.160 --> 00:00:51.440
Uh the you know, one pill solution to pretty much everything in the world today uh is is wise agents.

s11
00:00:51.440 --> 00:00:54.480
So, everyone's building one and trying to deploy that.

s12
00:00:54.480 --> 00:01:03.760
They sound great when you're sort of building that in your dev sort of landscape and then the moment you take this to from a proof of concept to production

s13
00:01:03.760 --> 00:01:05.040
things start failing.

s14
00:01:05.040 --> 00:01:18.560
Uh so we'll walk through these five different angles of like how uh or what we have seen uh at PO with VIA agents but just before that a quick uh intro from from my side

s15
00:01:18.560 --> 00:01:31.200
uh I am Wenke the founder and CEO uh she used the title agent engineering manager u I'm calling myself chief agent officer uh from a from a title standpoint

s16
00:01:31.200 --> 00:01:42.320
u okay so what is what is uh you know why why are even qualified for this this discussion and and like uh what what are we seeing that a lot of companies don't get to see?

s17
00:01:42.320 --> 00:01:48.640
I I'll talk a bit about our journey in terms of like how we've uh come along so far and then jump right in.

s18
00:01:48.640 --> 00:01:52.640
Uh you know we were we've been around for about 14 years.

s19
00:01:52.640 --> 00:01:57.840
Our journey has been a developer API platform and then now an uh you know an AI agent business.

s20
00:01:57.840 --> 00:02:10.319
We started with voice and SMS APIs back in the day uh 2011 and then uh you know now we are primarily focused on our AI agent offering uh the the full stack

s21
00:02:10.319 --> 00:02:11.440
on our platform.

s22
00:02:11.440 --> 00:02:16.560
We we see over a billion voice calls each month across the globe.

s23
00:02:16.560 --> 00:02:25.920
Uh and which is where we've seen a lot of these uh you know patterns emerge in terms of like how when we work with our customers what happens on their voice agents in in production.

s24
00:02:25.920 --> 00:02:33.440
Uh we're a uh 90 member team and uh we've we have uh 50 million funding in the bank.

s25
00:02:33.440 --> 00:02:36.879
Fun fact, this is not from external VC investors.

s26
00:02:36.879 --> 00:02:42.640
This is all from being a profitable company having put that cash in the bank over over these years.

s27
00:02:42.640 --> 00:02:47.120
Uh some customers we we power across the globe.

s28
00:02:47.120 --> 00:02:50.000
Uh you know, we've just left some some logos in there.

s29
00:02:50.000 --> 00:02:55.840
But primarily from an offering standpoint, u I would sort of cohort this into three different buckets.

s30
00:02:55.840 --> 00:02:58.959
One is a programmable AI agent offering.

s31
00:02:58.959 --> 00:03:05.360
We call it uh I mean it's a speech pipeline, not a true speech-to-pech product yet, but that's a that's a programmable offering.

s32
00:03:05.360 --> 00:03:07.280
We also have an AI agent studio.

s33
00:03:07.280 --> 00:03:10.319
It's a no code visual uh builder.

s34
00:03:10.319 --> 00:03:13.840
And then we like I said, we started with voice APIs.

s35
00:03:13.840 --> 00:03:19.360
So we obviously have built this out over the last 14 years, the SIP trunking and the audio streaming layers.

s36
00:03:19.360 --> 00:03:23.280
So we don't rely on other folks for the telefony or the carrier layer.

s37
00:03:23.280 --> 00:03:32.400
Like that's the breadandbut business we've built over all these years and and that's on top of which our AI uh agent platform sits.

s38
00:03:32.400 --> 00:03:49.360
Okay, with that uh let's get into this, right, which I'm I'm sure since you guys have all built AI agents, you've all seen this or you know built this in in one manner or another and we'll spend more time on this in terms of like how uh the entire pipeline looks, right?

s39
00:03:49.360 --> 00:04:01.040
Uh what we see with customers is and and I'm sure you guys can all relate to this is you know anyone thinking about AI agents what they do is they pick a bunch of these orchestration frameworks and

s40
00:04:01.040 --> 00:04:10.480
they do a pretty good job live kit or a pipecat you know build their AI oen on top of that uh they think they can just sort of orchestrate these different four layers

s41
00:04:10.480 --> 00:04:21.280
speechtoext lm uh and and TTS with turn detection in between and we're off to the races like my AI engine agent works in a in a P and it's good to work in production.

s42
00:04:21.280 --> 00:04:23.040
Uh typically that's what happens.

s43
00:04:23.040 --> 00:04:34.240
They sort of measure their latencies and you can see some indicative latencies on on this slide at at each layer and they're like yeah this this uh seems good for me for what I need.

s44
00:04:34.240 --> 00:04:46.800
So let let's position production and then the the production w uh sort of start to kick in and and you see all sort of failure modes which we are going to spend you know most of the time on on in this talk at least.

s45
00:04:46.800 --> 00:04:57.440
Uh I've kept some time at the end for Q&A if you guys want to have uh you know questions but we'll jump right in from from this to uh you know different failure modes we see.

s46
00:04:57.440 --> 00:05:06.160
Let's start with you know the first one which everyone talks about like this is the most spoken about failure mode which is latency.

s47
00:05:06.160 --> 00:05:10.800
U I think we have a few AI agent talks today or AI agent talks today.

s48
00:05:10.800 --> 00:05:23.520
Um I'm pretty sure like everyone everyone's going to touch upon this specific failure mode which is why I'm bringing this right up uh in in terms of uh you know some like how this entire experience is

s49
00:05:23.520 --> 00:05:41.919
for uh users right uh typically most folks measure this by time to first audio so the time when you user stop speaking to your agent starts speaking right and I think you've you've probably seen this if you guys have built voice agents on, you know, what

s50
00:05:41.919 --> 00:05:52.960
uh good or natural feels like, what uh sort of annoying feels like or noticeable feels like, and then what annoying feels like, which is, you know, different tiered steps.

s51
00:05:52.960 --> 00:06:05.600
Uh we notice, you know, most people want to be under 550 cuz that's what's advertised by, you know, platforms or uh you know, solutions or or or or layers.

s52
00:06:05.600 --> 00:06:11.039
But I think most end up between 750 to 1.2. uh that's where most of the folks end up at.

s53
00:06:11.039 --> 00:06:17.600
Uh the really bad performing ones end up you know more than 1.2 and then you start to see users uh hang up.

s54
00:06:17.600 --> 00:06:30.400
Uh now I I'll share with you like what we've seen practically in in uh production with uh customers using this with at at different layers and then you know

s55
00:06:30.400 --> 00:06:32.560
solutions to uh some of these.

s56
00:06:32.560 --> 00:06:43.600
The way we want to think about this layer is sort of a balance between these three which is cost, intelligence and latency, right?

s57
00:06:43.600 --> 00:06:45.919
And and and why do I bring these three up?

s58
00:06:45.919 --> 00:06:47.840
Because they're sort of interrelated.

s59
00:06:47.840 --> 00:06:55.759
I think one of the things I was just chatting with uh you know a couple of folks outside one of the things last one year we've seen lot of innovations lot of intelligence spike

s60
00:06:55.759 --> 00:07:10.720
on the LLM side of uh things right and most of the you know intelligence has come in in terms of thinking or uh you know reinforcement learning and and so on and so forth the irony with voice agents is

s61
00:07:10.720 --> 00:07:18.080
like almost always your the the LLM or the agent that's talking has to have thinking turned Right.

s62
00:07:18.080 --> 00:07:25.440
So all the advancements we've had in the LLM layer in the last one year like none of that even apply here now.

s63
00:07:25.440 --> 00:07:25.680
Right?

s64
00:07:25.680 --> 00:07:38.240
You obviously you have you know better models that can do you know better instruction following or tool calling but pretty much all of your intelligence that's been built in on the thinking layer is all off by default if you want it to be fast enough.

s65
00:07:38.240 --> 00:07:40.479
So so that's one of the ironies that we come up with.

s66
00:07:40.479 --> 00:07:44.560
So then how do you sort of balance intelligent cost and latency?

s67
00:07:44.560 --> 00:07:54.800
Let's let's look at some of these uh you know options uh that are out there in the market right so and I'm specifically picking LLM because if you looked at the previous chart LLM

s68
00:07:54.800 --> 00:08:06.639
is u you know sort of your highest latency bucket that adds to this right and uh if you look at you know frontier models which I think most folks

s69
00:08:06.639 --> 00:08:18.400
start by default your your openi your clouds your geminis u you know p50 ttfftd is roughly around 450 to 500 on on a good day and it can get spiky, right?

s70
00:08:18.400 --> 00:08:30.080
It can it can uh you know P90 P95 can go easily upwards of 1.2 1.3 seconds even uh and and that's not good for the overall agent experience.

s71
00:08:30.080 --> 00:08:32.000
So so that so that's your frontier model.

s72
00:08:32.000 --> 00:08:42.880
Now there's another options which is your your cerebrus or or the gro that is famous and popular for spitting out a lot of tokens or or tokens very fast, right?

s73
00:08:42.880 --> 00:08:53.360
uh these work but for you to get dedicated latency or time to first token on these you need dedicated capacity and that is really expensive that's where I spoke about the cost

s74
00:08:53.360 --> 00:09:06.640
uh as as being one of the things to balance right it's really expensive and then like you talk to anyone from the gro team or the cerebrus team they'll tell you you need to book 12 months in advance for dedicated capacity they're booked out for the next 12 months so

s75
00:09:06.640 --> 00:09:20.560
so that's that's a pretty expensive option and then you really need to be sure that the model you're deploying on some of these infra layers uh will be here 12 months from now and and it's a it's a big investment and a big unknown.

s76
00:09:20.560 --> 00:09:36.080
So, so what's a realistic option for production grade uh agents that are that are good quality and end up balancing uh three of these u this is what has worked for us

s77
00:09:36.080 --> 00:09:47.760
u which is the open source models u there are obviously a lot of them in terms of like the variety and and variations you can pick I'm specifically talking about the two we work with

s78
00:09:47.760 --> 00:09:51.120
u quen 3.5 and gemma four.

s79
00:09:51.120 --> 00:10:03.040
These are uh you know kind of cutting edge open source models right uh out in the market right now and we've done a lot of benchmarking around this in how they work.

s80
00:10:03.040 --> 00:10:15.200
It it can be scary to think like okay I have the models now I have to host them you know run them on my own GPUs and so on and so forth but if you are consistently targeting under

s81
00:10:15.200 --> 00:10:27.279
300 ms u this we've seen this to be a a great option to balance between latency cost and intelligence now some more deep dive here if you're doing only English

s82
00:10:27.279 --> 00:10:39.760
uh quen 3.5 or GMA both work fine but if you're doing multilingual uh right international audiences different languages uh Gemma 4 is a much better model for that uh we've seen

s83
00:10:39.760 --> 00:10:50.320
uh token fertility evals essentially what that means is if if I were to dejargonize that is like how many tokens does it take to generate one word in that language

s84
00:10:50.320 --> 00:11:02.240
okay so Gemma is much much better at least 2.5 to 3x better than quen 3.5 from that perspective so your time to words is much faster on Gemma or

s85
00:11:02.240 --> 00:11:06.480
everything else equal right on a on a multilingual basis.

s86
00:11:06.480 --> 00:11:09.600
Now what sizes do you pick at the LLM layer?

s87
00:11:09.600 --> 00:11:13.519
Uh the mixture of expert usually works fine.

s88
00:11:13.519 --> 00:11:16.880
Uh the three or four billion mixture of expert usually works fine.

s89
00:11:16.880 --> 00:11:26.079
The the problem with mixture of expert is like if anyone goes down wants to go down the direction of fine-tuning that can be a challenge uh because fine-tuning mixture of experts

s90
00:11:26.079 --> 00:11:27.440
models are not easy.

s91
00:11:27.440 --> 00:11:30.480
Uh you can end up breaking the model uh a a lot of times.

s92
00:11:30.480 --> 00:11:40.560
So, so that's one challenge we see with Make sure experts, but usually out of the box, it gets you 90% closer to where you want to be like even without any fine-tuning

s93
00:11:40.560 --> 00:11:43.680
or or or custom work done on the model.

s94
00:11:43.680 --> 00:11:46.720
Uh so that's the advantage of mixer experts.

s95
00:11:46.720 --> 00:11:54.959
Uh now, if you want to fine-tune and and you you want to go deeper and say like look, I'm working for a specific domain, healthcare, what have you, right?

s96
00:11:54.959 --> 00:11:57.600
uh and I want to make sure I I'm able to fine-tune my model.

s97
00:11:57.600 --> 00:12:02.800
You want to start at least with uh the 8 billion 12 billion at least uh from where we are today.

s98
00:12:02.800 --> 00:12:08.959
Maybe maybe six months from now a 4 billion 4 billion model beats the 8 billion model uh hands down.

s99
00:12:08.959 --> 00:12:15.519
But for today uh what we've seen is you minimum need a 8 billion or 12 billion model.

s100
00:12:15.519 --> 00:12:17.920
Uh cuz you're looking for two things in these models.

s101
00:12:17.920 --> 00:12:23.120
One obviously fast tokens but uh good instruction following.

s102
00:12:23.120 --> 00:12:23.519
Okay.

s103
00:12:23.519 --> 00:12:40.000
And the second thing is like very high uh success ratio in tool calling because if you can do these two things well then you are on to like 70 80% there from not even having to fine-tune it fine-tune any model like models will work out of the box right u so so that's

s104
00:12:40.000 --> 00:12:55.279
uh been our recipe we've actually uh we run two flavors one a fine tune model for specific industries and then for uh you know most generic use cases uh MOE model just works out of the box.

s105
00:12:55.279 --> 00:13:04.800
Uh there are a few more tips and tricks we'll talk about in the upcoming slides where we see failure models, but but that's where we stand from a from a latency LLM standpoint.

s106
00:13:04.800 --> 00:13:10.240
Um all right, I'm running tight on time, so I'm going to fast track this.

s107
00:13:10.240 --> 00:13:12.399
U now there are a couple of other flavors in this.

s108
00:13:12.399 --> 00:13:16.000
Uh people build agents with a a mixture of models.

s109
00:13:16.000 --> 00:13:29.600
What they do is you know for u the the talking part of it they have a conversational model which is a much lower smaller model and then you know maybe even a three billion model and then for tool calling they have a much larger model so that they have a

s110
00:13:29.600 --> 00:13:31.920
improved tool calling success ratio there.

s111
00:13:31.920 --> 00:13:48.720
Uh sorry the second one is uh assume your transcriptions are going to be brittle like that's that's uh something you want to sort of uh live by when you're building AI agents

s112
00:13:48.720 --> 00:13:58.959
even if you have the best transcription engine out there and I I I'll show you why right like the the the state-of-the-art transcription engines out out in the market

s113
00:13:58.959 --> 00:14:14.639
u you know sort of get you to four to 6% word error rate right and this is on known eval sets on real world noisy calls with you know sort of uh accents like people having different sort of accents

s114
00:14:14.639 --> 00:14:23.600
uh domain vocabulary and so on and so forth like those usually end up in the double digits from a word erate perspective right uh now you obviously you can fine-tune

s115
00:14:23.600 --> 00:14:32.480
you know pick up an open source model and fine-tune uh but we see typically like what breaks here often and there are patterns s here in terms of what breaks.

s116
00:14:32.480 --> 00:14:40.079
So, proper nouns, jarens, uh phone numbers like random missing digits with phone numbers, uh wrong substitutions.

s117
00:14:40.079 --> 00:14:46.720
I I'll walk through some examples of like how you solve for these addresses when you're trying to collect a long address.

s118
00:14:46.720 --> 00:14:52.320
Uh you know, the the transcription engine could just end up missing some parts of it.

s119
00:14:52.320 --> 00:14:53.600
Code switch languages.

s120
00:14:53.600 --> 00:14:59.279
I I'll just take a example of a language I speak because that's was easy for me to put on the slide.

s121
00:14:59.279 --> 00:15:09.680
uh where you know like if you were to sort of take English but written in a different script uh that's what's used for Hindi right like this is English written in that script

s122
00:15:09.680 --> 00:15:18.480
right whereas like the actual English version of this is hello how are you so if if I'm addressing an audience in a different country where I have code switched languages

s123
00:15:18.480 --> 00:15:33.680
and I start getting my English in a different uh sort of script everything starts breaking from the transcription engine to the LLM layer and then beyond because your LLM starts then producing output in that sort of script a lot of times and then your TTS messes up.

s124
00:15:33.680 --> 00:15:34.000
Okay.

s125
00:15:34.000 --> 00:15:44.480
So, so this is uh very important to be careful about and if you want to build your agent independent of the transcription engine, you need to build a layer that normalizes all of this, right?

s126
00:15:44.480 --> 00:15:46.000
We'll talk about solutions in a minute.

s127
00:15:46.000 --> 00:15:51.279
And there is the other case which is Hindi in Latin or or or you know Roman, right?

s128
00:15:51.279 --> 00:15:57.680
Which is like this is Hindi but it reads English which again messes up everything uh you know downstream.

s129
00:15:57.680 --> 00:15:58.800
Those are just examples.

s130
00:15:58.800 --> 00:16:02.800
This applies to, you know, Arabic, Mandarin, uh, Japanese, what have you.

s131
00:16:02.800 --> 00:16:04.480
Uh, pretty much any language.

s132
00:16:04.480 --> 00:16:09.519
So, what actually moves the needle with a at the transcription layer?

s133
00:16:09.519 --> 00:16:14.800
Uh, for prop proper nouns, we recommend uh you using not just keyword boosting.

s134
00:16:14.800 --> 00:16:22.959
I think a lot of transcription engine engines provide you keyword boosting where you can put in specific words into their engine, but doing dynamic keyword boosting.

s135
00:16:22.959 --> 00:16:27.120
What that means is don't keep the keyword for the entire state of the call.

s136
00:16:27.120 --> 00:16:33.839
just add that dynamically when you think you need that as an answer so that you get the highest accuracy.

s137
00:16:33.839 --> 00:16:41.279
Meaning at different states of the call, the transcription engine will have different uh keywords boosted during different phases, right?

s138
00:16:41.279 --> 00:16:49.680
Uh and that's what we've seen works best because if you just pollute your context of the transcription engine with tons of keywords, it'll start hallucinating again, right?

s139
00:16:49.680 --> 00:16:52.399
So, so that's what we see typically working best.

s140
00:16:52.399 --> 00:16:58.079
Uh yeah, post-process post-process your transcripts with an LLM, right?

s141
00:16:58.079 --> 00:17:00.160
Cuz your LLM has domain context.

s142
00:17:00.160 --> 00:17:02.160
Your transcription engine does not.

s143
00:17:02.160 --> 00:17:06.799
So a lot of words that it would say uh I'll give you some examples may not make sense.

s144
00:17:06.799 --> 00:17:10.480
This is transcription like a phone number from a transcription engine.

s145
00:17:10.480 --> 00:17:10.799
Right?

s146
00:17:10.799 --> 00:17:13.439
Like what do you think that E is?

s147
00:17:13.439 --> 00:17:13.919
Right?

s148
00:17:13.919 --> 00:17:16.319
If you give it to an LM, it knows that's a three.

s149
00:17:16.319 --> 00:17:19.199
Similarly, like what that one is, it's a digit one.

s150
00:17:19.199 --> 00:17:29.200
So, so your transcription engine a lot of times could mess that up, but when you postprocess it with the LLM layer, it'll instantly correct that from a collection standpoint.

s151
00:17:29.200 --> 00:17:41.840
I mean, uh, and and the last one, like I said, uh, transliteration is your ST output that's sort of u, you know, multilingual also gets normalized using either an NLM

s152
00:17:41.840 --> 00:17:47.760
you first transliterated or, you know, use some kind of a neural uh, transliteration engine.

s153
00:17:47.760 --> 00:17:48.880
There are a lot of them open source.

s154
00:17:48.880 --> 00:17:50.480
You can just pick one of them, right?

s155
00:17:50.480 --> 00:17:52.480
Uh that would do all of that work for you.

s156
00:17:52.480 --> 00:17:58.880
Send cleaned transcripts consistently independent of the transcription engine to your LLM.

s157
00:18:00.559 --> 00:18:00.880
All right.

s158
00:18:00.880 --> 00:18:03.280
The third one we typically see is collecting data.

s159
00:18:03.280 --> 00:18:08.240
This is where I think 50 to 60% of AI agents mess up pretty badly.

s160
00:18:08.240 --> 00:18:14.000
Uh and like we like to think of it as a UX problem.

s161
00:18:14.000 --> 00:18:15.919
Uh but just for voice.

s162
00:18:15.919 --> 00:18:23.200
So think data models uh and not a transcript coming into an LLM and and trying to figure out what the transcript said.

s163
00:18:23.200 --> 00:18:40.559
So let's take some inspiration from uh I'm assuming most of us are developers here um you know take inspiration from Python's data classes pantic zod from Typescript or form fields in the UI right like if you start thinking of it from that problem statement

s164
00:18:40.559 --> 00:18:48.960
we have seen accuracy grow up from grow from 30% to like 95% from a data collection standpoint when you start thinking in that manner.

s165
00:18:48.960 --> 00:18:52.559
So like decide your shape before you ask, right?

s166
00:18:52.559 --> 00:18:55.919
Like instead of keeping it open-ended, can you keep it constrained?

s167
00:18:55.919 --> 00:18:59.919
So can can a phone number be a phone number type field?

s168
00:18:59.919 --> 00:19:05.120
The moment you do that, right, you know like how many digits it needs to have.

s169
00:19:05.120 --> 00:19:07.840
You can do validation on on top of that, right?

s170
00:19:07.840 --> 00:19:10.880
And then what sort of allowed values can even be there.

s171
00:19:10.880 --> 00:19:21.600
So in the previous example we saw if an E comes in in middle of a phone number and you know it's a phone number you instantly know like either you smart guess that to three and confirm that with a user

s172
00:19:21.600 --> 00:19:31.280
or you know that's an error and then you validated that and asked the user to repeat again right so so that's I think one of the common patterns we've seen here from from a

s173
00:19:31.280 --> 00:19:45.280
a collection pattern name I think is the is the interesting one I've just picked a you know a a hard to pronounce name like There's no way a human is going to get this right and and no way a transcription engine will get this right.

s174
00:19:45.280 --> 00:19:46.880
How many ever times you do this right?

s175
00:19:46.880 --> 00:20:03.039
So the moment you start thinking of this as fields and then have rules and then confirmation mechanisms on on spelling this uh you know sort of uh letter by letter only then you kind of get it right otherwise it's going to mess up pretty badly in terms of how you collect this on a voice call and and

s176
00:20:03.039 --> 00:20:11.440
that's just an example of you know what u I'm talking about in terms of the the data collection piece of it.

s177
00:20:11.520 --> 00:20:17.120
Another place where it goes badly dramatically is relative uh values.

s178
00:20:17.120 --> 00:20:18.799
Date being one of the examples.

s179
00:20:18.799 --> 00:20:27.520
If somebody says next week uh Wednesday 8, 8 could mean 8:00 a.m. 8:00 p.m. and then figuring out what that date actually is.

s180
00:20:27.520 --> 00:20:29.840
Again, now becomes a very constrained problem.

s181
00:20:29.840 --> 00:20:38.320
If you knew this was a datetime field and I I'm collecting a datetime field and then you take the current date and then figure out what this value would be bases that, right?

s182
00:20:38.320 --> 00:20:50.080
So, so that's how you want to make sure like uh you do this with a combination of the LLM with the tool calling and the tool calling is doing a lot of this heavy lifting for you from a from a field standpoint.

s183
00:20:51.120 --> 00:20:51.360
Yeah.

s184
00:20:51.360 --> 00:20:55.520
And then you make you you run like this from a unit test perspective.

s185
00:20:55.520 --> 00:21:01.919
So all of your u evals need to start treating these fields as unit tests.

s186
00:21:01.919 --> 00:21:09.760
And as long as your unit tests uh sort of validate and pass, you know, your agent is going to be uh sort of reliable and repeatable.

s187
00:21:09.760 --> 00:21:16.559
You don't, you know, run uh hundreds of end to end agent test cases just to find out, you know, one field collection is broken.

s188
00:21:16.559 --> 00:21:22.320
You do your eval at a field level and a unit test uh level.

s189
00:21:25.120 --> 00:21:36.559
And then yeah, like I said, I think u you this this mindset makes everything more structured instead of hoping I'll put a ton of prompt, keep changing, you know, the prompt by a few

s190
00:21:36.559 --> 00:21:44.960
uh characters every time and somehow my prompt engineering is going to make LLM much more instruction tuned and sort of magically start following some of these things.

s191
00:21:44.960 --> 00:22:01.120
So in fact u like I said right like we have seen us get to 95 97% accuracy without having to fine-tune a model right and then and the trick is basically like just breaking down your context of what the agent is doing at that point with specific

s192
00:22:01.120 --> 00:22:04.960
u states of what the agent is going through.

s193
00:22:05.039 --> 00:22:11.280
All right u I'm just going to quickly u skip through this from a time standpoint.

s194
00:22:11.280 --> 00:22:13.360
I just see I got three more minutes.

s195
00:22:13.360 --> 00:22:17.360
Um hopefully that's a bug but but we'll leave it at that.

s196
00:22:17.360 --> 00:22:17.679
Okay.

s197
00:22:17.679 --> 00:22:23.600
Um so so this is the fourth area where we see issues coming in.

s198
00:22:23.600 --> 00:22:28.159
Most folks take the LLM output and then we send it to a TTS.

s199
00:22:28.159 --> 00:22:35.760
Obviously I think there are a lot of good TTS's in the market that take care of a lot of heavy lifting but a lot of times it it messes up.

s200
00:22:35.760 --> 00:22:44.880
Uh what we recommend and what we've seen is you usually want to have a normalization layer between your LLM and what is fed to a TTS.

s201
00:22:44.880 --> 00:22:49.280
You don't send your LLM output directly to a TTS, right?

s202
00:22:49.280 --> 00:22:52.320
And and we'll just walk through some examples.

s203
00:22:52.320 --> 00:22:59.200
The basics which is strip emojis uh markdown before before any synthesis into the TTS.

s204
00:22:59.200 --> 00:23:05.280
Most orchestration pipelines do this like you know a live kit or a pipecat would do that for you if you just set a few flags.

s205
00:23:05.280 --> 00:23:18.000
So I but but just make sure if you're not using them or buildings from scratch that you've set this explicitly because you don't want an emoji showing up on on on something read out or you know markdown showing up there.

s206
00:23:18.000 --> 00:23:18.559
Okay.

s207
00:23:18.559 --> 00:23:30.720
I think I think some more common ones uh custom uh dictionaries most TTS engines provide this to you like how to pronounce custom words whether it's you know proper nouns brands

s208
00:23:30.720 --> 00:23:32.880
uh acronyms and so on and so forth.

s209
00:23:32.880 --> 00:23:38.400
So set those in uh when you go from your LLM to your TTS output because if you don't, you're going to mess that up.

s210
00:23:38.400 --> 00:23:41.600
And I I'll I'll show you an example of like how we test that.

s211
00:23:41.600 --> 00:23:46.400
Uh the the other one is like most engines also give you speed.

s212
00:23:46.400 --> 00:23:49.280
So if you know you're pronouncing an entity, slow down.

s213
00:23:49.280 --> 00:23:50.720
Have your agent slow down.

s214
00:23:50.720 --> 00:24:01.440
So at point 8x or 7x so that it it's able to like inunciate on that specific entity and and doesn't mess up how it's pronouncing an email or a phone number

s215
00:24:01.440 --> 00:24:18.559
or a name letter by letter and yeah just normalize all the messy stuff right like emails currency dates don't leave it to the TTS to do it uh most of them do it but don't leave it to the TTS to do it like build your normalization

s216
00:24:18.559 --> 00:24:26.640
layer at your end so that tomorrow you think you need to switch TTS or you know for whatever reason the first one's down and you want to use another TTS

s217
00:24:26.640 --> 00:24:40.480
you're able to sort of not rely natively on the TTS's engine but you are building this in-house uh for for this to be managed and then yeah u I think I don't have my batch here but

s218
00:24:40.480 --> 00:24:50.559
I I don't have my last name on that so my first test is if it cannot pronounce my last name or my company's name it's already dropping the ball so my last name is uh Balas Subramanion

s219
00:24:50.559 --> 00:24:56.480
and if you cannot pronounce that using a voice AI agent uh like that's a check for me.

s220
00:24:56.480 --> 00:25:06.080
I I know like uh you know the agent will mess up a lot of words that uh you know need to be spelled out day by day.

s221
00:25:06.080 --> 00:25:08.080
The second one is our company name Po.

s222
00:25:08.080 --> 00:25:13.679
So a lot of engines pronounce pronounce it pivo or uh pleo and and so on and so forth.

s223
00:25:13.679 --> 00:25:18.400
But but I think specifically being able to control this in your pipeline is super critical.

s224
00:25:18.400 --> 00:25:26.799
And then if you're building a if you're building a customerf facing product then then um you know sort of give this option to your customers.

s225
00:25:26.799 --> 00:25:30.240
All right I'm just going to skim through the the the last two slides.

s226
00:25:30.240 --> 00:25:32.960
U I'm I'm running badly over time.

s227
00:25:32.960 --> 00:25:34.400
Uturn detection.

s228
00:25:34.400 --> 00:25:44.799
I think this is it own separate topic but I'm just going to quickly pull up all the points so you guys can skim through that and if if you need a chat u after this we can we can talk about this.

s229
00:25:44.799 --> 00:25:45.120
Right.

s230
00:25:45.120 --> 00:25:51.360
Uh I'm just going to leave that for like five seconds and then and then we can chat about this offline.

s231
00:25:51.360 --> 00:25:53.360
I'm quite over time.

s232
00:25:53.360 --> 00:25:57.440
And then the the the last one is uh bargin and and back channeling.

s233
00:25:57.440 --> 00:26:04.720
I think there's a lot of talk around speech to speech models that do some of this, but we've been able to see how we could do all of this in speech to speech pipelines.

s234
00:26:04.720 --> 00:26:08.080
You really don't need a speech to speech model to do all of this up.

s235
00:26:08.080 --> 00:26:13.279
Uh again, I'll just I just put put this up on the slide and and sort of close at that.

s236
00:26:13.279 --> 00:26:23.200
Um all right I don't think we have time for questions we can take them offline if you have any time but uh hopefully this was helpful and gave you some insights on

s237
00:26:23.200 --> 00:26:26.880
uh what we are seeing in productions uh with billions of calls at scale.

s238
00:26:26.880 --> 00:26:29.279
All right thanks