WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Zd5b40Jbp_k

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Zd5b40Jbp_k.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.880 --> 00:00:14.000
Hi everyone.

s3
00:00:14.000 --> 00:00:14.960
How's it going?

s4
00:00:14.960 --> 00:00:17.440
Hey Patricia, how are you?

s5
00:00:17.440 --> 00:00:21.840
Uh so last presentation of the day, so let's make it count.

s6
00:00:21.840 --> 00:00:23.840
Um, all right.

s7
00:00:23.840 --> 00:00:28.560
Let's, uh, let me start with a little bit of background on myself.

s8
00:00:28.560 --> 00:00:34.640
And, um, my background, I'm a voice subject matter expert.

s9
00:00:34.640 --> 00:00:49.120
I've been working in voice AI for a long time across different surfaces, devices, and um, both at Alexa, at at Roku, at my own startups, you know, in the app store.

s10
00:00:49.120 --> 00:00:56.559
And my perspective is a little different from a lot of other voice AI practitioners.

s11
00:00:56.559 --> 00:01:12.240
I think it's a combination of um a deep um voice user interface expertise and intu intuition mixed in with new technical approaches uh that I think can produce really magical experiences.

s12
00:01:12.240 --> 00:01:23.040
So I think it's both sides and I think that's especially true in this new area that we're in with frontier tech where the human interface is basically being redefined.

s13
00:01:23.040 --> 00:01:29.040
So let me start with uh I'll just blast through the first couple of slides then get to the premise.

s14
00:01:29.040 --> 00:01:32.960
I think everybody knows that voice has incredible potential.

s15
00:01:32.960 --> 00:01:36.400
There's the power of voice I think across everywhere.

s16
00:01:36.400 --> 00:01:37.840
It's the most natural interface.

s17
00:01:37.840 --> 00:01:39.920
Humans love talking.

s18
00:01:39.920 --> 00:01:43.920
And uh the problem is the other half is the pain of voice.

s19
00:01:43.920 --> 00:01:45.600
So it's the power and the pain.

s20
00:01:45.600 --> 00:01:47.360
Voice is errorprone.

s21
00:01:47.360 --> 00:01:52.320
And I think those errors are going to continue for a while.

s22
00:01:52.320 --> 00:02:07.200
And I think the cost or consequence of those errors is going to grow, especially as we go fromational AI bots to embodied AI where rather than just giving answers that might be erroneous,

s23
00:02:07.200 --> 00:02:19.760
we're going to have AI systems take physical actions or digital actions where, you know, if the robot throws your watch out with the trash, it's a lot worse than playing the wrong song.

s24
00:02:19.760 --> 00:02:29.120
So I do think that a new approach is definitely needed and here's the TLDDR of the premise we're going to walk through today.

s25
00:02:29.120 --> 00:02:43.200
Um there are two ways to improve customer or user satisfaction of a voice AI assistant and that is by increasing accuracy which people know about I mean technically accuracy

s26
00:02:43.200 --> 00:02:54.319
and the other is a different knob that we have that we are not using adequately and I'll call that a system decision which we will define which is orthogonal

s27
00:02:54.319 --> 00:03:10.159
which is different from accuracy and I believe This approach which I have used in several different environments and seen some success I think is a promising area that we should consider developing.

s28
00:03:10.159 --> 00:03:22.400
Um let me walk through this with a simple smart speaker example and we'll go step by step with this approach but it is a scalable approach that I think

s29
00:03:22.400 --> 00:03:25.680
uh can apply across different surfaces and devices.

s30
00:03:25.680 --> 00:03:27.280
So let's get started.

s31
00:03:27.280 --> 00:03:37.519
So suppose we all you know are making a smart speaker coincidentally called uh Alexa and Alexa is very simple.

s32
00:03:37.519 --> 00:03:46.159
It just allows you to you know ask for music and it'll play a song and of course it will play either the song you wanted or a different song.

s33
00:03:46.159 --> 00:03:48.799
So it'll be right or it'll be wrong.

s34
00:03:48.799 --> 00:03:51.840
This isn't that different from what you've seen out there.

s35
00:03:51.840 --> 00:03:55.599
Um now let's to first talk about accuracy.

s36
00:03:55.599 --> 00:03:56.720
Accuracy.

s37
00:03:56.720 --> 00:04:00.560
Let's say we define it as we you know take a thousand spoken requests.

s38
00:04:00.560 --> 00:04:02.879
We observe the input and the output.

s39
00:04:02.879 --> 00:04:05.439
We label it and we look at this.

s40
00:04:05.439 --> 00:04:13.760
This is the map of a thousand points and 79% of the time 790 dots here were actually the correct song.

s41
00:04:13.760 --> 00:04:19.359
This is let's say human annotated 20% 21% wrong song.

s42
00:04:19.359 --> 00:04:21.199
So that's the accuracy.

s43
00:04:21.199 --> 00:04:33.040
Now, like I said, knob one is to spend a lot of time working on improving the accuracy, you know, um, percentage point by percentage point at any layer in the stack.

s44
00:04:33.040 --> 00:04:48.800
There's, if it's a cascaded system, you know, there's a perhaps a wakeword layer and a speech ASR layer and a NLU layer which might have intent classification, entity extraction, a lot of different layers, VAD, etc. And any of those can contribute to errors.

s45
00:04:48.800 --> 00:05:06.080
So we spent time we might be able to reduce that 210 to a smaller number that is I think a known area that we're tackling but I think knob 2 which is what I was talking about is what we'll go through here which is keeping the accuracy exactly the same.

s46
00:05:06.080 --> 00:05:15.600
So 79% what could we do in conditions of uncertainty to improve user satisfaction apparent and I I think we can do a lot.

s47
00:05:15.600 --> 00:05:24.160
So let's start first with the original system is just acting like I said user says something system plays a song it's either the right song or the wrong song

s48
00:05:24.160 --> 00:05:33.840
immediately I think just common sense tells us that we could introduce at least one system behavior to stop or rather to reject the hypothesis and do nothing.

s49
00:05:33.840 --> 00:05:42.479
So uh there is now one more option to decide the system may decide and say sorry I didn't get that or sorry could you repeat that?

s50
00:05:42.479 --> 00:05:50.240
Uh the challenge of course is how how when do we decide to stop and I mean quantitatively.

s51
00:05:50.240 --> 00:06:01.440
Um here's one approach to kind of visualizing this because if we don't we'll just take probably some swag like some guesstimate and I'll prove that if we just took a guesstimate

s52
00:06:01.440 --> 00:06:05.039
we would end up with a worse situation than a more rigorous approach.

s53
00:06:05.039 --> 00:06:15.919
So let's just assume I took those thousand data points and like I said they've been annotated and we assign a confidence score a single confidence score to the hypothesis

s54
00:06:15.919 --> 00:06:20.720
that was generated by the system you know between zero and one and let's say it's reasonably calibrated.

s55
00:06:20.720 --> 00:06:27.919
This is a simplification of if it's a cascaded system there are multiple layers and multiple you know confidence scores but let's just assume that for now.

s56
00:06:27.919 --> 00:06:28.720
Whoops.

s57
00:06:28.720 --> 00:06:33.840
So we're going to have 790 points 200 that are correct 210 wrong.

s58
00:06:33.840 --> 00:06:38.800
Each one has a confidence score and we're going to plot it, you know, plot the distributions.

s59
00:06:38.800 --> 00:06:43.600
Uh on the x-axis, I've just converted from 0ero to one to percentages.

s60
00:06:43.600 --> 00:06:57.280
And the question is how do we choose a threshold t such that whatever that percentage is um to the left of it meaning if when the system um forms a hypothesis

s61
00:06:57.280 --> 00:07:09.919
if the confidence score c is less than that t stop and say sorry otherwise play question is how do we choose a t so far everything I'm saying is fairly common sensical but this is where

s62
00:07:09.919 --> 00:07:16.240
um intuition will fail us we might say something like okay I don't know let's do 65%.

s63
00:07:16.240 --> 00:07:21.840
It seems you know gut feeling like okay it's kind of confident that's probably when we should speak.

s64
00:07:21.840 --> 00:07:26.720
Um now here's where we start coming out with some sophistication.

s65
00:07:26.720 --> 00:07:30.240
Any tea we choose is producing bad outcomes.

s66
00:07:30.240 --> 00:07:32.960
Bad in the in two fields.

s67
00:07:32.960 --> 00:07:37.199
One is obviously on the left side anytime you stop it's bad.

s68
00:07:37.199 --> 00:07:38.880
The user doesn't want it to stop.

s69
00:07:38.880 --> 00:07:41.280
He wants to they want to hear their song.

s70
00:07:41.280 --> 00:07:47.440
The other bad is if you do play a wrong song, of course that's bad as well.

s71
00:07:47.440 --> 00:07:50.479
So these are two two kinds of bad outcomes.

s72
00:07:50.479 --> 00:07:52.479
But here's the important part.

s73
00:07:52.479 --> 00:07:58.000
Now I've like elaborated on the um tree diagram on the right hand side.

s74
00:07:58.000 --> 00:08:01.440
The bad outcomes are not equally bad.

s75
00:08:01.440 --> 00:08:03.919
They're not the same thing from a user perspective.

s76
00:08:03.919 --> 00:08:06.639
And obviously let's let's think about it.

s77
00:08:06.639 --> 00:08:16.160
If the wrong song plays, you said play kiss and it starts playing kiss by Chris Brown instead of the one by Prince.

s78
00:08:16.160 --> 00:08:20.639
That's going to be um the highest user cost.

s79
00:08:20.639 --> 00:08:23.599
Now I'm defining user cost from the user's perspective.

s80
00:08:23.599 --> 00:08:27.360
First I have to like hear music and realize that is not Prince.

s81
00:08:27.360 --> 00:08:34.000
Then I have to shout over my Alexa and um you know get it to stop and then I have to re-request.

s82
00:08:34.000 --> 00:08:35.919
All of that is a lot of effort.

s83
00:08:35.919 --> 00:08:53.360
that is definitely a worse outcome than the system stopping and saying sorry I didn't understand that however we should go further and try to quantify that relative badness and there many ways to do it and I think this is an area to be explored for now let's just consider this a heristic

s84
00:08:53.360 --> 00:09:04.240
of if that outcome happens how many more seconds additional seconds will it take for the user to get back to success which is to play the song they wanted kiss by Prince

s85
00:09:04.240 --> 00:09:07.279
and I'm I just put down some numbers here.

s86
00:09:07.279 --> 00:09:11.920
Let's say in the case of a bad song, it's 10 seconds if you add up all the things I got to do.

s87
00:09:11.920 --> 00:09:19.519
And if it's a I didn't understand you, it's 4 seconds because that's how long it would take you to respe and and the extra latency.

s88
00:09:19.519 --> 00:09:23.920
And now here's where we can start utilizing that.

s89
00:09:23.920 --> 00:09:34.640
If we go back to our distribution curve on trying to find out where is T. Now we've basically turned this into a problem of minimizing a cost function.

s90
00:09:34.640 --> 00:09:36.080
It's a user cost function.

s91
00:09:36.080 --> 00:09:46.560
It is the number of bad acts wherever that whatever the t causes times 10 because that was a unit cost we gave plus the number of stops times four because that's the the

s92
00:09:46.560 --> 00:09:48.160
unit cost we gave.

s93
00:09:48.160 --> 00:09:54.399
By the way, one thing I should have elaborated because I work in voice and we like language and we like puns.

s94
00:09:54.399 --> 00:09:59.040
So this whole thing is called an outcome user cost heruristic.

s95
00:09:59.040 --> 00:10:03.839
So that spells the word ouch and that is some expression of pain.

s96
00:10:03.839 --> 00:10:06.000
Yes, we are you know language nerds.

s97
00:10:06.000 --> 00:10:07.920
So these kinds of things amuse us.

s98
00:10:07.920 --> 00:10:12.560
Um so now let's consider that is the cost function is to minimize the ouch.

s99
00:10:12.560 --> 00:10:17.680
And now um that let's see if uh I'm going to bring up a tool.

s100
00:10:17.680 --> 00:10:20.399
Let's see if this works.

s101
00:10:20.399 --> 00:10:36.240
Where I have actually gotten or with one of my coding assistants gotten uh an interactive um graph where we have actually plotted those thousand points and as we vary

s102
00:10:36.240 --> 00:10:46.480
the threshold t you can see that the total user cost here which is that function of you know x * y + a * b actually changes.

s103
00:10:46.480 --> 00:11:01.120
So let's in the very beginning when we said the system was just playing the the cost across those thousand points was 2100 or divided by a,000 is 2.1 ouch points per turn.

s104
00:11:01.120 --> 00:11:07.760
Then we said okay let's insert a stop behavior and let's like wing it and say 65%.

s105
00:11:07.760 --> 00:11:09.600
That's when I want the threshold.

s106
00:11:09.600 --> 00:11:12.959
If we brought this up to 65 yeah that's better.

s107
00:11:12.959 --> 00:11:17.360
Now it's 1904 or 1.9 per turn, but it's not optimal.

s108
00:11:17.360 --> 00:11:27.600
As it turns out, if we do actually um ask for the AI to solve the uh the problem across this curve, it turns out 43%.

s109
00:11:27.600 --> 00:11:40.399
So I'll drag it now to 43 is in fact the optimal optimal point of t. This minimizes the cost function.

s110
00:11:40.399 --> 00:11:45.760
You can see it's the lowest point on this graph down here to 1 27.

s111
00:11:45.760 --> 00:11:49.440
So effectively we haven't changed the accuracy at all.

s112
00:11:49.440 --> 00:11:51.920
The system is not any smarter in that sense.

s113
00:11:51.920 --> 00:12:00.480
But with some clever system behavior, conversational behavior is what we'd call it and some optimization and a cost function called ouch.

s114
00:12:00.480 --> 00:12:06.639
Um we have from the user's perspective produced a more satisfactory assistant.

s115
00:12:06.639 --> 00:12:09.600
And this is not a trivial you know accomplishment.

s116
00:12:09.600 --> 00:12:10.000
Okay.

s117
00:12:10.000 --> 00:12:12.298
Now, let me go back to this.

s118
00:12:12.298 --> 00:12:14.639
[clears throat] Let me see if I can get this.

s119
00:12:14.639 --> 00:12:16.079
Oh, great.

s120
00:12:16.079 --> 00:12:18.240
Okay, let's continue this.

s121
00:12:18.240 --> 00:12:22.079
Let's continue this with by now adding one more behavior.

s122
00:12:22.079 --> 00:12:23.680
Let's call it the confirm behavior.

s123
00:12:23.680 --> 00:12:26.639
So, there was play obviously, then stop, confirm.

s124
00:12:26.639 --> 00:12:36.800
Confirm is basically the system after you said something saying uh kiss play kiss by Prince or maybe play kiss by Chris Brown.

s125
00:12:36.800 --> 00:12:41.600
And uh you know the user can either confirm like affirm it or they can correct it.

s126
00:12:41.600 --> 00:12:45.760
It is a different kind of behavior and again this is kind of how humans behave.

s127
00:12:45.760 --> 00:12:48.160
Um that's obviously the inspiration.

s128
00:12:48.160 --> 00:12:56.959
Now if we go back to our problem of optimization, we have a third obviously um option which is to confirm.

s129
00:12:56.959 --> 00:13:11.360
And so this would translate to two thresholds um two thresholds which are separating the distribution into three spaces of stop, confirm and uh act.

s130
00:13:11.360 --> 00:13:14.639
And the question is now where are these T's?

s131
00:13:14.639 --> 00:13:18.240
and we have now given up on guesstimating because we know it doesn't work.

s132
00:13:18.240 --> 00:13:29.440
So we're going to be a lot smarter and go back to the concept of user outcome cost and then you know use it go look for some optimization in that graph.

s133
00:13:29.440 --> 00:13:37.200
So let's uh define what are the what are all the possible bad outcomes that t1 and t2 um make for.

s134
00:13:37.200 --> 00:13:39.519
So good you can see my cursor.

s135
00:13:39.519 --> 00:13:43.440
So uh of course any stops are still bad.

s136
00:13:43.440 --> 00:13:46.399
Then in the middle are confirmations.

s137
00:13:46.399 --> 00:13:49.279
Confirmations are bad because they slow the user down.

s138
00:13:49.279 --> 00:13:57.680
There is a confirmation outcome called confirm yes where they just affirmed it by saying yeah or no where they had to correct it.

s139
00:13:57.680 --> 00:14:04.079
And going back to our formula these outcomes are not equally bad.

s140
00:14:04.079 --> 00:14:08.959
And in fact, nobody will, I think, argue here from a user's perspective.

s141
00:14:08.959 --> 00:14:16.880
Affirming, just saying yes is obviously less painful than saying no and then having to restate whatever it is that you wanted in the first place.

s142
00:14:16.880 --> 00:14:19.279
So now we I've assigned values of two or six.

s143
00:14:19.279 --> 00:14:21.279
And again, I said it was a heristic.

s144
00:14:21.279 --> 00:14:26.720
This would be roughly the amount of time it would take for the extra for the user to get to the song they want.

s145
00:14:26.720 --> 00:14:30.000
Saying listening and then saying yes is like two seconds.

s146
00:14:30.000 --> 00:14:47.199
Um and then now uh we restate the cost function for this you know added behavior as this number of you know bad type one times unit cost bad type plus bad type two times unit cost etc.

s147
00:14:47.199 --> 00:14:53.760
And now we try to minimize this user cost function and minimize the ouch.

s148
00:14:53.760 --> 00:14:56.160
Yes, I'm going to keep doing that pun.

s149
00:14:56.160 --> 00:14:58.959
Um let's go back.

s150
00:14:58.959 --> 00:15:19.760
So this is now the interactive graph but with um the cost values the unit costs here 10264 and uh you know we're just going to ask the AI to tell us here's the heat map because it's now two dimensions saying that the optimal values

s151
00:15:19.760 --> 00:15:31.519
are 41 for the the T1 and 49 for the T2 and if we employed that then we would go to 1464.

s152
00:15:31.519 --> 00:15:37.839
Uh, by the way, whatever numbers I put in here, like let's say I thought wrong act was 20.

s153
00:15:37.839 --> 00:15:44.160
It's really irritating and painful and takes way longer to actually correct it when you hear a wrong song.

s154
00:15:44.160 --> 00:15:48.399
That would change you know all these numbers uh and the optim optimal point.

s155
00:15:48.399 --> 00:15:55.600
So again it is about how what is the relative badness of these outcomes also of course the distribution curves naturally.

s156
00:15:55.600 --> 00:15:57.759
Uh let's go back here.

s157
00:15:57.759 --> 00:15:58.240
Okay.

s158
00:15:58.240 --> 00:16:04.880
So, um I'm gonna speed up a little bit.

s159
00:16:04.880 --> 00:16:07.759
Uh let's go back here.

s160
00:16:07.759 --> 00:16:09.839
Presentation mode.

s161
00:16:09.839 --> 00:16:10.399
Okay.

s162
00:16:10.399 --> 00:16:22.000
So, what have we shown that if we did the super naive approach, it's 2.1 act and stop 1.9 then 1.27 then 1.26.

s163
00:16:22.000 --> 00:16:31.839
We are able to bring this with every added layer of sophistication, adding more behaviors, being smart about outcome, uh, user cost and optimizing.

s164
00:16:31.839 --> 00:16:37.199
Um, we have made a tremendous difference without changing the accuracy at all.

s165
00:16:37.199 --> 00:16:39.839
Um, this was a super simplified example.

s166
00:16:39.839 --> 00:16:45.600
In real systems, you're not going to have obviously some offline decision threshold or two.

s167
00:16:45.600 --> 00:16:48.800
It's going to be a real time, you know, learned decision model.

s168
00:16:48.800 --> 00:16:50.240
But the principle is the same.

s169
00:16:50.240 --> 00:16:55.920
And I believe this is uh scalable across all voice AI surfaces.

s170
00:16:55.920 --> 00:17:03.920
Obviously this is a smart speaker but if we go across any of these surfaces you will find the equivalence.

s171
00:17:03.920 --> 00:17:14.720
If we um we will find the analogies with some differences but the spirit and the I think the the gain will be similar.

s172
00:17:14.720 --> 00:17:30.080
So just for example in the TV AI assistant space if you employ it here it's going to you're going to have the same thing when users express intents like on TV it's you know open a channel that's one of the most common obviously

s173
00:17:30.080 --> 00:17:48.880
um requests on a TV voice assistant same thing you're going to find you'll have exactly the same approach but the difference will be maybe in the the assignments of the user outcomes because the UI and the modalities are different when you have a TV you have a multimodal interface where

s174
00:17:48.880 --> 00:17:50.160
choices can be shown.

s175
00:17:50.160 --> 00:18:09.440
So instead of you know asking did you mean ABC you know uh news live by speech that you will the system would display choices and not just one it show ABC News live this that would be the confirm step and if it's visual and you can use your remote control to select something

s176
00:18:09.440 --> 00:18:23.280
it's less pain so you would change some of these values or if in fact launching the channel would kick you out of your current state then it would go in the other direction than cost of you know a bad act would go much higher.

s177
00:18:23.280 --> 00:18:40.320
So it's the same concept but in this new modalities um variables can change, values can change, arguments can change but the premise still holds and you can improve from the user's perspective because we're all about you know making humans happy.

s178
00:18:40.320 --> 00:18:49.360
Um you can make them happier and this as I said in conclusion can be applied across all surfaces.

s179
00:18:49.360 --> 00:18:57.440
I did say at the very beginning, just to recap for us, that voice is great when it works, bad when it doesn't.

s180
00:18:57.440 --> 00:19:14.640
And as we get into embodied AI, where these AI assistants are taking actions, physical or even digital, like making a phone call or sending an email, it is getting more and more difficult just to rely on accuracy to improve user satisfaction.

s181
00:19:14.640 --> 00:19:30.640
I believe there's a whole knob the second knob called smarter conversational behavior under uncertainty and um if we actually exploit that we can uh very much help these AI systems

s182
00:19:30.640 --> 00:19:43.039
reach a acceptable user experience otherwise I think this will continue to be a bottleneck like a lot of things will get better but if the voice interface as experienced by user

s183
00:19:43.039 --> 00:19:47.600
does not improve it is going to be a a a choke point.

s184
00:19:47.600 --> 00:19:59.679
And um if you just remember one word or two words from this whole um presentation, it would be to minimize the ouch of the experience.

s185
00:19:59.679 --> 00:20:01.520
Um so thank you.

s186
00:20:01.520 --> 00:20:04.640
I'll stick around for questions if you guys got any.

s187
00:20:04.640 --> 00:20:06.523
Thanks a lot.

s188
00:20:06.523 --> 00:20:08.523
[applause]

s189
00:20:21.589 --> 00:20:23.589
[music]