WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=IDNfAZVKvPE

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/IDNfAZVKvPE.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:15.440
Okay, hello everyone.

s3
00:00:16.120 --> 00:00:21.160
So, my name is Midam Kim.

s4
00:00:21.160 --> 00:00:30.960
I am an ML engineer from ServiceNow and I'll be talking about a linguistic framework for voice AI.

s5
00:00:33.240 --> 00:00:39.520
So, quick background of me so you know where I'm coming from.

s6
00:00:39.520 --> 00:00:47.800
Like I said, I'm an ML engineer at ServiceNow, but I'm also a researcher, lifelong researcher, of speech communication in the wild.

s7
00:00:47.800 --> 00:00:58.360
So, my motto is doing linguistics and what I'm going to be doing today is to hand you that lens of linguistics.

s8
00:01:00.440 --> 00:01:05.200
So, have you experienced voice AI failures?

s9
00:01:05.239 --> 00:01:06.810
Yeah, like everyone.

s10
00:01:06.810 --> 00:01:08.810
[laughter]

s11
00:01:10.240 --> 00:01:15.200
So, I'm going to introduce an example that I experienced myself.

s12
00:01:15.200 --> 00:01:20.560
So, the bot asked me, "Could you please spell your first name?"

s13
00:01:20.560 --> 00:01:24.480
And then I slowly start to spell my name.

s14
00:01:24.480 --> 00:01:35.800
Yes, it is m i d a m. And the bot says, "Confirming with you, is it m i d a n?"

s15
00:01:35.800 --> 00:01:45.600
And then I say, "No, it is m i d a m." Um and the bot says, "Thank you for your correction.

s16
00:01:45.600 --> 00:01:48.840
Happy to help you today, Madam."

s17
00:01:48.840 --> 00:01:55.240
And I then I get slightly annoyed, more annoyed, because my name is Midam, not Madam.

s18
00:01:55.240 --> 00:01:59.560
And then it asked me about, "Now, what is your account number?

s19
00:01:59.560 --> 00:02:01.760
And then, I start start getting confused.

s20
00:02:01.760 --> 00:02:04.680
What is that account number thing?

s21
00:02:04.680 --> 00:02:10.119
And then, I try to find uh information about that.

s22
00:02:10.119 --> 00:02:12.920
So, which one?

s23
00:02:12.920 --> 00:02:20.840
Um it must be and I start uh slowly start spelling the account number.

s24
00:02:20.840 --> 00:02:32.240
So, it is A X 4 5 1. And then, I take time because I'm not used to reading this strange number.

s25
00:02:32.240 --> 00:02:34.880
And then, the bot cuts me off.

s26
00:02:34.880 --> 00:02:37.560
And then, it says, I couldn't find your record.

s27
00:02:37.560 --> 00:02:41.880
And then, without even trying, it asked me to repeat that again.

s28
00:02:41.880 --> 00:02:43.560
Can you please repeat that?

s29
00:02:43.560 --> 00:02:48.160
And then, I get super annoyed and then, I can say, can I talk to a person?

s30
00:02:48.160 --> 00:02:50.680
I just don't want to deal with you anymore.

s31
00:02:50.680 --> 00:02:56.160
So, this is a very typical pattern of voice AI, unfortunately, at this point.

s32
00:02:56.160 --> 00:03:03.920
So, I just want to navigate how we can solve this problem with linguistics.

s33
00:03:06.120 --> 00:03:08.160
So, voice AI is booming.

s34
00:03:08.160 --> 00:03:13.640
But users are still often preferring human agents over voice agents.

s35
00:03:13.640 --> 00:03:17.000
How can we mitigate this issue?

s36
00:03:17.000 --> 00:03:21.560
But in the first place, what are the actual problems?

s37
00:03:21.600 --> 00:03:32.680
So, I think we can think about a fundamental frame framework to understand this into an architecture of voice AI, which is called linguistics.

s38
00:03:34.280 --> 00:03:43.959
So, as all of us already know, human communication is a joint activity, like the thing that we're doing right now.

s39
00:03:43.959 --> 00:03:47.800
So, I give you my sounds and words.

s40
00:03:47.800 --> 00:03:49.360
You hear them.

s41
00:03:49.360 --> 00:03:55.600
And then, if it is a conversation, you're going to give me your sounds and your words.

s42
00:03:55.680 --> 00:04:00.440
And then this is going back and forth through interaction.

s43
00:04:01.320 --> 00:04:09.080
And then in this process, we're continuously processing and updating our mental models.

s44
00:04:09.280 --> 00:04:14.800
So that's a joint activity for human communication.

s45
00:04:14.800 --> 00:04:24.480
And I would like to say in the voice AI human communication, it also has to be a joint activity like this.

s46
00:04:24.480 --> 00:04:30.160
Because that's the only thing that we know about human communication as a human being.

s47
00:04:30.160 --> 00:04:35.880
We have been evolving thousands of years as communicators, and this is what we know.

s48
00:04:35.880 --> 00:04:39.600
So we expect the same thing to bots.

s49
00:04:41.200 --> 00:04:49.240
So let me go over the failure scene of my call with the voice agent in this framework.

s50
00:04:49.240 --> 00:04:55.800
So you see there's listen and speak for each party.

s51
00:04:57.240 --> 00:05:00.240
So I start spelling my first name.

s52
00:05:00.240 --> 00:05:12.840
And then the bot did not hear that the difference between M and N correctly, so it's an SCT failure in the listening level.

s53
00:05:12.840 --> 00:05:25.000
And then the TTS applies only English-centric reading rules to my name, M I D A M, would read it as Midam in the American English version.

s54
00:05:25.480 --> 00:05:31.840
So I'm confused, but at this time I'm kind of generous because that happens a lot even with human beings.

s55
00:05:31.840 --> 00:05:34.160
So I'm okay.

s56
00:05:34.160 --> 00:05:42.960
But then when it brought brought up account number thing because I don't know what that is, I'm confused again.

s57
00:05:42.960 --> 00:05:46.280
But I'm adaptive, I can find I can look for it.

s58
00:05:46.280 --> 00:06:03.400
So I found the number, start reading it, but the STT did not recognize the word unit correctly, so it cuts me off, and uh uh finally, it's uh eventually talked over me.

s59
00:06:03.400 --> 00:06:08.160
So, I get really irritated.

s60
00:06:08.240 --> 00:06:18.400
And then, when it asked me for the repetition of the same information, and then, it is clear that the spot is not tracking the mental model with me.

s61
00:06:18.400 --> 00:06:27.919
And then, very rudely, it's uh does not even try interactive clarification, which is a common strategy by human beings.

s62
00:06:27.919 --> 00:06:32.840
So, I don't want to deal with this anymore, so I say, "Can I talk to a person?"

s63
00:06:34.080 --> 00:06:38.040
So, let's go over the uh the framework again.

s64
00:06:38.040 --> 00:06:47.520
So, the these are the linguistic components that are expected and well maintained in human-to-human voi- uh uh conversation.

s65
00:06:47.520 --> 00:06:55.680
So, there are listening channels, a listening channel and speaking channel, and there are different components like sounds, words, interaction, and mental model.

s66
00:06:55.680 --> 00:07:01.520
So, the first component is, does the bot recognize the user's speech well?

s67
00:07:01.520 --> 00:07:07.120
And all of these technical terms uh will fall under this.

s68
00:07:07.120 --> 00:07:13.000
And then, there was there's going to be this second component, which is words in the listening channel.

s69
00:07:13.000 --> 00:07:17.120
So, does the bot understand the user's words?

s70
00:07:17.400 --> 00:07:22.720
And then, the third one is, does the bot wait until the right timing to for its turn?

s71
00:07:22.720 --> 00:07:27.560
It's about It's going to be about uh listening channel interaction.

s72
00:07:28.640 --> 00:07:32.080
And then, uh the last part is mental model.

s73
00:07:32.080 --> 00:07:38.240
So, does the bot understand the user's intention in the listening part?

s74
00:07:38.280 --> 00:07:43.880
And then, we can also go to the speaking channel, so it's going to be about pronunciation for the sound.

s75
00:07:43.880 --> 00:07:53.560
And also there's about understand the the words users are uh there's about choose the words the user can understand.

s76
00:07:53.640 --> 00:07:58.800
And in the interaction part, there's about speak with the right timing.

s77
00:07:58.800 --> 00:08:04.680
And lastly, there's about speak with the information the user actually need.

s78
00:08:05.640 --> 00:08:13.120
So, there are a lot of engineering or linguistic or cognitive science terms that are in here that that are here.

s79
00:08:13.120 --> 00:08:21.320
Um you can see now see that all of those have their right spots in this linguistic framework.

s80
00:08:23.200 --> 00:08:30.080
And importantly, these components are interdependent, not separate or uh independent from each other.

s81
00:08:30.080 --> 00:08:33.159
They're interdependent and they're aligned.

s82
00:08:33.159 --> 00:08:39.719
So, when you want to do good things about sounds, you have to think about words level.

s83
00:08:39.719 --> 00:08:50.320
And then when you want to do good things about these sounds and words, you also have to uh account for interaction, so turn taking or turn detection.

s84
00:08:50.320 --> 00:08:59.520
And then finally, you want to uh have good uh task completion, which is the goal of these mental model uh layer.

s85
00:08:59.520 --> 00:09:02.520
Then you have to have all of these.

s86
00:09:02.520 --> 00:09:09.880
Without all of those, without any of those, any of those components, your voice agent will fail.

s87
00:09:09.880 --> 00:09:13.320
And then finally, uh it has to be well aligned.

s88
00:09:13.320 --> 00:09:16.440
All of these have to be well aligned.

s89
00:09:17.080 --> 00:09:25.720
And additionally, you have to keep your mind keep in mind that this is happening on the timeline.

s90
00:09:26.240 --> 00:09:30.680
What I mean by that is it is silently tracked.

s91
00:09:30.680 --> 00:09:37.200
Unlike in chat, in chat you see the history of what was said uh as text.

s92
00:09:37.200 --> 00:09:52.280
But in voice agent experience, uh, you say something, and the bot says something, you go back and forth, and then see, all these waveforms, the air via the vibration in the, uh, in the air,

s93
00:09:52.280 --> 00:09:53.760
they're all gone.

s94
00:09:53.760 --> 00:10:00.280
And only the user's mental model is the thing that's left, and that matters.

s95
00:10:00.480 --> 00:10:09.120
So, sounds, words, interactions vanish the moment they're spoken, but the mental model proceeds and grows over the timeline.

s96
00:10:09.120 --> 00:10:16.560
So, this is what you have to target for user satisfaction.

s97
00:10:16.560 --> 00:10:24.440
And then, what can we do for the bot to meet the standard of the user?

s98
00:10:25.160 --> 00:10:41.440
So, what we can do, uh, would include, of course, choosing good ASR models or configurations and do some post-processing, uh, choosing good TTS models, configurations, and pre-processing, and, uh, carefully curate

s99
00:10:41.440 --> 00:10:52.760
the vocabulary that can be shared between the bot and the user, and do good job of a turn-to-turn detection, latency, and turn-taking.

s100
00:10:52.800 --> 00:11:05.520
Um, and very importantly, we have to, it would be great if we can do good emotion detection and handling, and context retention, and by context, what I mean is context about all of these.

s101
00:11:07.880 --> 00:11:16.680
And importantly, uh, it has to be dynamic because things are always changing, uh, throughout over the course of the call.

s102
00:11:16.680 --> 00:11:23.960
So, we would have to do this management dynamically along the timeline for different kinds of people.

s103
00:11:23.960 --> 00:11:32.000
So, kids or different kinds of people like these will have different expectations that we have to satisfy.

s104
00:11:32.000 --> 00:11:36.680
Uh, not just when they're happy, but also when they're not happy.

s105
00:11:36.680 --> 00:11:43.120
So, only then you can pursue a dynamic and truly scalable orchestration of voice AI.

s106
00:11:43.120 --> 00:11:46.640
So, it's a very difficult job to do.

s107
00:11:48.240 --> 00:11:54.320
We always say that voice is the most natural way of communication, but it is not actually not easy.

s108
00:11:54.320 --> 00:11:59.200
Behind the scene, it is thanks to this linguistic orchestration.

s109
00:11:59.200 --> 00:12:05.280
When your bot is not good at it, it's a catastrophic failure.

s110
00:12:06.960 --> 00:12:22.040
Um, so paying attention to this linguistic framework would have lots of business implications because then you can uh decrease all of these user frustration, task failures, live agent escalation,

s111
00:12:22.040 --> 00:12:25.800
or abandoned calls, or silent failures.

s112
00:12:28.000 --> 00:12:35.040
So, in ServiceNow, we have made a a good uh benchmark end-to-end benchmark called Eva bench.

s113
00:12:35.040 --> 00:12:40.320
So, you can try that to diagnose your voice agent's uh status.

s114
00:12:43.400 --> 00:12:45.920
Um, key takeaways.

s115
00:12:45.960 --> 00:12:54.080
So, voice AI is a joint activity between the bot and the user, not just a pipeline.

s116
00:12:54.240 --> 00:12:59.280
And we must serve users' needs in multiple layers real time.

s117
00:12:59.760 --> 00:13:04.320
It's not that I have given you a fix today because there's nothing like that.

s118
00:13:04.320 --> 00:13:09.560
It just uh the fix is in you and your system.

s119
00:13:09.920 --> 00:13:21.560
But, what I have given you is today is the linguistic framework you can try to diagnose your system and to build your system upon.

s120
00:13:21.560 --> 00:13:27.120
You can try Eva, but also you can learn linguistics and hire linguists.

s121
00:13:27.680 --> 00:13:38.000
Um, another thing I want to remind you of is that business implications are linguistic implications and vice versa in this voice AI scene.

s122
00:13:38.000 --> 00:13:45.040
Because voice is fundamentally a linguistic and very human and cognitive experience.

s123
00:13:46.000 --> 00:13:50.200
I would like to ask you a longer term question.

s124
00:13:50.720 --> 00:13:52.440
Speakers adapt.

s125
00:13:52.440 --> 00:14:05.560
So, I I'm pretty sure that in this talk in my talk with you guys today, you have learned something about me, about my speaking style, what kind of accents I speak,

s126
00:14:05.560 --> 00:14:07.360
what kind of words I'm using.

s127
00:14:07.360 --> 00:14:15.360
So, next time I see you guys in person, you would find it more comfortable to talk to me because you have paid attention to me.

s128
00:14:15.360 --> 00:14:15.600
Right?

s129
00:14:15.600 --> 00:14:17.600
So, speakers are always adapting.

s130
00:14:17.600 --> 00:14:23.440
So, the user will be adapting to your voice agent throughout the call.

s131
00:14:23.440 --> 00:14:31.480
So, is your system ready for them to use you better, use it your voice agent better the next time?

s132
00:14:31.480 --> 00:14:34.600
And language is always change.

s133
00:14:34.600 --> 00:14:41.760
So, is your voice agent ready for language change in 1 year or 6 months even?

s134
00:14:44.360 --> 00:14:47.160
So, thank you.

s135
00:14:47.535 --> 00:14:49.535
[applause]