"My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow

AI Engineer · 15 min · 135 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Okay, hello everyone.
00:16So, my name is Midam Kim.
00:21I am an ML engineer from ServiceNow and I'll be talking about a linguistic framework for voice AI.
00:33So, quick background of me so you know where I'm coming from.
00:39Like I said, I'm an ML engineer at ServiceNow, but I'm also a researcher, lifelong researcher, of speech communication in the wild.
00:47So, my motto is doing linguistics and what I'm going to be doing today is to hand you that lens of linguistics.
01:00So, have you experienced voice AI failures?
01:05Yeah, like everyone.
01:06[laughter]
01:10So, I'm going to introduce an example that I experienced myself.
01:15So, the bot asked me, "Could you please spell your first name?"
01:20And then I slowly start to spell my name.
01:24Yes, it is m i d a m. And the bot says, "Confirming with you, is it m i d a n?"
01:35And then I say, "No, it is m i d a m." Um and the bot says, "Thank you for your correction.
01:45Happy to help you today, Madam."
01:48And I then I get slightly annoyed, more annoyed, because my name is Midam, not Madam.
01:55And then it asked me about, "Now, what is your account number?
01:59And then, I start start getting confused.
02:01What is that account number thing?
02:04And then, I try to find uh information about that.
02:10So, which one?
02:12Um it must be and I start uh slowly start spelling the account number.
02:20So, it is A X 4 5 1. And then, I take time because I'm not used to reading this strange number.
02:32And then, the bot cuts me off.
02:34And then, it says, I couldn't find your record.
02:37And then, without even trying, it asked me to repeat that again.
02:41Can you please repeat that?
02:43And then, I get super annoyed and then, I can say, can I talk to a person?
02:48I just don't want to deal with you anymore.
02:50So, this is a very typical pattern of voice AI, unfortunately, at this point.
02:56So, I just want to navigate how we can solve this problem with linguistics.
03:06So, voice AI is booming.
03:08But users are still often preferring human agents over voice agents.
03:13How can we mitigate this issue?
03:17But in the first place, what are the actual problems?
03:21So, I think we can think about a fundamental frame framework to understand this into an architecture of voice AI, which is called linguistics.
03:34So, as all of us already know, human communication is a joint activity, like the thing that we're doing right now.
03:43So, I give you my sounds and words.
03:47You hear them.
03:49And then, if it is a conversation, you're going to give me your sounds and your words.
03:55And then this is going back and forth through interaction.
04:01And then in this process, we're continuously processing and updating our mental models.
04:09So that's a joint activity for human communication.
04:14And I would like to say in the voice AI human communication, it also has to be a joint activity like this.
04:24Because that's the only thing that we know about human communication as a human being.
04:30We have been evolving thousands of years as communicators, and this is what we know.
04:35So we expect the same thing to bots.
04:41So let me go over the failure scene of my call with the voice agent in this framework.
04:49So you see there's listen and speak for each party.
04:57So I start spelling my first name.
05:00And then the bot did not hear that the difference between M and N correctly, so it's an SCT failure in the listening level.
05:12And then the TTS applies only English-centric reading rules to my name, M I D A M, would read it as Midam in the American English version.
05:25So I'm confused, but at this time I'm kind of generous because that happens a lot even with human beings.
05:31So I'm okay.
05:34But then when it brought brought up account number thing because I don't know what that is, I'm confused again.
05:42But I'm adaptive, I can find I can look for it.
05:46So I found the number, start reading it, but the STT did not recognize the word unit correctly, so it cuts me off, and uh uh finally, it's uh eventually talked over me.
06:03So, I get really irritated.
06:08And then, when it asked me for the repetition of the same information, and then, it is clear that the spot is not tracking the mental model with me.
06:18And then, very rudely, it's uh does not even try interactive clarification, which is a common strategy by human beings.
06:27So, I don't want to deal with this anymore, so I say, "Can I talk to a person?"
06:34So, let's go over the uh the framework again.
06:38So, the these are the linguistic components that are expected and well maintained in human-to-human voi- uh uh conversation.
06:47So, there are listening channels, a listening channel and speaking channel, and there are different components like sounds, words, interaction, and mental model.
06:55So, the first component is, does the bot recognize the user's speech well?
07:01And all of these technical terms uh will fall under this.
07:07And then, there was there's going to be this second component, which is words in the listening channel.
07:13So, does the bot understand the user's words?
07:17And then, the third one is, does the bot wait until the right timing to for its turn?
07:22It's about It's going to be about uh listening channel interaction.
07:28And then, uh the last part is mental model.
07:32So, does the bot understand the user's intention in the listening part?
07:38And then, we can also go to the speaking channel, so it's going to be about pronunciation for the sound.
07:43And also there's about understand the the words users are uh there's about choose the words the user can understand.
07:53And in the interaction part, there's about speak with the right timing.
07:58And lastly, there's about speak with the information the user actually need.
08:05So, there are a lot of engineering or linguistic or cognitive science terms that are in here that that are here.
08:13Um you can see now see that all of those have their right spots in this linguistic framework.
08:23And importantly, these components are interdependent, not separate or uh independent from each other.
08:30They're interdependent and they're aligned.
08:33So, when you want to do good things about sounds, you have to think about words level.
08:39And then when you want to do good things about these sounds and words, you also have to uh account for interaction, so turn taking or turn detection.
08:50And then finally, you want to uh have good uh task completion, which is the goal of these mental model uh layer.
08:59Then you have to have all of these.
09:02Without all of those, without any of those, any of those components, your voice agent will fail.
09:09And then finally, uh it has to be well aligned.
09:13All of these have to be well aligned.
09:17And additionally, you have to keep your mind keep in mind that this is happening on the timeline.
09:26What I mean by that is it is silently tracked.
09:30Unlike in chat, in chat you see the history of what was said uh as text.
09:37But in voice agent experience, uh, you say something, and the bot says something, you go back and forth, and then see, all these waveforms, the air via the vibration in the, uh, in the air,
09:52they're all gone.
09:53And only the user's mental model is the thing that's left, and that matters.
10:00So, sounds, words, interactions vanish the moment they're spoken, but the mental model proceeds and grows over the timeline.
10:09So, this is what you have to target for user satisfaction.
10:16And then, what can we do for the bot to meet the standard of the user?
10:25So, what we can do, uh, would include, of course, choosing good ASR models or configurations and do some post-processing, uh, choosing good TTS models, configurations, and pre-processing, and, uh, carefully curate
10:41the vocabulary that can be shared between the bot and the user, and do good job of a turn-to-turn detection, latency, and turn-taking.
10:52Um, and very importantly, we have to, it would be great if we can do good emotion detection and handling, and context retention, and by context, what I mean is context about all of these.
11:07And importantly, uh, it has to be dynamic because things are always changing, uh, throughout over the course of the call.
11:16So, we would have to do this management dynamically along the timeline for different kinds of people.
11:23So, kids or different kinds of people like these will have different expectations that we have to satisfy.
11:32Uh, not just when they're happy, but also when they're not happy.
11:36So, only then you can pursue a dynamic and truly scalable orchestration of voice AI.
11:43So, it's a very difficult job to do.
11:48We always say that voice is the most natural way of communication, but it is not actually not easy.
11:54Behind the scene, it is thanks to this linguistic orchestration.
11:59When your bot is not good at it, it's a catastrophic failure.
12:06Um, so paying attention to this linguistic framework would have lots of business implications because then you can uh decrease all of these user frustration, task failures, live agent escalation,
12:22or abandoned calls, or silent failures.
12:28So, in ServiceNow, we have made a a good uh benchmark end-to-end benchmark called Eva bench.
12:35So, you can try that to diagnose your voice agent's uh status.
12:43Um, key takeaways.
12:45So, voice AI is a joint activity between the bot and the user, not just a pipeline.
12:54And we must serve users' needs in multiple layers real time.
12:59It's not that I have given you a fix today because there's nothing like that.
13:04It just uh the fix is in you and your system.
13:09But, what I have given you is today is the linguistic framework you can try to diagnose your system and to build your system upon.
13:21You can try Eva, but also you can learn linguistics and hire linguists.
13:27Um, another thing I want to remind you of is that business implications are linguistic implications and vice versa in this voice AI scene.
13:38Because voice is fundamentally a linguistic and very human and cognitive experience.
13:46I would like to ask you a longer term question.
13:50Speakers adapt.
13:52So, I I'm pretty sure that in this talk in my talk with you guys today, you have learned something about me, about my speaking style, what kind of accents I speak,
14:05what kind of words I'm using.
14:07So, next time I see you guys in person, you would find it more comfortable to talk to me because you have paid attention to me.
14:15Right?
14:15So, speakers are always adapting.
14:17So, the user will be adapting to your voice agent throughout the call.
14:23So, is your system ready for them to use you better, use it your voice agent better the next time?
14:31And language is always change.
14:34So, is your voice agent ready for language change in 1 year or 6 months even?
14:44So, thank you.
14:47[applause]