1 Trillion Phone Calls/yr, 10% Error rate: The Crisis in Voice AI — Sumanyu Sharma, Hamming AI

AI Engineer · 16 min · 184 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12My name is Suman Yu and I'm the founder and CEO of Hamming.
3. 00:17And before working on voice agent reliability and safety, I worked at a company called Citizen out of New York.
4. 00:25Anybody here use Citizen app?
5. 00:28Awesome.
6. 00:30Thank you.
7. 00:31Uh, and at Citizen, we listened to crime, thousands of hours of police radio station data and sent millions of alerts to users in San Francisco, New York, LA, uh, Chicago, Baltimore, and so on.
8. 00:50Some obviously gory and pretty sad.
9. 00:53Uh, but others more funny like a person stealing bags of ice cream from Safeway or report of a man hanging off the side of the house after a woman stole his ladder.
10. 01:07If I actually take a look at the citizen app right now for those who are customers or users, I can see that there is a man yelling at person.
11. 01:16There's indecent exposure.
12. 01:17This is real.
13. 01:18This is real time.
14. 01:19This is, you know, couple hours ago.
15. 01:22These are real-time alerts that we're sending.
16. 01:27Now, voice agents scare me more because they're finally graduating from demos and PC's to production.
17. 01:33We should be super excited, but I'm nervous.
18. 01:34I'm personally nervous.
19. 01:36Uh, they're talking to users at a scale that would make Gary Tan and Polygram proud.
20. 01:43When I got started in voice agent reliability in early 2024, voice was just starting to work.
21. 01:49It was not quite good yet, but it was just starting to work.
22. 01:53You would have to pay me a lot of money for me to stop using, you know, Aqua voice, Super Whisper, uh, Whisper Flow, and so on.
23. 01:59These products are just getting super, super good.
24. 02:01And a big reason is because the underlying infrastructure is getting better, and the orchestration layer is getting meaningfully better.
25. 02:08It's getting much faster to build products and voice experiences that maybe are 60% good in a pretty short period of time, but the long tail is still Hey, Gorov.
26. 02:18the long tail is still uh wise away.
27. 02:21I think speech speech models are getting better.
28. 02:23Um teams are experiment experimenting with hybrid architectures of combining more voicetovoice modalities and also cascading stacks to make the experience reliable but still pretty low latency.
29. 02:39Things are obviously getting better.
30. 02:41Agents are being connected to calendars, CRM, HRs, reservation systems, and so on.
31. 02:46Voice agents can now take actions.
32. 02:50However, reliability is still the number one problem holding back most voice agent deployments at scale.
33. 02:55This is still the number one problem.
34. 02:57This is an example I found on Twitter pretty randomly, you know, two weeks ago and a person is trying to get information for a tradein and gets absolutely confused with the information that they're receiving.
35. 03:09Alex now has to correct for this loss of trust but trying to, you know, call the person and see what see what happened and fix the situation.
36. 03:17Let me see if audio works here.
37. 03:20Screwed up with another customer.
38. 03:22We're getting it fixed, but I got to call him and see if I can work it out.
39. 03:25I'm like, dude, half the time I'm like, I don't know if I'm talking to AI.
40. 03:28I don't know if I'm talking to a person.
41. 03:29It was just confusing, but we got there.
42. 03:32It probably is AI and human.
43. 03:34And
44. 03:37so I think voices sound very confident.
45. 03:39They sound very natural, but the information provided is often, you know, not correct.
46. 03:43That's the biggest problem here.
47. 03:46This example is more personal.
48. 03:47I had booked an appointment with a physician a couple weeks ago or I thought I did.
49. 03:52I showed up to the appointment and turns out I was not actually on the schedule.
50. 03:57So the front desk, you me turned me away.
51. 03:59I wasted 2 hours.
52. 04:01For me, this was a waste of time.
53. 04:02But what if this was actually your parent?
54. 04:06What if this was your grandparent?
55. 04:08What if this appointment was for a procedure instead of a regular checkup?
56. 04:13The costs for these different permutations of the same failure mode can actually be super super high.
57. 04:20Now, let's compare crime to voice agents.
58. 04:22Um, I think observation number one is crime is actually decreasing over time.
59. 04:27This is a good thing and I hope it crosses the x- axis at some point you know in the future.
60. 04:35Voice on the other hand is generally taking off right we're seeing a pretty fast takeoff of voice agents being deployed in production.
61. 04:41There's at least a trillion calls that are done every single year and majority of these will be done by conversational voice agents over the next you know five years.
62. 04:50If you assume a 1% error rate that is still 10 billion incidents per year.
63. 04:54That's a lot.
64. 04:58In practice, we currently monitor 10,000 agents and the error rate is closer to 10% in practice.
65. 05:05These range from agents saying they found the right policy when they actually skipped the eligibility or verification steps or applying discounts when they were not really supposed to, misharing what the person said, providing incorrect information,
66. 05:18or claiming they booked an appointment when they actually did not, just like it happened for me.
67. 05:23Now, not every single call has an equally, you know, bad cost.
68. 05:28Uh, some range, you know, in the crime land, some range from trash fires, which are kind of funny, annoying, not really hurting somebody.
69. 05:36For a voice equivalent, that would be annoyances like repetition, um, or just sort of not quite understanding what the user is saying.
70. 05:44all the way to safety risks like mass shootings or in the voice agent equivalent, it would be um a drive-thru that's deploying um voice agents at scale like a Taco Bell or McDonald's
71. 05:55and a person orders a vegan burger with peanut allergies.
72. 05:59If one of those two situations are not handled correctly, that is definitely a safety concern at scale.
73. 06:07The other big difference between crime and and voice agent deployments is is crime generally tends to be pretty hyper local, tends to be very decentralized, right?
74. 06:19Things like robbery or motor vehicle theft or lararseny.
75. 06:24They're impacting a finite set of individuals that are involved in that um situation.
76. 06:32On the other hand, voice agents are much more centralized.
77. 06:35a single prompt change or an architecture change can have pretty massive implications downstream for all of the millions of you know users that are um in the crossfire.
78. 06:46So the blast radius is is quite quite massive.
79. 06:49So the natural question is how do you make these incidents much more visible and obvious?
80. 06:53That's the kind of obvious question here.
81. 06:57I'll borrow a framework from a couple of my friends who were OG growth folks at Facebook.
82. 07:02So step one is to identify okay what are all the challenges and problems that um exist in your conversation experience.
83. 07:09Step two is to prioritize an impact size.
84. 07:11There's a frequency and severity analysis that's pretty important.
85. 07:15Step three is to understand okay how do we actually fix this?
86. 07:18Step four execute.
87. 07:20Step five okay did my change actually work and did it cause any regressions somewhere else.
88. 07:25And lastly we continue to monitor in production.
89. 07:30On the y- axis, I think it's important to highlight there are known problems that already exist.
90. 07:36Things like turnover latency, interruptions, um maybe some ASR problems you're, you know, aware of.
91. 07:41And these are known problems that exist that the team should track over time.
92. 07:46On the other axis is actually emerging behavior or patterns that are only obvious across lots of conversations.
93. 07:54Um on the x- axis, you have coverage just like insurance.
94. 07:58Are you analyzing few conversations?
95. 08:01Are you analyzing many, many conversations?
96. 08:03Most teams will typically start by listening to calls manually.
97. 08:08And I think that's the best place to start.
98. 08:10I don't think you should skip that step.
99. 08:12There's a lot of depth and insights to get by actually listening to specific conversations and building that texture that that comes from that intuition.
100. 08:20However, it's obviously not scalable.
101. 08:22So most teams end up having a spreadsheet of I don't know five or 10 different rubrics around greetings, closing, validation, um, core logic and so on.
102. 08:34To scale that up even further, you then end up investing in some eval product, right?
103. 08:39You might run some element as a judge and compute classic metrics and also more more deterministic and stoastic scoring logic.
104. 08:48Um, but there you're still stuck with checking for consistency of known problems, but you're not really discovering novel insights that are actually happening across conversations.
105. 08:58We're spending a ton of time on performing cross conversation analysis, not a pattern on a single call, but across conversations.
106. 09:05And some of the best teams that we work with are are doing the same.
107. 09:10Now, to prioritize an impact size, I think there's problems that are one-off that are low impact.
108. 09:15I mean, who cares?
109. 09:17uh even low impact and systematic problems in the crime world that would be a trash fire in a voice aation world it could be some repetitions the team is experiencing
110. 09:26they're still annoying at scale and if you are doing a bake off it's still worth solving for them I would not ignore these class of problems oneoff and high impact well hope it doesn't chronic
111. 09:37and I think systematic and high impact are obviously the P 0 you know target areas um for the team to solve an example of that would be in a fins serve capacity There's a voice agent
112. 09:48that um helps users freeze their credit cards.
113. 09:51And if it doesn't do that, well, that's a massive fail.
114. 09:56All right.
115. 09:56So, understand and execute.
116. 09:58I'm pretty sure everyone's doing this.
117. 10:00Please fix my agent.
118. 10:01Uh I think fixing or rather attempting to make a fix is the simplest and the lowest effort component of this debugging pipeline and loop.
119. 10:12Um the next step is all right, I made a change to my system.
120. 10:16How do I actually know this thing works um for real?
121. 10:20A great way that's naive is to take a real call, for example, in my case, I booked an appointment and it didn't get scheduled and replay that exact conversation
122. 10:30and run that maybe 5, 10, 20, 50 times and see, okay, what is my probability of passing this type of issue?
123. 10:38A better way is to keep the same intent but change the wordings, change the patterns, change the accents, change the style, add one more intent to the mix.
124. 10:48And that gives teams much more, you know, better coverage to feel confident that yes, I actually made a change and my changes are net positive instead of net negative.
125. 11:01There are certain fixes and I guess hypothesis that are very difficult to test in a pre-eployment synthetic setting.
126. 11:10And so AB testing ends up being, you know, pretty pretty critical for those circumstances.
127. 11:14For example, if you have an outbound agent, the first 5 seconds of a conversation tends to be the most important.
128. 11:21And so the vocal quality um and the specific words you end up using, they matter the most.
129. 11:27And so AB testing that is the only way in in kind of real life setting to to get results.
130. 11:32You can't really do it through simulations alone.
131. 11:37And so there we have the loop.
132. 11:38Identify, prioritize, impact size, understand the fix, execute, check, make sure it didn't break anything, and then continue monitoring.
133. 11:51So I think making voice agents useful is already hard as it is.
134. 11:57even when dealing with earnest users on the other line, right?
135. 12:01These are people who who just want their problem solved.
136. 12:03They're not trying to mess with you.
137. 12:04These are like legit normal people.
138. 12:08Now, what happens when mythos learns how to dial?
139. 12:13So, if it can extract trade secrets and uh you know, from the NSA, it can certainly, you know, seduce you into revealing PHI and PII data as well.
140. 12:26And I think both voice agents and humans will [clears throat] be targeted here.
141. 12:31Voice agents because there's a pressure to make these more capable.
142. 12:36Give them access to more data.
143. 12:38Give them access to more tools.
144. 12:41Deploy them quickly.
145. 12:45The more the capability, the bigger the surface area.
146. 12:47This is this is pretty pretty common sense.
147. 12:50And the more the voice agents become natural and human sounding, the more humans will be tricked along the way as well for those who are weaponizing.
148. 13:01Uh we ship a we shipped a red tipping product um back in April just to test out this hypothesis for how many agents can we actually break from a adversarial capacity
149. 13:11and we can probably break one in five agents at this point.
150. 13:14We've tested this across financial services, healthcare, um consumer and so on.
151. 13:19We've bypassed verification.
152. 13:21Uh we've definitely had agents, you know, we've been able to promject uh several agents and and gotten data we should not have.
153. 13:29So this is not theoretical.
154. 13:30This is actually a real a real concern.
155. 13:37I think the only real defense against the dark arts is step one to invest deeply in pre-eployment testing.
156. 13:45This could be textto text.
157. 13:47This could be voice to voice.
158. 13:48There's pros and cons to both.
159. 13:50Happy to chat offline if folks are interested.
160. 13:53And this is just making sure you're not self-owning, you know, when you're talking to real people who just want to get their problem solved.
161. 13:59Step two is to have a great monitoring system of all kinds.
162. 14:03And I've highlighted, you know, different flavors of monitoring per call scoring, manual kind of evals, you know, listening to conversations and cross call analysis.
163. 14:13And this is helpful both for monitoring what the agent is saying and behaving and how it's actually doing, but also the users.
164. 14:19Are the users being adversarial?
165. 14:21Are they being annoying?
166. 14:21Are are they trying to trick the agent into doing things it's not supposed to be doing?
167. 14:26And I think our new recommendation now is to run 24/7 red teaming um for your agents, especially if you believe the cost of bad interactions can be can be rather large.
168. 14:38So, I think voice agents um have this awesome potential of of making the world feel much more human compared to interacting with clunky IVR trees or chat bots or worse
169. 14:52um being stuck on a on a hold.
170. 14:55And when we think about crime, we often think of crime happening to somebody else.
171. 15:00You know, crime does not happen to you typically with voice agents, especially bad actors.
172. 15:06As these agents are deployed and as bad actors start to exploit a lot of the vulnerabilities, the number of incidents is about to kind of go way way up.
173. 15:15And so the reason I fear voice agents more than crime is that one of these incidents is going to impact you.
174. 15:21It already did for me.
175. 15:24Awesome.
176. 15:24So it's time for me to shill.
177. 15:25Well, we burn a lot of tokens.
178. 15:27If you are interested in working in this space, please come and talk to us.
179. 15:32And if you are deploying voice agents and want to validate whether your architectured or your eval are set up correctly, please come and talk to us.
180. 15:41We'll be outside.
181. 15:42And here's here's my number.
182. 15:43Here's my WhatsApp.
183. 15:44Thanks everyone.
184. 16:02[music]