Voice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval

AI Engineer · 17 min · 182 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Um so I'm Brooke.
00:13I'm the founder of Koval and we build evaluations for voice agents.
00:18My background's from Waymo, so I led our evaluation infrastructure team at Waymo and my team was responsible for all of our developer tools for launching and running simulations.
00:28You might be like, what does that have to do with with voice agents?
00:32Well, autonomous vehicles are actually very similar to autonomous voice agents.
00:36How do you make sure that something reliably gets from point A to point B, all the possible paths in between, and also how do you architect lots of models working together
00:45in order to build an autonomous system that reacts and responds to the world reliably?
00:50Um so we work with enterprises, everyone from really fast-growing AI companies like Perplexity to Fortune 500s across the board, um to make sure that their voice agents are working well for customer support, logistics, etc.
01:02Cool, and I'm Shiv.
01:03Uh I lead our voice AI at Upstart.
01:06Uh for those not familiar with Upstart, we are a fintech company and we primarily do lending.
01:15Okay, cool.
01:16So, let me talk about voice a little bit.
01:19So, Upstart in general does uh thousands of calls, well, every day.
01:25Um I think um we talked about domain, which is lending.
01:30So, what is unique about lending is obviously it's a highly regulated space, um you know.
01:35And when when we have voice agents doing these calls with with applicants or borrowers, um if things go wrong, it's not just a uh you know, customer experience issue.
01:47It is It could be a potential regulatory issue, which is worse.
01:51Right?
01:51So, um whenever we we launch these agents, we want to make sure that uh you know, we are we are first and foremost compliant with all the rules and regulations.
02:03So for this one, I'm going to talk about a use case that you know, we we launched.
02:11The process the the agent that we launched was automating verification calls.
02:16So what happens is about 90% of people who apply to Upstart are instantly decisioned.
02:23The 10% that are not, they have to call in and they have to verify some information.
02:28So use case is fairly simple.
02:31We ask three questions.
02:32Will your income sources continue for the next three months?
02:35How do you plan to use the funds?
02:36And do you have any outstanding obligations such as you know, alimony, wage garnishments, etc. Okay, so those three questions aside, you know, we we first need to verify that the person calling in is
02:51actually the person.
02:53So they don't ask like, "Hey, can you tell me about some random application that they don't own?"
02:57Um then sort of you may think that this is a very simple use case, but this is where sort of regulatory things come into the picture.
03:04So I'll give a couple of examples.
03:06If someone says that I plan to use the funds for medical purposes, we can't ask the amount.
03:12We just prevented from doing that.
03:14If we do ask it, it creates a big issue for us.
03:18Um on the other hand, if someone says, "I plan to use the funds to travel to graduation."
03:23Right?
03:24Well, whose graduation is it?
03:25Your graduation?
03:26Your kids' graduation?
03:28Someone else's?
03:29That really determines the use of funds, right?
03:31So depending on on sort of what the applicant or the human at the other end says, we have to do specific things.
03:39We can ask or cannot ask things.
03:43And lastly, I think the it was very important for my compliance team that when we handle this use case, we just handle this use case, right?
03:51A lot of applicants are very eager to know what their status is or when they'll get the money because we can't just say like it'll be up to three business days.
04:01We just sort of sidestep the question in a nice and polite way.
04:04Um so that's that's sort of the use case that we launched.
04:10Cool.
04:10So um So that's the call.
04:14Now obviously when we launched this we want to make sure that the agent was doing well what it set out to do.
04:21Uh you know, so when we start audit auditing was very important.
04:25So every call had to be reviewed by a human.
04:28Um now obviously when this started out well how do you review all these calls?
04:35So we started with the humble Google sheet.
04:38You know, we start maintaining it.
04:40Our operations team that were reviewing these calls started putting in data there.
04:46In a few weeks five columns went to 10 then like 15 people started adding columns randomly putting stuff in columns creating things that you know, just just sort of messed up the whole process, right?
04:59So ultimately like what we need is data to improve both our agent as well as you know, the uh the things that we look in terms of performance, right?
05:11So that's where we sort of uh needed to have a better and more structured process for human review, right?
05:23And that's where Kowal came in and helped us build this human review process.
05:27Um but before I go there let me just talk a little bit about metrics, right?
05:31So when I talk about metrics here think about LM as a judge running once a call is complete.
05:39Right?
05:39So one important like criteria is were the compliance requirements met?
05:44So I it's a simple like yes no question with an explanation, right?
05:48So, uh one of the things we did is post post run, we run this judge and it produces an output with an explanation.
05:57Now, we want to make sure that the judge is working.
06:00So, we we put it and and we get a human to review the output of the LLM judge.
06:09Yeah, so when we started off, you know, we had metrics and we were tracking like a whole bunch of things like call time, you know, obviously accuracy, clarity, etc.
06:20I think over time we sort of whittled down the list to the metrics that were sort of the most important, right?
06:25And you can see some of the ones here like accuracy, clarity, completeness, policy alignment, whether it was escalated correctly, and ultimately CSAT score.
06:35One important thing I'd say is don't be too attached to your metrics.
06:41Start off with the ones that seem logical.
06:44If things change over time, you know, discard those metrics and track the other ones that really like impact your use case.
06:52Um Cool.
06:54So, we talked about LLM as judge and metrics a little bit.
06:57So, again, all these are run automatically once a call is complete.
07:03Now, I'll talk about one specific one here which was, you know, adherence to compliance, right?
07:09Or compliance violation.
07:10So, basically we we score the call.
07:14It goes into a queue.
07:16A human reviews the output of the of the metric.
07:21And then puts in their feedback, right?
07:23Why this is important is because we want to make sure that whatever the judge is outputting aligns with humans or experts in that particular area, right?
07:32And as you can see here like when we launched a while back, the compliance sort of agreement metric was 33%.
07:39That means a human reviewer and a LLM agent or a LLM judge agreed only 33% of the time.
07:48Now, well, we don't we we can't get to 100% but we can get like very close, right?
07:54So, we really start looking into like what is the gap there and that's where this process of providing feedback by experts sort of really helped, right?
08:02And ultimately it came down to a couple of things, right?
08:05There was a lot of tribal knowledge when we launched the agent that was not coded as part of the agent.
08:10So, we changed that and over time the the metric seems number went up.
08:15Then we realized that amongst humans in who are reviewing this call there was disagreement as well, right?
08:22Because there was a lot of stuff that was not written or or like tribal knowledge.
08:26So, we tried to like encode that.
08:29And then like sort of the next big bump came as we started to like look at more use cases.
08:34We started to see that well, the judge itself needed to be improved, right?
08:38So, ultimately we went up from like a 33% agreement rate to now today it's at 92%.
08:45[clears throat]
08:46Um we'll talk a little bit about how this process works in detail.
08:50So, I'll hand it over to Brooke.
08:53Amazing.
08:53So, from there what we really implemented was this continuous this complete loop of how you continuously improve the agents.
09:02So, I think this is something that Upstart has done really well is actually setting up a process that doesn't allow you to just launch once but allows you Um amazing.
09:10I'm super happy this is useful.
09:12Um so, everyone taking pictures.
09:14Um but it really allows you to like continuously improve your agent.
09:17I think this is something that enterprises get wrong is that they set up how whether the release gate so that we can launch one time but then it makes it very hard to change a model or change a prompt or anything underneath.
09:28And so, we go from like um I'll step through all of these processes of how you go from monitoring conversations, defining what your metrics look like, flagging calls, and then using that to constantly improve your agent.
09:42Um and so we also use this human review that Shiv just talked about and really believe that human review is a super important part of the evaluation process and where you should be spending most of your time is thinking about
09:54the product like evals are really the product definition.
09:57So where should you be spending your time um is what should my product be doing?
10:04Um so the first step is defining your metrics.
10:05So there are different types of evaluations.
10:07Like I think about it in three categories of did it get to the end?
10:11Did it adhere to the steps in order to get there?
10:14And then what was the conversation quality or the experience?
10:17Um and so in Cobal, and I'll be showing you how this actually works in Cobal.
10:21So you can build out these LLM judge or metrics that align with this original criteria that human reviewers were going through and labeling.
10:29That's also a really important step though is to have these human reviewers labeling what do they care about and then you can take that and turn it into automated evals
10:38versus the reverse is starting with like we care about word error rate but that's not actually what we're actioning on.
10:44Um and then the types of metrics are you know LLM as a judge, heuristics, audio processing.
10:50Um and so what we do is we monitor then take these metrics to monitor live calls.
10:55And so we can detect known failure patterns at scale.
10:58So being able to show exactly where for example we didn't verify um the correct pieces for the loan um or maybe where it didn't have the wording correct.
11:09Um and this is really important because no system works uh 100% of the time at scale.
11:14Uh even Google you know six nines of reliability there will will be failures.
11:18Um and then we also detect unknown unknowns.
11:21So things that we didn't know were going to happen doing by doing like clustering analysis and anomaly detection.
11:29Um and then we can route each one of these failures for human review.
11:33So you can configure review queues where they're actioning on any one of these metrics that we defined and then having these amazing human reviewers look through those failures.
11:44So now instead of looking at every call, they're only looking at the calls that matter most.
11:51And so from there we route all of these different failures to the right teams.
11:55This is a really critical step because you want to set up an operational practice so that every failure that's coming in you're doing something with that.
12:04So either you're routing it to you're routing for human review to detect real failures.
12:09So let's say like you know off script or wrong answer is 34% of the time.
12:14These are just made up numbers.
12:15Um then like maybe we send those to the agent team versus compliance violations we send that to a compliance team so that they can file a process or you know take some like tactical action on it versus sending to an engineering team etc.
12:31Um and so now once we have a set of real failures, we can then re-simulate or we then for example Steven who's here can figure out what is the right fix to the agent.
12:42Um and then we can re-simulate that that fix to see what exactly the result is.
12:49Like did it actually fix this issue?
12:52Um and so our simulations do voice to voice evaluation.
12:56We will call your agent and actually run a simulation to um you know test different background noises, different accents, um the actual voice quality, interruptions, etc. And then we can verify that that improvement actually fixed things before it's impacting your users.
13:13So once you know that there's an issue like a compliance violation or maybe like a poor user experience, you don't want to launch that again to production to see if it fixed it because you're now risking impact to your users.
13:25So we want to shift left to use a devops term.
13:28Um we want to detect as many failures as we can in simulation instead of doing that in production.
13:33So, here you can see that, you know, we improved um those cases in the report.
13:39And so, this finally completes the loop of um we monitor we define what our metrics are based on our product definition.
13:47This is honestly where you should spend the most of your time is what does my agent do and what should it not do.
13:53Um and then we can monitor and score every live call, um detect those failures, send them for human review, find failure patterns across all of these to prioritize engineering work.
14:04And one of the most exciting things is now you can enable the self-healing loop where we can actually have agents that look at all of these evaluations and identify fixes,
14:14but we can also use RL pipelines or fine-tuned LLMs in order to make our system even better over time automatically.
14:23Um and then actually I have to go back.
14:26Okay.
14:27Um and then from there we can automate fixes and then prove that this actually is improving things in production through simulation.
14:36And all of these things obviously are gated by human review and integrating human review in the loop.
14:42Um and so, these defined setups can uh both make sure our metrics are really accurate.
14:48So, as I should have talked through, every time you label a feedback every time you label a metric as it was being as it was incorrect, now you can align your metrics to human judgment
14:59or it it actually is a true failure, in which case you can use that to automatically improve your agents.
15:06Um
15:09Cool.
15:10Um so, yeah, in conclusion, right?
15:12Evals are important.
15:13Don't don't skip them.
15:15Uh evals uh sort of help us uh sort of monitor and score the calls, but also determine which calls need attention or which calls need to be like reviewed by a human because humans can't review like thousands of calls a day.
15:28Um, the human reviewer annotations are important because they help improve the LM judge.
15:35Um, all this data that we collect ultimately helps improve the agent as well.
15:39Um, changes are always gated behind a feature flag, right?
15:44So, we when we say self-healing or self-improving, it doesn't mean like humans are completely out of the loop.
15:50Uh, with uh, fintech, uh, we we need these, uh, changes to be sort of monitored and and signed on by like someone, right?
15:58So, judges usually the engineers do.
16:01Uh, but if uh, there was a compliance change, then compliance is involved in approving like, uh, the behavior of the agent.
16:07Um, and lastly, like this has served like as a blueprint for us to launch agents safely meeting regulatory requirements and also improving their performance at the same time.
16:16And I think that's why like actually being able to set up really robust evals defines a really high-quality product.
16:24Um, in all cases, but I think especially in regulated industries, you just can't, um, deploy the applications that you want to without having really robust evals that give you confidence
16:34that you're doing it like deploying these autonomous systems with precision and robust, reliable, um, you know, great products that can actually scale.
16:43Cool.
16:44Uh, we are almost at time.
16:45So, yeah, feel free to like, uh, you know, connect with us on LinkedIn or you know, just reach out if you, uh, see us, uh, anywhere, uh, today.
16:53So, but that's, uh, that's our presentation.