Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest

AI Engineer · 18 min · 257 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi guys.
00:13So my name is Jonathan Osborne.
00:15I am a lead AI engineer from that west group.
00:18I'll tell you more about what that is in a second.
00:21And today I'm going to be talking about taking agents from prototype to production customerf facing that is in a regulated banking environment.
00:28And I guess maybe some spoilers with the access to coding assistants and tools that we have today.
00:33The actual creation of the agent that's kind of the easy part.
00:36The much harder part I think is proving that it's safe.
00:41So let me tell you a bit about Nat West Group.
00:44So Nat West is the second largest retail bank in the UK and currently we have somewhere in the range of 19 million retail customers.
00:5219 million.
00:53Hold on to that number because it's really what makes everything else in this talk matter.
00:57Kora is our customer's primary point of contact.
01:00So it's our digital assistant and it serves somewhere in the realm of a 100,000 conversations a day.
01:06And until now the vast vast bulk of those answers were scripted.
01:10So if a customer comes to us and they ask a conversation then if we haven't fought to script that then uh you know we're not going to be able to answer.
01:19Customers going to get frustrated.
01:20They're probably going to have to cue to speak to a human agent.
01:23That's changing.
01:24So as of this year we are now working to bring approximately a third of the journeys within Kora to AI generated answers backed by Rag and our knowledge bases
01:34and agents agents that are able to take action on the behalf of customers.
01:38So for example helping them make a payment or change their address.
01:43I guess the scary part here really isn't the model like by and large models are quite good.
01:48They provide good answers.
01:49Now it's the blast radius.
01:51So with 19 million customers, if we get a few of these utterances wrong, that isn't a bug ticket.
01:57That isn't kind of a maybe a funny headline, it's a real customer and they're real money.
02:02It's slightly very difficult question from the regulator and it's reputational damage for us as a company.
02:08So this talk is about what we had to go through to get the approval to go live and I guess what it took us to maybe trust some of our own metrics as well because as I say the interesting part it's not the agent it's proving and making it safe.
02:24Let's maybe make that more concrete by considering an example here.
02:27So kind of same question two very different conversations.
02:30So customer comes, they ask, "Can I overpay my mortgage without being charged?"
02:35Today in the scripted solution, that's going to recognize that keyword mortgage, and we're going to present the customer with a menu.
02:41So apply rates, my mortgage, customer is going to have to select none of the above.
02:45Maybe we'll ask them to rephrase their question, but more than likely, game over.
02:49This customer is going to have to cue, speak to a human agent.
02:51They're going to be frustrated.
02:53On the right hand side, bit of a different story.
02:55So AI has been able to answer that question using our knowledge bases.
02:59You can overpay up to 10% of your balance in a year.
03:01No early repayment charges.
03:03Shall I check your deal?
03:05But then the customer does something which is very normal but is can be frustrating.
03:09So they change this topic.
03:11So actually can I close the account instead?
03:14This happens all the time and a traditional decision tree is really really bad at this type of digression.
03:20It's worth mentioning that digressions are going to come back.
03:22So we'll hold on to that point.
03:25I guess the key point though is if we build an agent that can answer any question, it can also answer any question wrong.
03:31So that 10% if that doesn't apply to this particular customer's deal, it's a real customer, is their real mortgage, and is there real money at stake?
03:40So why is it so much harder to make an agent back solution, you know, more, you know, how do we protect against it as opposed to that scripted bot?
03:49Well, it's because it reasons, it decides, it generates.
03:51And there are at least kind of four failure mode here, which we wouldn't have in that scripted solution.
03:56One, it can be confidently wrong.
03:59Hallucination against a fictitious source and sound absolutely sure of itself while it does it.
04:04Two, it can be talked into things, prompt injection.
04:07The input itself can become adversarial.
04:10Three, right, but unhelpful, so it can provide accurate on policy answers that just go round and around and around and never actually answer the customer's conversation.
04:18incredibly frustrating.
04:20And four, it can be nondeterministic.
04:22Same inputs, completely different output.
04:25That means that your traditional software engineering assertions just don't really work very well here.
04:29And as a consequence of that, regression tests that have been built up over years by the existing core team, they really no longer apply.
04:37A scripted systems failure surface is completely bounded.
04:40We can write out every situation that might occur.
04:43we can test it and we can be really confident it's going to do what we want it to with an agent that's really not possible and we can't really traditionally
04:50test our way to safety.
04:52So how do we deal with this?
04:53Well, the first part for us was shrinking the attack surface before we actually then thought about hardening it introducing what we call the routing funnel.
05:03So every customer conversation comes in at the top here.
05:06First we apply a contextual filter and that allows us to remove kind of common events like button clicks.
05:11That's about 20%.
05:13Next, we have some deterministic pre-classification rules.
05:17So, with these rules, we're kind of using the customer's utterance and some metadata to route them to a destination with no ML or AI involved at all.
05:24At this point, it's only utterances that make it past these first two gates that we then actually start the models on.
05:30First, there's a relevancy and rephrasing step that's kind of gated.
05:33It helps with those really detailed multicontext questions.
05:36And then our intent classifier.
05:38This is actually a neural network over embeddings.
05:40This isn't generative AI which is able to use a customer's utterance to predict the right back end.
05:45Finally, there's some policy rules to ensure that this customer should actually be getting into that particular backend.
05:52Why do we build it this way?
05:53Well, four reasons a lot of them about being a bank.
05:56Firstly, access control.
05:57So, we need to enforce that to ensure the customer should be getting to this act this this back end.
06:02Auditability.
06:03So, every decision here needs to carry a code.
06:06It needs to be fully auditable and reproducible.
06:08The regulator ensures this resilience.
06:10In the case that a model or the gender isn't available, we still need to provide the customer an answer and I guess governance.
06:17So all of this is version config.
06:18We can make changes to it without a full redeployment.
06:21That's really important because it means that we can react fast to changing market conditions.
06:25We didn't just take a language model, slap some context into it, and point it at 19 million customers.
06:30We first thought really hard about what actually needed a model to solve.
06:34This brings us on to the really important question of is this actually working?
06:38And the key metric that they looked at in kind of traditional Kora was accuracy when doing this.
06:42So if I told you this intent writer is 94% accurate, that probably sounds really good.
06:47But does that actually answer the important question?
06:50So is it grounded in a real source?
06:52Did the customer's question actually get resolved?
06:54Did it know when it was out of its depth?
06:56When to stop bring in a human?
06:58You can't embody that with one single score.
07:01And that's why we've moved away from quality as a score, started treating it more as a vector.
07:07Uh, every score, every kind of answer we score across three different categories.
07:11So, safety, could it harm the customer?
07:13Task, did it actually do its job?
07:15Experience, how did it feel to deal with?
07:17All of these can be monitored and alarmed independently.
07:20And they allow us to differentiate between is it hallucinating versus is it right but just insufferable to deal with.
07:27There are different answers and they require really different solutions.
07:31It's what the key point here is a routter that's 94% accurate but 100% confident the 6% of the time it's wrong is way more dangerous than one that's 85%
07:41confident and knows it.
07:43The first thing we did here before we worried about automating this is what we always do and we started off with some human validation.
07:48Uh the numbers that you see here are some June validation that we did on our payment agent.
07:53111 conversations over 37 different payment journeys.
07:57Four brands, three independent human scorers per conversation.
08:01Headline here, 83% pass rate across the board.
08:04But as ever, the headline's pretty useless.
08:06Let's dig into some of that data.
08:08So 97% at on the happy path scenarios.
08:11That's great.
08:1276% edge cases, not quite so good.
08:15Topic changes like that mortgage example that we discussed earlier, only 43%.
08:20So that single 83% score was hiding that 54% spread.
08:25The second point on the right is probably more nuanced and less talked about and that's do human validators actually agree with each other
08:31[snorts]
08:31on something objective like bias and fairness you do actually see a 95% agreement rate which is great but on something more subjective intent understanding so you know are does the bot actually understand what they what the user wants
08:45only a 64% agreement rate.
08:48The common thing when we start automating this bringing LLM as a judge is that we want to make our judge match our human validators.
08:55But what are they meant to be matching here?
08:56There's only a 60 form agreement rate between humans.
09:00So that can't simply be the target here essentially.
09:03One one caveat by the way like the two biggest causes of failure here actually turned out to be engineering issues.
09:08Not problem with the model at all, but nevertheless that proves useful value of the human validation because it caught them.
09:15So let's turn to actually automating this then.
09:18So LLM as a judge.
09:19So core plus that's not live yet.
09:22So I'm going to turn to another internal project.
09:24So this is an internal complaints agent that we have.
09:27Essentially the goal here is that customers will occasionally complain.
09:30This will take the complaint.
09:32It has access to a load of tools to then be able to write a summary for complaint handlers to essentially explain why has this particular user complained and how we can help them with it.
09:41It went live in February and we've got around 14,000 of these cases which have been scored by both a human validator and an LLM as a judge.
09:48If we dive into the data, so this is the faithfulness score here on the right.
09:52So you can see upper bar in the red there, that's the LM as a judge mean.97 right up at the top very little spread.
09:59Human mean 083 much much wider spread.
10:04Key point here is that if you look at that average error versus the human it's 2. you might conclude that the judge is doing okay but the correlation is almost zero.
10:15Uh in in this case like a judge which doesn't discriminate between the actually good cases and the bad cases is basically worthless.
10:21You don't want your judge to be generous and that is what LLMs want to do by default.
10:26Maybe one other point to make here is that so that amber dotted line that represents the threshold for this particular metric for this judge model.
10:34You can see we're comfortably above that.
10:36However, we made a change to that model.
10:38Didn't change anything else.
10:39Same inputs, same outputs.
10:40Just change the actual model behind it.
10:42And that dropped all the way from point 8 above to 2 above.
10:46That means that the judge model itself actually becomes part of your configuration.
10:50And it can lead to breaking changes.
10:52A key point we took here, by the way, is that we moved our judge to a different model family from that of the uh, you know, the actual agent.
10:59So, judged and judged, different blind spots, and that's actually been really effective in pinning the two scores closer together.
11:07Coming back to the core example.
11:08So how does this scoring actually run?
11:10So live conversations between the agents and its guardrails.
11:14All of those are captured ving open source standards like hotel and open inference.
11:18All of that detail is then sampled.
11:20So by default we take 5% random sample but we also want to look at everything interesting.
11:25By interesting I mean conversations are handoff fail those are particularly long ones where the user is angry.
11:31We then score these with our LLM as a judge.
11:33So that's that raft and dimension of metrics that I talked about earlier.
11:36And then absolutely all of this ends up in comedic.
11:39So traces, scores, experiments.
11:42It's governed by a couple of bank constraints here.
11:44So we work in a completely airgapped environment.
11:46So no internet access to anything.
11:48So judge model is reached over a private path.
11:50We have an internal geni gateway.
11:52Comet a SAS product.
11:53We're consuming that as a signed partner app via AWS.
11:57Second around personally identifiable information.
11:59We have a zero tolerance policy for any PII leaving our network.
12:02So that means a lot of this evaluation happens without that present different cadences here.
12:08So tracers are being captured as they happen.
12:10We're doing offline scoring every four hours.
12:13And then regress this is also built into our CI/CD.
12:16So if we make changes to prompts or to context that can have like an outsized impact on what the agent is actually responding.
12:22So we need to constantly be comparing versus a known good baseline.
12:27We do that in CI/CD.
12:28There's a hard regression gate.
12:30So changes can't be merged in without us understanding exactly what's going to occur.
12:34I guess key point is that continuous evaluation is part of your production infrastructure.
12:39It's not a notebook that you run a couple of times when your model goes live.
12:44So there's different models here and they've got different failure modes.
12:47So on the right hand side we've got the agents, we've got the generative AI.
12:50So generative AI regresses.
12:52So as we've just talked about, somebody can make changes to a prompt and that can have outsized impact on the behavior.
12:57That's why we have the CI/CD in place.
12:59You can see here a real opic trace which is looking from our payment agent.
13:03Every node in the LAN graph uh becomes a different span and when the agent is slow or is breaking in a particular place, this allows us or an agent to jump in and investigate exactly why
13:14on the classical.
13:15So the intent classifier is traditional ML and thus we've got to monitor it in a more classical way.
13:20It drifts over time, right?
13:21So human conversation will naturally evolve from the point at which the model was trained.
13:25So we need to look at more traditional things like input feature drift, F1 score based off human scores, sorry, human labels.
13:32And then you know we have rag thresholds defined as to exactly when we need to trigger retraining for that model.
13:38On the right you can see an npm data drift chart.
13:40I I really wouldn't underestimate like input feature drift here.
13:43It's a really good canary as to when your model is starting to decline before your F1 score does implying the need for retraining.
13:50So two different problems, two different instruments, but what we don't want is two different observability and evaluation frameworks.
13:57Comet's really useful there because it brings in both OPIC for the Genai and MPM model performance monitor for the ML.
14:03So one platform how do we actually deploy this then without kind of breaking the bank and creating a load of headlines.
14:11We start off with a shadow deployment.
14:12So what I mean by that is we put this live uh we use it to start classifying cutter utterancerances but this existing scripted solution is what will actually respond to the customer
14:21and that means that we can get a sense of the real data you know the the input uh the input space for different customer queries with almost zero risk.
14:30I think this shadow mode is like one of the more transferable bits of this project for us because often we have this deadlock in a regulated industry where obviously we can't go live and get production data without the approval to do so.
14:42But equally to actually enable us to go live in many cases our model risk colleagues increasingly want to see how our models are going to react to that actual production data.
14:52This allows us to kind of break that apart so we can break apart observe safely from actually answer customer queries.
15:00uh and it takes one really big decision point, a big bang release into a series of smaller ones which we can make confidently.
15:06We then go brand by brand.
15:07So we have like percentage based rollout within the brand and we have kind of KPIs which allow us to determine go no-go at various points along there.
15:15Some of those decisions are automated and some of those require manual approval.
15:20Um I think and yeah some of those require manual approval.
15:24What kind of makes that possible is that we make a lot of use of feature flagging here.
15:28So there's like a global flag that we can trigger to send everything back to the scripted solution, but also every single backend can be isolated either globally or on a particular dimension such as brand
15:39and all of that can be done in seconds.
15:42Before I wrap up, let me maybe just talk very briefly and pull this back to that complaints project.
15:47So just to remind you that was the colleague facing agent which writes investigation summaries for summaries for complaint handlers.
15:53In that project, human feedback is a gate.
15:55It's not a thumbs up.
15:56It's not a tickbox activity.
15:58So if a complaint handler wants to rerun their investigation, they can't do so until they provided human feedback.
16:04We want to understand why they felt the need to do that rerun.
16:07But also it's allowed us to build up this quality data set of 14 14,000 customer of uh of colleague labels I guess in this case and that's really valuable in a lot of projects.
16:18We struggle to get a tenth of that number and I think the projects have also traded nicely here.
16:23So you know they found that judge problem in production we have judge calibration and regression checks because of that but equally the architecture that we built which relies heavily on commodopic
16:32they don't use that yet and they're going to be adopting that for their production monitoring I guess it's a glimpse inside an enterprise right so the the agent was the cheap part here that's code that we can produce to solve the business problem the valuation spine is the bit that we can then reuse again and again
16:48so to wrap up like if you are going to be shipping customerf facing agents in a regulated environment, then these are probably the points I would give you.
16:56I'm not going to go through everything.
16:57I'll maybe just pick a couple.
16:59So, three and four here, they're the they're the points with the actual production judge data, right?
17:04So, that judge pinned near the top, it's a broken instrument.
17:08Your judges are going to want to be overly positive.
17:10You need to fight against that.
17:11Compare the spans, not just the averages.
17:13Ensure the judge is actually matching the human reviewer.
17:16Fourth, the judge model itself does become part of that threshold config, right?
17:21So if you are going to be making changes to the judge model then uh you know you can introduce a breaking change without knowing it.
17:29Comet is obviously absolutely key here.
17:30So we have that for both halves both the generative AI and the traditional ML which we do still use a little bit.
17:36So and I guess the the final point here is next month I'll find out how much of this is actually true when we go live.
17:43So thanks a lot for listening.
17:44It was really a pleasure to get to speak to all of you.
17:47So I'll be at the comment booth after this.
17:48So if you do have any questions, you know, disagreements, etc., really happy to hear them.
17:52But thanks a lot for your time.