WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=5UgnatzpZ9E NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/5UgnatzpZ9E.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.559 --> 00:00:13.599 Hi guys. s3 00:00:13.599 --> 00:00:15.599 So my name is Jonathan Osborne. s4 00:00:15.599 --> 00:00:18.560 I am a lead AI engineer from that west group. s5 00:00:18.560 --> 00:00:21.199 I'll tell you more about what that is in a second. s6 00:00:21.199 --> 00:00:28.560 And today I'm going to be talking about taking agents from prototype to production customerf facing that is in a regulated banking environment. s7 00:00:28.560 --> 00:00:33.600 And I guess maybe some spoilers with the access to coding assistants and tools that we have today. s8 00:00:33.600 --> 00:00:36.960 The actual creation of the agent that's kind of the easy part. s9 00:00:36.960 --> 00:00:41.600 The much harder part I think is proving that it's safe. s10 00:00:41.600 --> 00:00:44.640 So let me tell you a bit about Nat West Group. s11 00:00:44.640 --> 00:00:52.559 So Nat West is the second largest retail bank in the UK and currently we have somewhere in the range of 19 million retail customers. s12 00:00:52.559 --> 00:00:53.280 19 million. s13 00:00:53.280 --> 00:00:57.920 Hold on to that number because it's really what makes everything else in this talk matter. s14 00:00:57.920 --> 00:01:00.960 Kora is our customer's primary point of contact. s15 00:01:00.960 --> 00:01:06.880 So it's our digital assistant and it serves somewhere in the realm of a 100,000 conversations a day. s16 00:01:06.880 --> 00:01:10.880 And until now the vast vast bulk of those answers were scripted. s17 00:01:10.880 --> 00:01:19.200 So if a customer comes to us and they ask a conversation then if we haven't fought to script that then uh you know we're not going to be able to answer. s18 00:01:19.200 --> 00:01:20.479 Customers going to get frustrated. s19 00:01:20.479 --> 00:01:23.680 They're probably going to have to cue to speak to a human agent. s20 00:01:23.680 --> 00:01:24.799 That's changing. s21 00:01:24.799 --> 00:01:34.960 So as of this year we are now working to bring approximately a third of the journeys within Kora to AI generated answers backed by Rag and our knowledge bases s22 00:01:34.960 --> 00:01:38.880 and agents agents that are able to take action on the behalf of customers. s23 00:01:38.880 --> 00:01:43.840 So for example helping them make a payment or change their address. s24 00:01:43.840 --> 00:01:48.399 I guess the scary part here really isn't the model like by and large models are quite good. s25 00:01:48.399 --> 00:01:49.600 They provide good answers. s26 00:01:49.600 --> 00:01:51.600 Now it's the blast radius. s27 00:01:51.600 --> 00:01:57.439 So with 19 million customers, if we get a few of these utterances wrong, that isn't a bug ticket. s28 00:01:57.439 --> 00:02:02.079 That isn't kind of a maybe a funny headline, it's a real customer and they're real money. s29 00:02:02.079 --> 00:02:08.160 It's slightly very difficult question from the regulator and it's reputational damage for us as a company. s30 00:02:08.160 --> 00:02:23.920 So this talk is about what we had to go through to get the approval to go live and I guess what it took us to maybe trust some of our own metrics as well because as I say the interesting part it's not the agent it's proving and making it safe. s31 00:02:24.080 --> 00:02:27.120 Let's maybe make that more concrete by considering an example here. s32 00:02:27.120 --> 00:02:30.480 So kind of same question two very different conversations. s33 00:02:30.480 --> 00:02:35.120 So customer comes, they ask, "Can I overpay my mortgage without being charged?" s34 00:02:35.120 --> 00:02:41.280 Today in the scripted solution, that's going to recognize that keyword mortgage, and we're going to present the customer with a menu. s35 00:02:41.280 --> 00:02:45.840 So apply rates, my mortgage, customer is going to have to select none of the above. s36 00:02:45.840 --> 00:02:49.360 Maybe we'll ask them to rephrase their question, but more than likely, game over. s37 00:02:49.360 --> 00:02:51.840 This customer is going to have to cue, speak to a human agent. s38 00:02:51.840 --> 00:02:53.360 They're going to be frustrated. s39 00:02:53.360 --> 00:02:55.760 On the right hand side, bit of a different story. s40 00:02:55.760 --> 00:02:59.519 So AI has been able to answer that question using our knowledge bases. s41 00:02:59.519 --> 00:03:01.920 You can overpay up to 10% of your balance in a year. s42 00:03:01.920 --> 00:03:03.519 No early repayment charges. s43 00:03:03.519 --> 00:03:05.120 Shall I check your deal? s44 00:03:05.120 --> 00:03:09.599 But then the customer does something which is very normal but is can be frustrating. s45 00:03:09.599 --> 00:03:11.120 So they change this topic. s46 00:03:11.120 --> 00:03:14.400 So actually can I close the account instead? s47 00:03:14.400 --> 00:03:20.319 This happens all the time and a traditional decision tree is really really bad at this type of digression. s48 00:03:20.319 --> 00:03:22.959 It's worth mentioning that digressions are going to come back. s49 00:03:22.959 --> 00:03:25.519 So we'll hold on to that point. s50 00:03:25.519 --> 00:03:31.440 I guess the key point though is if we build an agent that can answer any question, it can also answer any question wrong. s51 00:03:31.440 --> 00:03:40.480 So that 10% if that doesn't apply to this particular customer's deal, it's a real customer, is their real mortgage, and is there real money at stake? s52 00:03:40.480 --> 00:03:49.040 So why is it so much harder to make an agent back solution, you know, more, you know, how do we protect against it as opposed to that scripted bot? s53 00:03:49.040 --> 00:03:51.599 Well, it's because it reasons, it decides, it generates. s54 00:03:51.599 --> 00:03:56.879 And there are at least kind of four failure mode here, which we wouldn't have in that scripted solution. s55 00:03:56.879 --> 00:03:59.040 One, it can be confidently wrong. s56 00:03:59.040 --> 00:04:04.000 Hallucination against a fictitious source and sound absolutely sure of itself while it does it. s57 00:04:04.000 --> 00:04:07.120 Two, it can be talked into things, prompt injection. s58 00:04:07.120 --> 00:04:10.080 The input itself can become adversarial. s59 00:04:10.080 --> 00:04:18.400 Three, right, but unhelpful, so it can provide accurate on policy answers that just go round and around and around and never actually answer the customer's conversation. s60 00:04:18.400 --> 00:04:20.079 incredibly frustrating. s61 00:04:20.079 --> 00:04:22.479 And four, it can be nondeterministic. s62 00:04:22.479 --> 00:04:25.120 Same inputs, completely different output. s63 00:04:25.120 --> 00:04:29.600 That means that your traditional software engineering assertions just don't really work very well here. s64 00:04:29.600 --> 00:04:37.199 And as a consequence of that, regression tests that have been built up over years by the existing core team, they really no longer apply. s65 00:04:37.199 --> 00:04:40.000 A scripted systems failure surface is completely bounded. s66 00:04:40.000 --> 00:04:43.360 We can write out every situation that might occur. s67 00:04:43.360 --> 00:04:50.560 we can test it and we can be really confident it's going to do what we want it to with an agent that's really not possible and we can't really traditionally s68 00:04:50.560 --> 00:04:52.400 test our way to safety. s69 00:04:52.400 --> 00:04:53.759 So how do we deal with this? s70 00:04:53.759 --> 00:05:03.199 Well, the first part for us was shrinking the attack surface before we actually then thought about hardening it introducing what we call the routing funnel. s71 00:05:03.199 --> 00:05:06.479 So every customer conversation comes in at the top here. s72 00:05:06.479 --> 00:05:11.680 First we apply a contextual filter and that allows us to remove kind of common events like button clicks. s73 00:05:11.680 --> 00:05:13.520 That's about 20%. s74 00:05:13.520 --> 00:05:17.120 Next, we have some deterministic pre-classification rules. s75 00:05:17.120 --> 00:05:24.400 So, with these rules, we're kind of using the customer's utterance and some metadata to route them to a destination with no ML or AI involved at all. s76 00:05:24.400 --> 00:05:30.160 At this point, it's only utterances that make it past these first two gates that we then actually start the models on. s77 00:05:30.160 --> 00:05:33.120 First, there's a relevancy and rephrasing step that's kind of gated. s78 00:05:33.120 --> 00:05:36.400 It helps with those really detailed multicontext questions. s79 00:05:36.400 --> 00:05:38.560 And then our intent classifier. s80 00:05:38.560 --> 00:05:40.880 This is actually a neural network over embeddings. s81 00:05:40.880 --> 00:05:45.680 This isn't generative AI which is able to use a customer's utterance to predict the right back end. s82 00:05:45.680 --> 00:05:52.080 Finally, there's some policy rules to ensure that this customer should actually be getting into that particular backend. s83 00:05:52.080 --> 00:05:53.440 Why do we build it this way? s84 00:05:53.440 --> 00:05:56.160 Well, four reasons a lot of them about being a bank. s85 00:05:56.160 --> 00:05:57.600 Firstly, access control. s86 00:05:57.600 --> 00:06:02.880 So, we need to enforce that to ensure the customer should be getting to this act this this back end. s87 00:06:02.880 --> 00:06:03.840 Auditability. s88 00:06:03.840 --> 00:06:06.000 So, every decision here needs to carry a code. s89 00:06:06.000 --> 00:06:08.000 It needs to be fully auditable and reproducible. s90 00:06:08.000 --> 00:06:10.720 The regulator ensures this resilience. s91 00:06:10.720 --> 00:06:17.199 In the case that a model or the gender isn't available, we still need to provide the customer an answer and I guess governance. s92 00:06:17.199 --> 00:06:18.880 So all of this is version config. s93 00:06:18.880 --> 00:06:21.280 We can make changes to it without a full redeployment. s94 00:06:21.280 --> 00:06:25.759 That's really important because it means that we can react fast to changing market conditions. s95 00:06:25.759 --> 00:06:30.240 We didn't just take a language model, slap some context into it, and point it at 19 million customers. s96 00:06:30.240 --> 00:06:34.880 We first thought really hard about what actually needed a model to solve. s97 00:06:34.880 --> 00:06:38.240 This brings us on to the really important question of is this actually working? s98 00:06:38.240 --> 00:06:42.960 And the key metric that they looked at in kind of traditional Kora was accuracy when doing this. s99 00:06:42.960 --> 00:06:47.919 So if I told you this intent writer is 94% accurate, that probably sounds really good. s100 00:06:47.919 --> 00:06:50.000 But does that actually answer the important question? s101 00:06:50.000 --> 00:06:52.240 So is it grounded in a real source? s102 00:06:52.240 --> 00:06:54.960 Did the customer's question actually get resolved? s103 00:06:54.960 --> 00:06:56.720 Did it know when it was out of its depth? s104 00:06:56.720 --> 00:06:58.960 When to stop bring in a human? s105 00:06:58.960 --> 00:07:01.440 You can't embody that with one single score. s106 00:07:01.440 --> 00:07:07.039 And that's why we've moved away from quality as a score, started treating it more as a vector. s107 00:07:07.039 --> 00:07:11.039 Uh, every score, every kind of answer we score across three different categories. s108 00:07:11.039 --> 00:07:13.039 So, safety, could it harm the customer? s109 00:07:13.039 --> 00:07:15.280 Task, did it actually do its job? s110 00:07:15.280 --> 00:07:17.680 Experience, how did it feel to deal with? s111 00:07:17.680 --> 00:07:20.720 All of these can be monitored and alarmed independently. s112 00:07:20.720 --> 00:07:27.759 And they allow us to differentiate between is it hallucinating versus is it right but just insufferable to deal with. s113 00:07:27.759 --> 00:07:31.759 There are different answers and they require really different solutions. s114 00:07:31.759 --> 00:07:41.120 It's what the key point here is a routter that's 94% accurate but 100% confident the 6% of the time it's wrong is way more dangerous than one that's 85% s115 00:07:41.120 --> 00:07:43.759 confident and knows it. s116 00:07:43.759 --> 00:07:48.880 The first thing we did here before we worried about automating this is what we always do and we started off with some human validation. s117 00:07:48.880 --> 00:07:53.759 Uh the numbers that you see here are some June validation that we did on our payment agent. s118 00:07:53.759 --> 00:07:57.120 111 conversations over 37 different payment journeys. s119 00:07:57.120 --> 00:08:01.039 Four brands, three independent human scorers per conversation. s120 00:08:01.039 --> 00:08:04.720 Headline here, 83% pass rate across the board. s121 00:08:04.720 --> 00:08:06.879 But as ever, the headline's pretty useless. s122 00:08:06.879 --> 00:08:08.400 Let's dig into some of that data. s123 00:08:08.400 --> 00:08:11.599 So 97% at on the happy path scenarios. s124 00:08:11.599 --> 00:08:12.479 That's great. s125 00:08:12.479 --> 00:08:15.680 76% edge cases, not quite so good. s126 00:08:15.680 --> 00:08:20.160 Topic changes like that mortgage example that we discussed earlier, only 43%. s127 00:08:20.160 --> 00:08:25.120 So that single 83% score was hiding that 54% spread. s128 00:08:25.120 --> 00:08:31.560 The second point on the right is probably more nuanced and less talked about and that's do human validators actually agree with each other s129 00:08:31.560 --> 00:08:31.680 [snorts] s130 00:08:31.680 --> 00:08:45.920 on something objective like bias and fairness you do actually see a 95% agreement rate which is great but on something more subjective intent understanding so you know are does the bot actually understand what they what the user wants s131 00:08:45.920 --> 00:08:48.800 only a 64% agreement rate. s132 00:08:48.800 --> 00:08:55.040 The common thing when we start automating this bringing LLM as a judge is that we want to make our judge match our human validators. s133 00:08:55.040 --> 00:08:56.640 But what are they meant to be matching here? s134 00:08:56.640 --> 00:09:00.320 There's only a 60 form agreement rate between humans. s135 00:09:00.320 --> 00:09:03.519 So that can't simply be the target here essentially. s136 00:09:03.519 --> 00:09:08.480 One one caveat by the way like the two biggest causes of failure here actually turned out to be engineering issues. s137 00:09:08.480 --> 00:09:15.760 Not problem with the model at all, but nevertheless that proves useful value of the human validation because it caught them. s138 00:09:15.760 --> 00:09:18.320 So let's turn to actually automating this then. s139 00:09:18.320 --> 00:09:19.839 So LLM as a judge. s140 00:09:19.839 --> 00:09:22.000 So core plus that's not live yet. s141 00:09:22.000 --> 00:09:24.480 So I'm going to turn to another internal project. s142 00:09:24.480 --> 00:09:27.600 So this is an internal complaints agent that we have. s143 00:09:27.600 --> 00:09:30.880 Essentially the goal here is that customers will occasionally complain. s144 00:09:30.880 --> 00:09:32.080 This will take the complaint. s145 00:09:32.080 --> 00:09:41.200 It has access to a load of tools to then be able to write a summary for complaint handlers to essentially explain why has this particular user complained and how we can help them with it. s146 00:09:41.200 --> 00:09:48.959 It went live in February and we've got around 14,000 of these cases which have been scored by both a human validator and an LLM as a judge. s147 00:09:48.959 --> 00:09:52.320 If we dive into the data, so this is the faithfulness score here on the right. s148 00:09:52.320 --> 00:09:59.920 So you can see upper bar in the red there, that's the LM as a judge mean.97 right up at the top very little spread. s149 00:09:59.920 --> 00:10:04.000 Human mean 083 much much wider spread. s150 00:10:04.000 --> 00:10:15.120 Key point here is that if you look at that average error versus the human it's 2. you might conclude that the judge is doing okay but the correlation is almost zero. s151 00:10:15.120 --> 00:10:21.839 Uh in in this case like a judge which doesn't discriminate between the actually good cases and the bad cases is basically worthless. s152 00:10:21.839 --> 00:10:26.320 You don't want your judge to be generous and that is what LLMs want to do by default. s153 00:10:26.320 --> 00:10:34.000 Maybe one other point to make here is that so that amber dotted line that represents the threshold for this particular metric for this judge model. s154 00:10:34.000 --> 00:10:36.240 You can see we're comfortably above that. s155 00:10:36.240 --> 00:10:38.320 However, we made a change to that model. s156 00:10:38.320 --> 00:10:39.360 Didn't change anything else. s157 00:10:39.360 --> 00:10:40.720 Same inputs, same outputs. s158 00:10:40.720 --> 00:10:42.880 Just change the actual model behind it. s159 00:10:42.880 --> 00:10:46.800 And that dropped all the way from point 8 above to 2 above. s160 00:10:46.800 --> 00:10:50.720 That means that the judge model itself actually becomes part of your configuration. s161 00:10:50.720 --> 00:10:52.880 And it can lead to breaking changes. s162 00:10:52.880 --> 00:10:59.440 A key point we took here, by the way, is that we moved our judge to a different model family from that of the uh, you know, the actual agent. s163 00:10:59.440 --> 00:11:07.040 So, judged and judged, different blind spots, and that's actually been really effective in pinning the two scores closer together. s164 00:11:07.040 --> 00:11:08.560 Coming back to the core example. s165 00:11:08.560 --> 00:11:10.560 So how does this scoring actually run? s166 00:11:10.560 --> 00:11:14.000 So live conversations between the agents and its guardrails. s167 00:11:14.000 --> 00:11:18.560 All of those are captured ving open source standards like hotel and open inference. s168 00:11:18.560 --> 00:11:20.480 All of that detail is then sampled. s169 00:11:20.480 --> 00:11:25.200 So by default we take 5% random sample but we also want to look at everything interesting. s170 00:11:25.200 --> 00:11:31.279 By interesting I mean conversations are handoff fail those are particularly long ones where the user is angry. s171 00:11:31.279 --> 00:11:33.600 We then score these with our LLM as a judge. s172 00:11:33.600 --> 00:11:36.959 So that's that raft and dimension of metrics that I talked about earlier. s173 00:11:36.959 --> 00:11:39.839 And then absolutely all of this ends up in comedic. s174 00:11:39.839 --> 00:11:42.240 So traces, scores, experiments. s175 00:11:42.240 --> 00:11:44.240 It's governed by a couple of bank constraints here. s176 00:11:44.240 --> 00:11:46.800 So we work in a completely airgapped environment. s177 00:11:46.800 --> 00:11:48.640 So no internet access to anything. s178 00:11:48.640 --> 00:11:50.800 So judge model is reached over a private path. s179 00:11:50.800 --> 00:11:52.640 We have an internal geni gateway. s180 00:11:52.640 --> 00:11:53.839 Comet a SAS product. s181 00:11:53.839 --> 00:11:57.120 We're consuming that as a signed partner app via AWS. s182 00:11:57.120 --> 00:11:59.440 Second around personally identifiable information. s183 00:11:59.440 --> 00:12:02.640 We have a zero tolerance policy for any PII leaving our network. s184 00:12:02.640 --> 00:12:08.240 So that means a lot of this evaluation happens without that present different cadences here. s185 00:12:08.240 --> 00:12:10.880 So tracers are being captured as they happen. s186 00:12:10.880 --> 00:12:13.440 We're doing offline scoring every four hours. s187 00:12:13.440 --> 00:12:16.399 And then regress this is also built into our CI/CD. s188 00:12:16.399 --> 00:12:22.880 So if we make changes to prompts or to context that can have like an outsized impact on what the agent is actually responding. s189 00:12:22.880 --> 00:12:27.680 So we need to constantly be comparing versus a known good baseline. s190 00:12:27.680 --> 00:12:28.959 We do that in CI/CD. s191 00:12:28.959 --> 00:12:30.320 There's a hard regression gate. s192 00:12:30.320 --> 00:12:34.639 So changes can't be merged in without us understanding exactly what's going to occur. s193 00:12:34.639 --> 00:12:39.200 I guess key point is that continuous evaluation is part of your production infrastructure. s194 00:12:39.200 --> 00:12:44.000 It's not a notebook that you run a couple of times when your model goes live. s195 00:12:44.240 --> 00:12:47.120 So there's different models here and they've got different failure modes. s196 00:12:47.120 --> 00:12:50.800 So on the right hand side we've got the agents, we've got the generative AI. s197 00:12:50.800 --> 00:12:52.560 So generative AI regresses. s198 00:12:52.560 --> 00:12:57.279 So as we've just talked about, somebody can make changes to a prompt and that can have outsized impact on the behavior. s199 00:12:57.279 --> 00:12:59.680 That's why we have the CI/CD in place. s200 00:12:59.680 --> 00:13:03.600 You can see here a real opic trace which is looking from our payment agent. s201 00:13:03.600 --> 00:13:14.480 Every node in the LAN graph uh becomes a different span and when the agent is slow or is breaking in a particular place, this allows us or an agent to jump in and investigate exactly why s202 00:13:14.480 --> 00:13:15.760 on the classical. s203 00:13:15.760 --> 00:13:20.000 So the intent classifier is traditional ML and thus we've got to monitor it in a more classical way. s204 00:13:20.000 --> 00:13:21.519 It drifts over time, right? s205 00:13:21.519 --> 00:13:25.600 So human conversation will naturally evolve from the point at which the model was trained. s206 00:13:25.600 --> 00:13:32.480 So we need to look at more traditional things like input feature drift, F1 score based off human scores, sorry, human labels. s207 00:13:32.480 --> 00:13:38.399 And then you know we have rag thresholds defined as to exactly when we need to trigger retraining for that model. s208 00:13:38.399 --> 00:13:40.800 On the right you can see an npm data drift chart. s209 00:13:40.800 --> 00:13:43.519 I I really wouldn't underestimate like input feature drift here. s210 00:13:43.519 --> 00:13:50.800 It's a really good canary as to when your model is starting to decline before your F1 score does implying the need for retraining. s211 00:13:50.800 --> 00:13:57.519 So two different problems, two different instruments, but what we don't want is two different observability and evaluation frameworks. s212 00:13:57.519 --> 00:14:03.519 Comet's really useful there because it brings in both OPIC for the Genai and MPM model performance monitor for the ML. s213 00:14:03.519 --> 00:14:11.120 So one platform how do we actually deploy this then without kind of breaking the bank and creating a load of headlines. s214 00:14:11.120 --> 00:14:12.480 We start off with a shadow deployment. s215 00:14:12.480 --> 00:14:21.760 So what I mean by that is we put this live uh we use it to start classifying cutter utterancerances but this existing scripted solution is what will actually respond to the customer s216 00:14:21.760 --> 00:14:30.800 and that means that we can get a sense of the real data you know the the input uh the input space for different customer queries with almost zero risk. s217 00:14:30.800 --> 00:14:42.720 I think this shadow mode is like one of the more transferable bits of this project for us because often we have this deadlock in a regulated industry where obviously we can't go live and get production data without the approval to do so. s218 00:14:42.720 --> 00:14:52.480 But equally to actually enable us to go live in many cases our model risk colleagues increasingly want to see how our models are going to react to that actual production data. s219 00:14:52.480 --> 00:15:00.160 This allows us to kind of break that apart so we can break apart observe safely from actually answer customer queries. s220 00:15:00.160 --> 00:15:06.639 uh and it takes one really big decision point, a big bang release into a series of smaller ones which we can make confidently. s221 00:15:06.639 --> 00:15:07.920 We then go brand by brand. s222 00:15:07.920 --> 00:15:15.680 So we have like percentage based rollout within the brand and we have kind of KPIs which allow us to determine go no-go at various points along there. s223 00:15:15.680 --> 00:15:20.639 Some of those decisions are automated and some of those require manual approval. s224 00:15:20.959 --> 00:15:24.399 Um I think and yeah some of those require manual approval. s225 00:15:24.399 --> 00:15:28.399 What kind of makes that possible is that we make a lot of use of feature flagging here. s226 00:15:28.399 --> 00:15:39.440 So there's like a global flag that we can trigger to send everything back to the scripted solution, but also every single backend can be isolated either globally or on a particular dimension such as brand s227 00:15:39.440 --> 00:15:42.480 and all of that can be done in seconds. s228 00:15:42.480 --> 00:15:47.199 Before I wrap up, let me maybe just talk very briefly and pull this back to that complaints project. s229 00:15:47.199 --> 00:15:53.600 So just to remind you that was the colleague facing agent which writes investigation summaries for summaries for complaint handlers. s230 00:15:53.600 --> 00:15:55.920 In that project, human feedback is a gate. s231 00:15:55.920 --> 00:15:56.880 It's not a thumbs up. s232 00:15:56.880 --> 00:15:58.880 It's not a tickbox activity. s233 00:15:58.880 --> 00:16:04.639 So if a complaint handler wants to rerun their investigation, they can't do so until they provided human feedback. s234 00:16:04.639 --> 00:16:07.759 We want to understand why they felt the need to do that rerun. s235 00:16:07.759 --> 00:16:18.800 But also it's allowed us to build up this quality data set of 14 14,000 customer of uh of colleague labels I guess in this case and that's really valuable in a lot of projects. s236 00:16:18.800 --> 00:16:23.360 We struggle to get a tenth of that number and I think the projects have also traded nicely here. s237 00:16:23.360 --> 00:16:32.079 So you know they found that judge problem in production we have judge calibration and regression checks because of that but equally the architecture that we built which relies heavily on commodopic s238 00:16:32.079 --> 00:16:48.480 they don't use that yet and they're going to be adopting that for their production monitoring I guess it's a glimpse inside an enterprise right so the the agent was the cheap part here that's code that we can produce to solve the business problem the valuation spine is the bit that we can then reuse again and again s239 00:16:48.480 --> 00:16:56.560 so to wrap up like if you are going to be shipping customerf facing agents in a regulated environment, then these are probably the points I would give you. s240 00:16:56.560 --> 00:16:57.839 I'm not going to go through everything. s241 00:16:57.839 --> 00:16:59.440 I'll maybe just pick a couple. s242 00:16:59.440 --> 00:17:04.240 So, three and four here, they're the they're the points with the actual production judge data, right? s243 00:17:04.240 --> 00:17:08.079 So, that judge pinned near the top, it's a broken instrument. s244 00:17:08.079 --> 00:17:10.400 Your judges are going to want to be overly positive. s245 00:17:10.400 --> 00:17:11.600 You need to fight against that. s246 00:17:11.600 --> 00:17:13.760 Compare the spans, not just the averages. s247 00:17:13.760 --> 00:17:16.559 Ensure the judge is actually matching the human reviewer. s248 00:17:16.559 --> 00:17:21.199 Fourth, the judge model itself does become part of that threshold config, right? s249 00:17:21.199 --> 00:17:29.120 So if you are going to be making changes to the judge model then uh you know you can introduce a breaking change without knowing it. s250 00:17:29.120 --> 00:17:30.880 Comet is obviously absolutely key here. s251 00:17:30.880 --> 00:17:36.880 So we have that for both halves both the generative AI and the traditional ML which we do still use a little bit. s252 00:17:36.880 --> 00:17:43.039 So and I guess the the final point here is next month I'll find out how much of this is actually true when we go live. s253 00:17:43.039 --> 00:17:44.960 So thanks a lot for listening. s254 00:17:44.960 --> 00:17:47.039 It was really a pleasure to get to speak to all of you. s255 00:17:47.039 --> 00:17:48.640 So I'll be at the comment booth after this. s256 00:17:48.640 --> 00:17:52.880 So if you do have any questions, you know, disagreements, etc., really happy to hear them. s257 00:17:52.880 --> 00:17:55.840 But thanks a lot for your time.