WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=-uBABM2AuFg NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/-uBABM2AuFg.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.640 --> 00:00:16.880 Uh I'm leading AI engineering solutions at LangFuse uh by ClickHouse. s3 00:00:16.880 --> 00:00:23.320 And uh we are an a agent observability and e-boss platform that a lot of users use open source. s4 00:00:23.320 --> 00:00:32.240 And uh I'm here to talk to you about an experience that we had with one of a top five bank helping them getting through how they can put a safety s5 00:00:32.240 --> 00:00:36.560 uh guardrails whenever they were shipping AI applications into production. s6 00:00:36.560 --> 00:00:41.840 And uh to make sure that they are running with the e-proper e-vals on that. s7 00:00:41.840 --> 00:00:52.440 I I like to start with uh you know, something that we learned with the experience with this customer is that regulated industries they don't necessarily care about the public benchmarks. s8 00:00:52.440 --> 00:01:04.040 I know that every week uh when anytime a new model comes out um we see all the benchmarks, you know, uh Opus 5 is now leading the benchmarks in a in a multitude of dimensions s9 00:01:04.040 --> 00:01:05.440 and that's great. s10 00:01:05.440 --> 00:01:15.480 Um that tells us what the models are really good at in terms of like the technical capabilities and maybe using sometimes like open data sets that are out there. s11 00:01:15.480 --> 00:01:23.400 But sometimes that doesn't mean that it will translate effectively to internal data and use cases that you have in in your organization. s12 00:01:23.400 --> 00:01:39.920 Specifically around you know, financial services where banks are very very sensitive to the their data, their use cases and for the most part they like to believe that that's part of their secret sauce, the way that they you know, build their own models, build their own s13 00:01:39.920 --> 00:01:44.760 uh data sets and also not exposing a lot of that data to uh to the public. s14 00:01:44.760 --> 00:01:56.840 Um so that's why we needed to figure out a way that the you know, financial services and this bank in particular that they can then say, "Look, we don't s15 00:01:56.840 --> 00:01:59.200 necessarily care what the model can do. s16 00:01:59.200 --> 00:02:05.412 What we care about is really whether this application that we're trying to build can ship into production. s17 00:02:05.412 --> 00:02:10.572 A good example would be like a 10K analyst that they were trying to build for their investment bankers. s18 00:02:10.572 --> 00:02:22.972 You know, every time an earnings call is is out there, they want to make sure that their teams can accelerate and get leverage from all the information that is out there and as soon as an earnings call is done, they can get an analysis on all the transcripts s19 00:02:22.972 --> 00:02:24.052 and all that. s20 00:02:24.052 --> 00:02:37.812 However, that is not just a simple task of just taking that the call transcript and just like summarizing it and what not because there's a lot of knowledge that the analyst has and a lot of context about how the bank and the firm operates s21 00:02:37.812 --> 00:02:42.852 that is not necessarily ingrained or trained in the models out of the box. s22 00:02:42.852 --> 00:02:47.132 So, how do then you bring that knowledge and that context into the model? s23 00:02:47.132 --> 00:02:57.972 And then furthermore, once you've done that, how do then you reliably are able to evaluate that the output that is giving you are the the the correct ones and that you can kind of clear s24 00:02:57.972 --> 00:03:01.252 those gates that you have put in place. s25 00:03:01.252 --> 00:03:09.852 So, that was that was again a kind of a a big learning for us that they and they told us directly, "Look, we don't certify models here. s26 00:03:09.852 --> 00:03:11.772 We certify applications, right? s27 00:03:11.772 --> 00:03:17.972 So, it it could be a 10K analyst, it could be an investment analysis and what not." s28 00:03:17.972 --> 00:03:30.132 The other learning from this experience is that in regulated industries and in finance in particular, there are three main requirements when it comes to building the solutions. s29 00:03:30.132 --> 00:03:35.332 The first one that more more often than not, they wanted to stay on their own premises. s30 00:03:35.332 --> 00:03:47.412 Um and this could be either their own data centers because it's it's funny how things are kind of coming back to to to center because we have done we have been doing all this movement into the cloud, s31 00:03:47.412 --> 00:04:03.334 but now because of AI and how sensitive the information that we're feeding these models is, then especially banks are rethinking and repurposing a lot of their data centers that they had and actually standing it up again and and kind of standing up GPU infrastructure s32 00:04:03.334 --> 00:04:10.854 to run models locally because of the sensitivity of the data that they are dealing with on these models. s33 00:04:10.854 --> 00:04:13.734 The other thing is the other requirements that it has to be gated. s34 00:04:13.734 --> 00:04:17.534 It cannot be just going out to the internet obvious for obvious reasons. s35 00:04:17.534 --> 00:04:30.214 They don't want to have even the the appearance that some model provider will train on their data even if there is like zero data retention and all and all these, you know, guarantees. s36 00:04:30.214 --> 00:04:40.414 But banks are very sensitive to that and they want to make sure that that stays within their control and sometimes you have even banks that actually have kind of research divisions that actually s37 00:04:40.414 --> 00:04:45.694 spend time and have PhDs training their own models and fine-tuning them and all that. s38 00:04:45.694 --> 00:04:50.614 And the third aspect is with all of this is that the deployment needs to be air-gapped, right? s39 00:04:50.614 --> 00:04:55.654 So that only the people that needs to see that data and it's within their infrastructure. s40 00:04:55.654 --> 00:04:58.214 That is very very critical. s41 00:04:58.214 --> 00:05:07.014 The The other thing is when when we were talking to them it's like, "Okay, now you have all these platform, we have all these models, but then we need to prove s42 00:05:07.014 --> 00:05:11.774 that the application is actually working to the way that we want. s43 00:05:11.774 --> 00:05:17.614 One, second, that is actually meeting the compliance requirements that we have as a regulated industry. s44 00:05:17.614 --> 00:05:28.334 And third, that our users are not going to come to us complaining that is not usable because then, you know, it becomes a situation where you don't get user adoption and then there's a dissatisfaction s45 00:05:28.334 --> 00:05:32.014 and a bad sentiment about using AI-enabled solutions. s46 00:05:32.014 --> 00:05:40.734 So we needed to find a way to help them come up with a framework and a methodology for them to have like a stamp of approval pass fail if an application s47 00:05:40.734 --> 00:05:44.894 or an AI agent had any change on their configuration. s48 00:05:44.894 --> 00:05:58.174 That could be the system prompt, that could be the model that they're using, a new tool, but then how do I give that, "Okay, this is ready to go into production or not based on on these changes that s49 00:05:58.174 --> 00:06:00.574 we have given them?" s50 00:06:00.574 --> 00:06:05.855 And then if we go like very high-level, how does that work mechanically? s51 00:06:05.855 --> 00:06:06.255 Right? s52 00:06:06.255 --> 00:06:10.095 The first thing that we need is to build a golden data set. s53 00:06:10.095 --> 00:06:10.815 That's the first thing. s54 00:06:10.815 --> 00:06:12.535 That's kind of the anchor. s55 00:06:12.535 --> 00:06:20.095 Um a lot of this good good news is a lot of these banks they typically already have a golden data set that they want to test against. s56 00:06:20.095 --> 00:06:28.215 But sometimes what is missing is how they can operationalize the testing of the test cases against that data set. s57 00:06:28.215 --> 00:06:30.535 But that's to me typically is the anchor. s58 00:06:30.535 --> 00:06:36.815 The way that I like to think about this if you're coming from a software engineering background is is these are your test cases, right? s59 00:06:36.815 --> 00:06:41.015 You will never push anything into production without passing it through QA, right? s60 00:06:41.015 --> 00:06:42.735 So this is the same principle. s61 00:06:42.735 --> 00:06:49.575 Then once you run this um and and you operationalize it, you're going to generate a bunch of traces. s62 00:06:49.575 --> 00:06:53.855 And the traces are the evidence of what the model was trained on. s63 00:06:53.855 --> 00:06:57.135 Or sorry, what the model was evaluated on, right? s64 00:06:57.135 --> 00:06:58.375 Against this data set. s65 00:06:58.375 --> 00:07:00.455 So you have that evidence on the trace. s66 00:07:00.455 --> 00:07:11.015 And then at the end with the trace as the as the as the kind of the uh as that, you know, evidence that you have, you can then generate the scores s67 00:07:11.015 --> 00:07:17.415 that are going to give you the pass fail across different dimensions that you uh select. s68 00:07:17.455 --> 00:07:27.615 When we started out this journey with this customer, it was we first started with a couple of with a bunch of models, um a small large models with different levels of intelligence. s69 00:07:27.615 --> 00:07:30.255 And we noticed something interesting, right? s70 00:07:30.255 --> 00:07:35.175 So this one is is a particular example of two numbers that stood out for us. s71 00:07:35.175 --> 00:07:48.735 The The first one, 97% in groundness, is that this model was really good at extracting information from, you know, internal sources and grounding the responses on actual data that the bank had. s72 00:07:48.735 --> 00:07:49.175 Perfect. s73 00:07:49.175 --> 00:07:51.375 97% That is a passing bar. s74 00:07:51.375 --> 00:07:52.415 Amazing. s75 00:07:52.415 --> 00:08:00.455 But then when it came down to give, you know, a accurate numbers, then the number was trip the model was tripping up. s76 00:08:00.455 --> 00:08:04.535 Because again, LLMs inherently are not great at math. s77 00:08:04.535 --> 00:08:06.375 They're just next token predictors. s78 00:08:06.375 --> 00:08:20.297 And then what happened was that even though it was grounded the responses on actual information from the bank, whenever it needed to give financial numbers or just do simple math, the model was failing. s79 00:08:20.297 --> 00:08:24.057 And then, I'll show you in a minute how we were able to go about that. s80 00:08:24.057 --> 00:08:29.857 And this leads us to then, again, what I said before, that they don't approve models. s81 00:08:29.857 --> 00:08:31.657 They approve applications. s82 00:08:31.657 --> 00:08:46.257 So, how they can put a harness or controls around this model, so irrespective of if tomorrow I swap a model, then the quality of my application will still remain the same. s83 00:08:46.577 --> 00:08:52.017 Now, we're expanding into kind of the process that we help them establish. s84 00:08:52.017 --> 00:08:57.897 You're starting from the left with the golden data set, then you go into the inside of the agent. s85 00:08:57.897 --> 00:08:59.817 The agent is going to do a bunch of tasks. s86 00:08:59.817 --> 00:09:01.417 It's going to have a planning phase. s87 00:09:01.417 --> 00:09:05.657 It's going to retrieve data from your official official sources. s88 00:09:05.657 --> 00:09:10.417 And then, you see the little box that is is circled in blue, he's going to calculate. s89 00:09:10.417 --> 00:09:18.857 In this case, what we did, we gave the model a tool for those areas where the model was com- coming up short, right? s90 00:09:18.857 --> 00:09:22.417 The 60% where it was not scoring well, then we said, "You know what? s91 00:09:22.417 --> 00:09:25.617 This is a task that the model doesn't need to do on its own. s92 00:09:25.617 --> 00:09:28.897 We can just give it a tool because math, you know, is deterministic. s93 00:09:28.897 --> 00:09:31.257 It's it's going to be the same all the time. s94 00:09:31.257 --> 00:09:35.337 And then, we can improve the the accuracy of the model." s95 00:09:35.337 --> 00:09:45.537 But at the same time, if you notice all the dimensions that we have, we are evaluating the application against a multitude of dimensions, not just one. s96 00:09:45.537 --> 00:09:52.937 We're not going to say, "Well, because everything in the average is giving me above 90%, then it's going to pass." s97 00:09:52.937 --> 00:09:54.017 That might not be the case. s98 00:09:54.017 --> 00:10:03.137 You might have different degrees or thresholds for the different categories that you have here in terms of the accuracy, regulatory compliance, and whatnot. s99 00:10:03.137 --> 00:10:05.737 And then, at the end is where you define your gate. s100 00:10:05.737 --> 00:10:13.658 You see, not anything that doesn't clear the gate based on your thresholds, doesn't get promoted into production. s101 00:10:13.658 --> 00:10:22.258 This is just a a snippet of the code and and at the end I'll link to the repo and to the blog post so you can read more in depth s102 00:10:22.258 --> 00:10:30.138 about the work that we did here, but it's effectively you want to build your own benchmark instead of relying on the public ones that are available there. s103 00:10:30.138 --> 00:10:35.378 And the the first thing you need to do that is to build the data set like I said before. s104 00:10:35.378 --> 00:10:38.898 This is an example of what a typical data set would look like. s105 00:10:38.898 --> 00:10:41.138 It's They are key pairs. s106 00:10:41.138 --> 00:10:57.338 You have inputs, expected outputs, and some metadata that you are then using to evaluate how the agent is performing both in terms of quality, behavior, and the responses that it's giving you as well. s107 00:10:57.338 --> 00:11:04.018 This is how it looks after we have a scored or evaluated one agent run. s108 00:11:04.018 --> 00:11:15.618 And you notice that I circled the calculate tool call because in this case now, this is how we were able to get the model from 60% in numerical accuracy s109 00:11:15.618 --> 00:11:19.978 all the way down to like 95% because we gave it a tool. s110 00:11:19.978 --> 00:11:25.738 Anytime there's a number that needs to be calculated, just use this little Python code and off you go. s111 00:11:25.738 --> 00:11:32.578 In In that way, you can improve the the results that you're giving back to your to your customers. s112 00:11:32.578 --> 00:11:36.058 The other step on on this is about scoring. s113 00:11:36.058 --> 00:11:40.098 How do you score each of the dimensions that I was telling you before? s114 00:11:40.098 --> 00:11:42.018 We have a couple of ways that you can do this. s115 00:11:42.018 --> 00:11:48.858 You can have deterministic scores that again, like the name implies, is, you know, very very simple. s116 00:11:48.858 --> 00:11:50.778 This is This is not LLM based. s117 00:11:50.778 --> 00:11:57.218 You just have a code that is running and it's for the same inputs, it's going to generate the same outputs all the time. s118 00:11:57.218 --> 00:12:11.298 And what we've noticed is with these solutions as they get more complicated, you need a blend of scores and evaluators that are deterministic and non-deterministic because otherwise relying 100% s119 00:12:11.298 --> 00:12:13.659 just on LLM judges is not going to cut it. s120 00:12:13.659 --> 00:12:24.659 Um and then this the second one, the path, is you also want to figure out if the agent is taking the right, you know, route or path to get to the answer. s121 00:12:24.659 --> 00:12:36.299 Because it might be that the answer might look okay or complete uh or semi-complete, but then maybe the route it took to get there, it might not be the appropriate one. s122 00:12:36.299 --> 00:12:37.619 Might be missing steps. s123 00:12:37.619 --> 00:12:39.859 So, you also can evaluate that. s124 00:12:39.859 --> 00:12:48.939 And the third one how you use an LLM as a judge to judge some of the more qualitative aspects of that answer. s125 00:12:48.979 --> 00:13:01.339 And by doing all of this, then we got to taking this little model that was failing with 60%, we got it to over 90% and then we are clearing the 85% s126 00:13:01.339 --> 00:13:15.539 gate that the bank had for this particular use case by adding that tool, but also by adding this evaluation framework on top that then we can measure the accuracy and the quality of the agent across many different dimensions. s127 00:13:15.539 --> 00:13:32.939 And this is something very important because now you can go back and show the evidence of how you actually measure the quality of the agent, but also more importantly, you can link it to the changes that were made into the agent, into the configuration, s128 00:13:32.939 --> 00:13:44.579 system prompt, model, tools that were used, and then you say, "Oh, but look at how by adding this tool, changing this model, we went from 60% accuracy over to 95, s129 00:13:44.579 --> 00:13:47.739 90 plus percent accuracy." s130 00:13:48.139 --> 00:13:51.579 And now this is where you start defining your gates, right? s131 00:13:51.579 --> 00:13:56.939 And this is where working with your domain experts becomes very, very critical. s132 00:13:56.939 --> 00:14:01.939 You don't want to just rely on technical kind of ability for this. s133 00:14:01.939 --> 00:14:13.219 You want to make sure that if you're delivering uh an a risk use case that is going to be in a in a enabled, that you have the risk people involved to tell you s134 00:14:13.219 --> 00:14:21.341 what is the bar that it needs to meet for them to feel comfortable putting this either internally or in front of customers? s135 00:14:21.341 --> 00:14:24.181 And that bar might be different depending on the use case. s136 00:14:24.181 --> 00:14:32.341 So, that is why you need to have first the dimensions or the scores uh on which you're going to evaluate this application. s137 00:14:32.341 --> 00:14:41.541 And then for each of those and for each use case or application, you end up with different thresholds that the agent needs to meet in order for it to have {quote} s138 00:14:41.541 --> 00:14:46.501 that stamp of approval uh for it to go into production. s139 00:14:46.741 --> 00:14:59.781 And this is what it looks like when you we have like a full end-to-end run across, you know, we had one data set that it was like 150 questions, another one was like 1,500. s140 00:14:59.781 --> 00:15:07.861 So, but now you can measure the configuration of the agent across these different dimensions and vertices. s141 00:15:07.861 --> 00:15:19.021 And you see how even though it is passing on groundness, completeness to use, you might say, "Well, if I take the average of all of these, then it should be okay." s142 00:15:19.021 --> 00:15:22.501 But in regulatory compliance is getting a zero. s143 00:15:22.501 --> 00:15:24.901 So, that is one that is just binary. s144 00:15:24.901 --> 00:15:26.821 It's just either zero or 100. s145 00:15:26.821 --> 00:15:37.701 You don't want to put anything if you're a bank out there that is going to violate any regulatory compliance requirement that you have because you might get penalties, you might incur into reputational damage, s146 00:15:37.701 --> 00:15:38.341 and all of that. s147 00:15:38.341 --> 00:15:46.821 So, it might be that one dimension actually completely shuts down the gate for you to move this into production. s148 00:15:46.821 --> 00:15:53.341 So, that's why it's critical to have a multi-pronged view on how you evaluate your agents. s149 00:15:53.341 --> 00:16:03.141 And this is where when you start to kind of picking up all these failures or what we call them failure modes, this is where bringing back to the main experts is really critical. s150 00:16:03.141 --> 00:16:07.061 You can curate for them what we would call an annotation queue. s151 00:16:07.061 --> 00:16:24.502 So, then say, "Anytime that there is a failure on regulatory compliance, I want to curate those specific traces or evidence, right, for my evaluation and put it in a place that my domain expert can come in and look at the trace, look at what happened, s152 00:16:24.502 --> 00:16:26.182 and then give me feedback. s153 00:16:26.182 --> 00:16:32.582 And with that feedback, I can then go back and, you know, feedback into the agent to improve it. s154 00:16:32.582 --> 00:16:40.102 And this is what it would look like connecting it to the process before, but including that human expert into the middle. s155 00:16:40.102 --> 00:16:51.822 If you notice down below, that anytime a couple of things are happening here, anytime a prompt gets promoted or a change in the configuration, then it triggers a rerun s156 00:16:51.822 --> 00:16:55.622 of the gate clearance process or the evaluation process. s157 00:16:55.622 --> 00:17:05.382 But then, anytime a rewriter or a human expert comes in and gives me feedback, that is a potential candidate to go back into my golden data set. s158 00:17:05.382 --> 00:17:16.502 Because now we have evidence from runs of the agent that have been reviewed, and now we can say, "Oh, actually this one piece is a better, you know, bar s159 00:17:16.502 --> 00:17:20.262 to evaluate this particular dimension on the agent." s160 00:17:20.262 --> 00:17:31.622 And this is kind of the whole process end to end to make sure that we are including the the domain experts, but also making sure that we're clearing our our quality gates. s161 00:17:31.622 --> 00:17:45.582 Then this got us to a place where we were able to help them go from weeks to minutes when it comes to, you know, evaluating a applications, especially in like risk and compliance, which typically take even months sometimes. s162 00:17:45.582 --> 00:17:48.942 So, this process it becomes very iterative. s163 00:17:48.942 --> 00:17:53.582 We can be very specific and focus on the areas that we want on the agent. s164 00:17:53.582 --> 00:17:55.862 Then there's no template for this. s165 00:17:55.862 --> 00:17:59.582 Um, you know, this industry's been been developed right now. s166 00:17:59.582 --> 00:18:07.422 So, typically you need to define what is it that you want to evaluate in in in kind of the metrics that matter to you. s167 00:18:07.422 --> 00:18:18.902 And the three things that I like to leave you with is you need to write your own benchmark because like I said, the public ones don't really matter for your data because they were not done on your data per se. s168 00:18:18.902 --> 00:18:23.303 The second one, you got you have to gate the application, not the model. s169 00:18:23.303 --> 00:18:34.103 That gives you more flexibility if tomorrow you want to swap a model, you have already all the controls in the application and in the configuration of the agent, and then it becomes easier for you to adjust. s170 00:18:34.103 --> 00:18:46.063 And then the third, let me make sure that the domain experts come come in and set the bar or the thresholds that you are deem reasonable in order for you to promote this into production. s171 00:18:46.063 --> 00:18:57.463 And finally, you know, use your data, use your bar, use your gates, especially in regulated industries where you want to keep things in your premises and you're probably not relying s172 00:18:57.463 --> 00:18:59.303 on publicly available information. s173 00:18:59.303 --> 00:19:00.663 So, thank you for the time. s174 00:19:00.663 --> 00:19:06.623 I know I'm a little bit over, but I wanted to give you with the detailed blog post of this work that we did. s175 00:19:06.623 --> 00:19:12.343 It also links to the repo that you can reproduce this yourself, and I'm around if you have any questions. s176 00:19:12.343 --> 00:19:14.743 So, thank you.