Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse

AI Engineer · 19 min · 176 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Uh I'm leading AI engineering solutions at LangFuse uh by ClickHouse.
00:16And uh we are an a agent observability and e-boss platform that a lot of users use open source.
00:23And uh I'm here to talk to you about an experience that we had with one of a top five bank helping them getting through how they can put a safety
00:32uh guardrails whenever they were shipping AI applications into production.
00:36And uh to make sure that they are running with the e-proper e-vals on that.
00:41I I like to start with uh you know, something that we learned with the experience with this customer is that regulated industries they don't necessarily care about the public benchmarks.
00:52I know that every week uh when anytime a new model comes out um we see all the benchmarks, you know, uh Opus 5 is now leading the benchmarks in a in a multitude of dimensions
01:04and that's great.
01:05Um that tells us what the models are really good at in terms of like the technical capabilities and maybe using sometimes like open data sets that are out there.
01:15But sometimes that doesn't mean that it will translate effectively to internal data and use cases that you have in in your organization.
01:23Specifically around you know, financial services where banks are very very sensitive to the their data, their use cases and for the most part they like to believe that that's part of their secret sauce, the way that they you know, build their own models, build their own
01:39uh data sets and also not exposing a lot of that data to uh to the public.
01:44Um so that's why we needed to figure out a way that the you know, financial services and this bank in particular that they can then say, "Look, we don't
01:56necessarily care what the model can do.
01:59What we care about is really whether this application that we're trying to build can ship into production.
02:05A good example would be like a 10K analyst that they were trying to build for their investment bankers.
02:10You know, every time an earnings call is is out there, they want to make sure that their teams can accelerate and get leverage from all the information that is out there and as soon as an earnings call is done, they can get an analysis on all the transcripts
02:22and all that.
02:24However, that is not just a simple task of just taking that the call transcript and just like summarizing it and what not because there's a lot of knowledge that the analyst has and a lot of context about how the bank and the firm operates
02:37that is not necessarily ingrained or trained in the models out of the box.
02:42So, how do then you bring that knowledge and that context into the model?
02:47And then furthermore, once you've done that, how do then you reliably are able to evaluate that the output that is giving you are the the the correct ones and that you can kind of clear
02:57those gates that you have put in place.
03:01So, that was that was again a kind of a a big learning for us that they and they told us directly, "Look, we don't certify models here.
03:09We certify applications, right?
03:11So, it it could be a 10K analyst, it could be an investment analysis and what not."
03:17The other learning from this experience is that in regulated industries and in finance in particular, there are three main requirements when it comes to building the solutions.
03:30The first one that more more often than not, they wanted to stay on their own premises.
03:35Um and this could be either their own data centers because it's it's funny how things are kind of coming back to to to center because we have done we have been doing all this movement into the cloud,
03:47but now because of AI and how sensitive the information that we're feeding these models is, then especially banks are rethinking and repurposing a lot of their data centers that they had and actually standing it up again and and kind of standing up GPU infrastructure
04:03to run models locally because of the sensitivity of the data that they are dealing with on these models.
04:10The other thing is the other requirements that it has to be gated.
04:13It cannot be just going out to the internet obvious for obvious reasons.
04:17They don't want to have even the the appearance that some model provider will train on their data even if there is like zero data retention and all and all these, you know, guarantees.
04:30But banks are very sensitive to that and they want to make sure that that stays within their control and sometimes you have even banks that actually have kind of research divisions that actually
04:40spend time and have PhDs training their own models and fine-tuning them and all that.
04:45And the third aspect is with all of this is that the deployment needs to be air-gapped, right?
04:50So that only the people that needs to see that data and it's within their infrastructure.
04:55That is very very critical.
04:58The The other thing is when when we were talking to them it's like, "Okay, now you have all these platform, we have all these models, but then we need to prove
05:07that the application is actually working to the way that we want.
05:11One, second, that is actually meeting the compliance requirements that we have as a regulated industry.
05:17And third, that our users are not going to come to us complaining that is not usable because then, you know, it becomes a situation where you don't get user adoption and then there's a dissatisfaction
05:28and a bad sentiment about using AI-enabled solutions.
05:32So we needed to find a way to help them come up with a framework and a methodology for them to have like a stamp of approval pass fail if an application
05:40or an AI agent had any change on their configuration.
05:44That could be the system prompt, that could be the model that they're using, a new tool, but then how do I give that, "Okay, this is ready to go into production or not based on on these changes that
05:58we have given them?"
06:00And then if we go like very high-level, how does that work mechanically?
06:05Right?
06:06The first thing that we need is to build a golden data set.
06:10That's the first thing.
06:10That's kind of the anchor.
06:12Um a lot of this good good news is a lot of these banks they typically already have a golden data set that they want to test against.
06:20But sometimes what is missing is how they can operationalize the testing of the test cases against that data set.
06:28But that's to me typically is the anchor.
06:30The way that I like to think about this if you're coming from a software engineering background is is these are your test cases, right?
06:36You will never push anything into production without passing it through QA, right?
06:41So this is the same principle.
06:42Then once you run this um and and you operationalize it, you're going to generate a bunch of traces.
06:49And the traces are the evidence of what the model was trained on.
06:53Or sorry, what the model was evaluated on, right?
06:57Against this data set.
06:58So you have that evidence on the trace.
07:00And then at the end with the trace as the as the as the kind of the uh as that, you know, evidence that you have, you can then generate the scores
07:11that are going to give you the pass fail across different dimensions that you uh select.
07:17When we started out this journey with this customer, it was we first started with a couple of with a bunch of models, um a small large models with different levels of intelligence.
07:27And we noticed something interesting, right?
07:30So this one is is a particular example of two numbers that stood out for us.
07:35The The first one, 97% in groundness, is that this model was really good at extracting information from, you know, internal sources and grounding the responses on actual data that the bank had.
07:48Perfect.
07:4997% That is a passing bar.
07:51Amazing.
07:52But then when it came down to give, you know, a accurate numbers, then the number was trip the model was tripping up.
08:00Because again, LLMs inherently are not great at math.
08:04They're just next token predictors.
08:06And then what happened was that even though it was grounded the responses on actual information from the bank, whenever it needed to give financial numbers or just do simple math, the model was failing.
08:20And then, I'll show you in a minute how we were able to go about that.
08:24And this leads us to then, again, what I said before, that they don't approve models.
08:29They approve applications.
08:31So, how they can put a harness or controls around this model, so irrespective of if tomorrow I swap a model, then the quality of my application will still remain the same.
08:46Now, we're expanding into kind of the process that we help them establish.
08:52You're starting from the left with the golden data set, then you go into the inside of the agent.
08:57The agent is going to do a bunch of tasks.
08:59It's going to have a planning phase.
09:01It's going to retrieve data from your official official sources.
09:05And then, you see the little box that is is circled in blue, he's going to calculate.
09:10In this case, what we did, we gave the model a tool for those areas where the model was com- coming up short, right?
09:18The 60% where it was not scoring well, then we said, "You know what?
09:22This is a task that the model doesn't need to do on its own.
09:25We can just give it a tool because math, you know, is deterministic.
09:28It's it's going to be the same all the time.
09:31And then, we can improve the the accuracy of the model."
09:35But at the same time, if you notice all the dimensions that we have, we are evaluating the application against a multitude of dimensions, not just one.
09:45We're not going to say, "Well, because everything in the average is giving me above 90%, then it's going to pass."
09:52That might not be the case.
09:54You might have different degrees or thresholds for the different categories that you have here in terms of the accuracy, regulatory compliance, and whatnot.
10:03And then, at the end is where you define your gate.
10:05You see, not anything that doesn't clear the gate based on your thresholds, doesn't get promoted into production.
10:13This is just a a snippet of the code and and at the end I'll link to the repo and to the blog post so you can read more in depth
10:22about the work that we did here, but it's effectively you want to build your own benchmark instead of relying on the public ones that are available there.
10:30And the the first thing you need to do that is to build the data set like I said before.
10:35This is an example of what a typical data set would look like.
10:38It's They are key pairs.
10:41You have inputs, expected outputs, and some metadata that you are then using to evaluate how the agent is performing both in terms of quality, behavior, and the responses that it's giving you as well.
10:57This is how it looks after we have a scored or evaluated one agent run.
11:04And you notice that I circled the calculate tool call because in this case now, this is how we were able to get the model from 60% in numerical accuracy
11:15all the way down to like 95% because we gave it a tool.
11:19Anytime there's a number that needs to be calculated, just use this little Python code and off you go.
11:25In In that way, you can improve the the results that you're giving back to your to your customers.
11:32The other step on on this is about scoring.
11:36How do you score each of the dimensions that I was telling you before?
11:40We have a couple of ways that you can do this.
11:42You can have deterministic scores that again, like the name implies, is, you know, very very simple.
11:48This is This is not LLM based.
11:50You just have a code that is running and it's for the same inputs, it's going to generate the same outputs all the time.
11:57And what we've noticed is with these solutions as they get more complicated, you need a blend of scores and evaluators that are deterministic and non-deterministic because otherwise relying 100%
12:11just on LLM judges is not going to cut it.
12:13Um and then this the second one, the path, is you also want to figure out if the agent is taking the right, you know, route or path to get to the answer.
12:24Because it might be that the answer might look okay or complete uh or semi-complete, but then maybe the route it took to get there, it might not be the appropriate one.
12:36Might be missing steps.
12:37So, you also can evaluate that.
12:39And the third one how you use an LLM as a judge to judge some of the more qualitative aspects of that answer.
12:48And by doing all of this, then we got to taking this little model that was failing with 60%, we got it to over 90% and then we are clearing the 85%
13:01gate that the bank had for this particular use case by adding that tool, but also by adding this evaluation framework on top that then we can measure the accuracy and the quality of the agent across many different dimensions.
13:15And this is something very important because now you can go back and show the evidence of how you actually measure the quality of the agent, but also more importantly, you can link it to the changes that were made into the agent, into the configuration,
13:32system prompt, model, tools that were used, and then you say, "Oh, but look at how by adding this tool, changing this model, we went from 60% accuracy over to 95,
13:4490 plus percent accuracy."
13:48And now this is where you start defining your gates, right?
13:51And this is where working with your domain experts becomes very, very critical.
13:56You don't want to just rely on technical kind of ability for this.
14:01You want to make sure that if you're delivering uh an a risk use case that is going to be in a in a enabled, that you have the risk people involved to tell you
14:13what is the bar that it needs to meet for them to feel comfortable putting this either internally or in front of customers?
14:21And that bar might be different depending on the use case.
14:24So, that is why you need to have first the dimensions or the scores uh on which you're going to evaluate this application.
14:32And then for each of those and for each use case or application, you end up with different thresholds that the agent needs to meet in order for it to have {quote}
14:41that stamp of approval uh for it to go into production.
14:46And this is what it looks like when you we have like a full end-to-end run across, you know, we had one data set that it was like 150 questions, another one was like 1,500.
14:59So, but now you can measure the configuration of the agent across these different dimensions and vertices.
15:07And you see how even though it is passing on groundness, completeness to use, you might say, "Well, if I take the average of all of these, then it should be okay."
15:19But in regulatory compliance is getting a zero.
15:22So, that is one that is just binary.
15:24It's just either zero or 100.
15:26You don't want to put anything if you're a bank out there that is going to violate any regulatory compliance requirement that you have because you might get penalties, you might incur into reputational damage,
15:37and all of that.
15:38So, it might be that one dimension actually completely shuts down the gate for you to move this into production.
15:46So, that's why it's critical to have a multi-pronged view on how you evaluate your agents.
15:53And this is where when you start to kind of picking up all these failures or what we call them failure modes, this is where bringing back to the main experts is really critical.
16:03You can curate for them what we would call an annotation queue.
16:07So, then say, "Anytime that there is a failure on regulatory compliance, I want to curate those specific traces or evidence, right, for my evaluation and put it in a place that my domain expert can come in and look at the trace, look at what happened,
16:24and then give me feedback.
16:26And with that feedback, I can then go back and, you know, feedback into the agent to improve it.
16:32And this is what it would look like connecting it to the process before, but including that human expert into the middle.
16:40If you notice down below, that anytime a couple of things are happening here, anytime a prompt gets promoted or a change in the configuration, then it triggers a rerun
16:51of the gate clearance process or the evaluation process.
16:55But then, anytime a rewriter or a human expert comes in and gives me feedback, that is a potential candidate to go back into my golden data set.
17:05Because now we have evidence from runs of the agent that have been reviewed, and now we can say, "Oh, actually this one piece is a better, you know, bar
17:16to evaluate this particular dimension on the agent."
17:20And this is kind of the whole process end to end to make sure that we are including the the domain experts, but also making sure that we're clearing our our quality gates.
17:31Then this got us to a place where we were able to help them go from weeks to minutes when it comes to, you know, evaluating a applications, especially in like risk and compliance, which typically take even months sometimes.
17:45So, this process it becomes very iterative.
17:48We can be very specific and focus on the areas that we want on the agent.
17:53Then there's no template for this.
17:55Um, you know, this industry's been been developed right now.
17:59So, typically you need to define what is it that you want to evaluate in in in kind of the metrics that matter to you.
18:07And the three things that I like to leave you with is you need to write your own benchmark because like I said, the public ones don't really matter for your data because they were not done on your data per se.
18:18The second one, you got you have to gate the application, not the model.
18:23That gives you more flexibility if tomorrow you want to swap a model, you have already all the controls in the application and in the configuration of the agent, and then it becomes easier for you to adjust.
18:34And then the third, let me make sure that the domain experts come come in and set the bar or the thresholds that you are deem reasonable in order for you to promote this into production.
18:46And finally, you know, use your data, use your bar, use your gates, especially in regulated industries where you want to keep things in your premises and you're probably not relying
18:57on publicly available information.
18:59So, thank you for the time.
19:00I know I'm a little bit over, but I wanted to give you with the detailed blog post of this work that we did.
19:06It also links to the repo that you can reproduce this yourself, and I'm around if you have any questions.
19:12So, thank you.