WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=obfPzbapMIM NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/obfPzbapMIM.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.719 --> 00:00:14.400 All right, let's get started. s2 00:00:14.400 --> 00:00:18.880 Welcome everybody to our workshop on evaluating AI applications with Lancuse. s3 00:00:18.880 --> 00:00:19.920 My name is Annabelle. s4 00:00:19.920 --> 00:00:24.640 I'm a um AI engineer uh AI solutions engineer at Langfuse. s5 00:00:24.640 --> 00:00:29.920 And I'm today joined by my um colleague Donelli who is leading AI solution architecture at Langfus. s6 00:00:29.920 --> 00:00:35.680 So working with our biggest customers on making sure they actually shipping reliable agents to production. s7 00:00:35.680 --> 00:00:43.760 I myself uh was in my previous life how I like to call it because it feels like a lifetime in a company building AI agents for medical device compliance. s8 00:00:43.760 --> 00:00:48.879 And in this industry it was super important to make sure we are running. s9 00:00:48.879 --> 00:00:58.079 We're making sure we have ground truth data set that is validated by our customers and we also ran a lot of human in the loop systems and a lot of my time I actually spent in Lenfuse s10 00:00:58.079 --> 00:01:09.040 and at some point I decided okay why not swap sites and also help other teams that are shipping agents especially also in regulated industries and uh joined the langu team uh roughly seven months ago. s11 00:01:09.040 --> 00:01:16.880 So but before we dive right into it we wanted to get a little bit of a feeling of who is here today and who we are um yeah coming together with. s12 00:01:16.880 --> 00:01:31.520 So uh we were wondering who of you has brought AI agents to production before and with that I mean been part of a team doing it not necessarily yes someone in the company did it so actually like been part of a process to bring it to production s13 00:01:31.520 --> 00:01:46.320 okay that's good amazing it's a lot of people and we hoped uh to have this also and who of you has worked with languages before okay quite some and any other kind of observability in evas tool s14 00:01:46.320 --> 00:01:46.640 Okay. s15 00:01:46.640 --> 00:01:49.119 Yeah, also a few uh amazing. s16 00:01:49.119 --> 00:01:53.680 And then finally, who has solved EVAs for AI applications? s17 00:01:54.159 --> 00:01:58.320 For the confident hands, I would love you to come by at our booth and explain us how. s18 00:01:58.320 --> 00:02:06.799 Um because actually it is quite hard and I think um all of you have experienced it if you're working with AI or also just chatting with Chat GPT or Claude. s19 00:02:06.799 --> 00:02:13.599 H it is quite tough uh because AI agents can fail while the system metrics themselves look green. s20 00:02:13.599 --> 00:02:21.599 So if you for example look at this example here we have a user support agent and the user is asking can I get a refund. s21 00:02:21.599 --> 00:02:33.120 The agent takes in the request looks up some policies and then it's like yeah sure I issued your refund and the user is like thank you and it will take some days until the user sees if it actually worked out s22 00:02:33.120 --> 00:02:34.640 and the status is green. s23 00:02:34.640 --> 00:02:38.080 Nothing went wrong in traditional observability terms. s24 00:02:38.080 --> 00:02:40.160 There's also no latency issues. s25 00:02:40.160 --> 00:02:45.200 But if we dig deeper into what actually happened, the agent never did refund anything. s26 00:02:45.200 --> 00:02:46.640 So the tool was never called. s27 00:02:46.640 --> 00:02:47.760 It was never executed. s28 00:02:47.760 --> 00:02:49.760 And the agent lied about doing it to the user. s29 00:02:49.760 --> 00:02:57.120 And we will only know about it in a few weeks when the user come backs and is like, "Hey, by the way, I never get I've never gotten my refund." s30 00:02:57.120 --> 00:03:01.200 And this is the problem that we have with those nondetermin deterministic systems. s31 00:03:01.200 --> 00:03:03.599 They might tell us they do things, but they actually don't. s32 00:03:03.599 --> 00:03:14.560 And this is why you have to find an approach and a way to continuously mine for those signals where things are going wrong and make sure they don't happen again or maybe not even at all in the first place. s33 00:03:14.560 --> 00:03:24.480 And this is a problem our founders also faced three years ago when they were in the way combinator batch and they realized okay there's nothing like a unit test for AI applications and we need a solution for it. s34 00:03:24.480 --> 00:03:31.599 So they built Langfuse which is by now an open source platform to continuously trace monitor and improve your AI systems. s35 00:03:31.599 --> 00:03:37.440 And while we started out as a dev tool mostly by now we also have domain experts as a target group. s36 00:03:37.440 --> 00:03:42.480 So collaborating with AI engineers to make sure the system does what it actually should be doing. s37 00:03:42.480 --> 00:03:53.840 Uh but also a new user group for the past probably 8 to 12 months uh which are coding agents that are helping our engineers also to work with AI applications and improve them. s38 00:03:54.560 --> 00:04:01.760 Lenfus is by now adopted by more than 150,000 uh developers globally also largely driven by our open source um setup. s39 00:04:01.760 --> 00:04:07.920 Of course, 21 of the Fortune 50 have us as part of their core tech stack when shipping AI applications. s40 00:04:07.920 --> 00:04:12.480 And every month more than 90 billion observations are ingested into uh length use instances. s41 00:04:12.480 --> 00:04:19.840 Uh of course increasing tendency as agents are taking more more and more steps and also taking over more and more tasks. s42 00:04:19.840 --> 00:04:28.240 Companies like RAMP, Canva, Khan Academy and SUP uh work with us and over the last three years we of course spend a lot of time with them and one thing we realized s43 00:04:28.240 --> 00:04:42.080 working with those companies is that the ones that successfully manage to ship reliable agents and also continuously improve them follow something that we call like some form of this and the AI engineering loop s44 00:04:42.080 --> 00:04:50.479 and it is about a set of practices that help you to continuously improve your system by taking a systematic approach. s45 00:04:50.479 --> 00:04:52.479 The foundation to this is always tracing. s46 00:04:52.479 --> 00:05:01.440 Tracing is about getting all the steps of what your agent is doing into a more or less lock, but ideally also in a way you can inspect it as a human. s47 00:05:01.440 --> 00:05:03.840 Monitoring is about finding the interesting ones. s48 00:05:03.840 --> 00:05:05.280 So what is going wrong? s49 00:05:05.280 --> 00:05:06.400 What is going right? s50 00:05:06.400 --> 00:05:07.759 Where should we look deeper into? s51 00:05:07.759 --> 00:05:11.520 But also understanding how is it performing on average. s52 00:05:11.520 --> 00:05:20.240 Building data sets is about finding a representative set of examples that you want your application to be able to handle and also how you want it to handle it s53 00:05:20.240 --> 00:05:22.240 and then you want to run experiments on it. s54 00:05:22.240 --> 00:05:33.759 So try different variations of your system to check um how it is performing, how it's going and then of course you want to evaluate how it's going and this is not something you do once but you continuously close this loop s55 00:05:33.759 --> 00:05:39.919 and want to improve uh those AI agents and learn from production and get get it back into data sets. s56 00:05:39.919 --> 00:05:45.199 Uh but let's dive in each of those steps a little bit more into detail. s57 00:05:45.680 --> 00:05:52.720 So for traces, the most important goal is that you want to get deep insights into what your agent is actually doing. s58 00:05:52.720 --> 00:06:00.000 On the right side of the screen, we can see a trace that is also a screenshot from Lenfuse and in this case it's an IT support agent. s59 00:06:00.000 --> 00:06:04.319 So an agent that is answering questions from your parents so you don't have to do it. s60 00:06:04.319 --> 00:06:12.000 And we can see that there's first like the root trace and we can see aggregate um time and also uh costs of this step. s61 00:06:12.000 --> 00:06:16.240 But we then also get a breakdown of each of the individual steps that agent is taking. s62 00:06:16.240 --> 00:06:23.520 So both the generations which are the open AAI chat ones and also the tool calls which we can see with a little tool in front of it. s63 00:06:23.520 --> 00:06:32.400 So we have a detailed view of what's going on and if the screen would be wider we would also see that we can see the inputs and outputs and all the metadata involved. s64 00:06:32.960 --> 00:06:47.199 This is then the foundation to track the cost, the latency, the tokens involved, and also have a log to inspect for you as a human to really dig into what's going on, but also for your coding agents to work with it and look at it at scale and just have this this inspectable lock of what's actually been s65 00:06:47.199 --> 00:06:49.360 going on. s66 00:06:50.000 --> 00:06:59.599 Monitoring is then about making sense of it at scale because especially now with increasing uh automation levels, longer traces and like a lot a lot of use cases. s67 00:06:59.599 --> 00:07:04.479 It's just not feasible anymore for you as a human to look through those logs completely manually. s68 00:07:04.479 --> 00:07:12.080 But you actually want a way to get aggregate metrics on what's actually going on and also uh detect signals where things are going wrong. s69 00:07:12.080 --> 00:07:22.479 So for example, cost and latency metrics, but maybe also quality signals like our compliance guardrails um kept true to or also like how many users are actually frustrated with our system. s70 00:07:22.479 --> 00:07:36.960 How many are writing all caps messages because they are maybe a bit upset with what uh the agent just did and um yeah all of this aggregating and then u of course also finding the interesting stuff that you want to dig deeper into. s71 00:07:37.440 --> 00:07:46.479 what you get from production through this monitoring is then also one of the foundations um that you can use to build out your data sets and data sets are in the end a representation s72 00:07:46.479 --> 00:07:49.680 of what you want your agent to handle in production. s73 00:07:49.680 --> 00:08:01.039 So all the cases that it might be getting also maybe out of scope requests that you want to like correctly also be deferred and make sure that like the agent is not trying to give investment advice or also s74 00:08:01.039 --> 00:08:06.000 uh answering what the weather in New York is going to be like if it's actually just a dead IT support agent. s75 00:08:06.000 --> 00:08:19.120 And those um items you can then have like the input but also what you expect um it to have as the output which can both be like a sample reply or also expected behavior that you want to see or keywords you want to have covered. s76 00:08:19.120 --> 00:08:27.039 And this can then be used as a stable benchmark either for like improving and iterating offline or also as part of your CI/CD. s77 00:08:29.199 --> 00:08:36.880 Those data sets are then used to run experiments on and experiments are um if you maybe look back into university and statistics or anything that you might have done. s78 00:08:36.880 --> 00:08:41.519 It's in the end this idea of varying something about your setup and looking what happens then. s79 00:08:41.519 --> 00:08:53.120 So for example, changing a prompt, changing a model, changing the context or the tool description and to see if it's then better dealing what you actually what you with what you actually wanted to deal with and ideally isolating concerns. s80 00:08:53.120 --> 00:09:02.959 So if you want to check if the same model performs better with a different prompt, you ideally only change the prompt and not five other things because otherwise it's going to be hard for you to interpret s81 00:09:02.959 --> 00:09:04.880 what you actually see. s82 00:09:04.880 --> 00:09:13.600 And then finally, you need evals or evaluation approaches to make sure you can also measure what's happening in those experiments because again you don't want to look at everything yourself. s83 00:09:13.600 --> 00:09:17.120 Uh which you by the way should should probably sometimes do. s84 00:09:17.120 --> 00:09:21.519 Um but still you also want the scalable approaches which are for example LLM as a judge. s85 00:09:21.519 --> 00:09:25.279 So using another LLM to check what the LLM did. s86 00:09:25.279 --> 00:09:34.480 um or also code based evals for things like um classification tasks where you can check for an exact match or um also checking behavior can be a a deterministic s87 00:09:34.480 --> 00:09:40.000 check or if you are checking for all caps type um things. s88 00:09:40.640 --> 00:09:49.600 So this is the AI engineering loop and what we observe with um many of our customers is that there's depending on your use case a few different entry points into this. s89 00:09:49.600 --> 00:10:05.040 So if you're for example in a low stakes environment and you're only building your internal chat agent, you might first not have a observability tool in place at all and then you maybe get started with tracing and add some monitors to just check where are people getting frustrated s90 00:10:05.040 --> 00:10:09.360 but you don't care about the eval part too much in the first place. s91 00:10:09.360 --> 00:10:16.160 You just build it out over time and just like add those test cases as you progress but you can go to production directly. s92 00:10:16.160 --> 00:10:27.760 However, if you're building a case that is customerf facing and especially if you're building a case that is in regulated environments, you probably don't want to risk the agent doing something that it's actually not supposed to do. s93 00:10:27.760 --> 00:10:34.959 So for those regulated industries, we've seen a few approaches emerging and the foundation is always tracing. s94 00:10:34.959 --> 00:10:42.560 So you always want to know what is actually happening, what steps is it taking, what inputs and outputs is it seeing and also what tools are being called. s95 00:10:42.560 --> 00:10:56.880 And then on top of this um those teams in regulated industries found a few approaches on how to go about building out these ground truth data sets that you can then also loop on improve on and also you probably all heard about auto improvement. s96 00:10:56.880 --> 00:11:03.120 Um all of this definitely needs the foundation of a ground truth data set and you yeah have a few approaches to get there. s97 00:11:03.120 --> 00:11:10.320 For example expert curation so working with your domain experts within the company to make sure you're building out the crown to data set. s98 00:11:10.320 --> 00:11:12.480 Then for example shadow runs. s99 00:11:12.480 --> 00:11:19.760 So you run the process continuously manually but in parallel you also have the agent running and you can always compare how would the agent have handled it? s100 00:11:19.760 --> 00:11:29.040 How would the agent have gone about it and then like this measure the accuracy but also build out your ground truth and then finally um like we did it also in my previous company. s101 00:11:29.040 --> 00:11:40.160 Run a fully human in the loop system and over time slice the data into parts where you are confident to automate versus things where you always want a human in the loop. s102 00:11:40.240 --> 00:11:42.079 So, so much on the introduction. s103 00:11:42.079 --> 00:11:44.320 Uh, we now want to go in a more practical example. s104 00:11:44.320 --> 00:11:54.560 And for this, I'm also welcoming Donelli again on stage and we're going to look together into how to trace your agent execution into length use, how to evaluate and improve your system, and how to continuously s105 00:11:54.560 --> 00:11:56.320 close this loop. s106 00:11:56.320 --> 00:11:56.959 Hi everybody. s107 00:11:56.959 --> 00:12:00.959 Like Annabelle said, I'm Don, but you can call me Don to keep it easier for everybody. s108 00:12:00.959 --> 00:12:04.000 Only my mom calls me Don when I'm in trouble. s109 00:12:04.000 --> 00:12:06.399 Um, yeah, that okay, that's great. s110 00:12:06.399 --> 00:12:07.120 heard a few giggles. s111 00:12:07.120 --> 00:12:09.120 So, you know, nice nice icebreaker. s112 00:12:09.120 --> 00:12:10.480 So, that's the first eval. s113 00:12:10.480 --> 00:12:12.240 It went perfectly. s114 00:12:12.240 --> 00:12:29.360 Um, so now what I'm going to show you is an example that we built for a few financials institutions that we're working with because they're in the in in kind of this process of they build their first prototype, they build their first demo, but then the question comes from leadership s115 00:12:29.360 --> 00:12:30.480 or maybe from the teams. s116 00:12:30.480 --> 00:12:35.200 How do we we evaluate this customerf facing agentic solutions that we're building? s117 00:12:35.200 --> 00:12:39.040 How many of you here are in financial services or in finance? s118 00:12:39.040 --> 00:12:40.000 Yeah, we're in New York. s119 00:12:40.000 --> 00:12:41.519 So, of course, you all are. s120 00:12:41.519 --> 00:12:47.120 Um, any other industries others than financial services, regulated industries like healthcare and stuff like that? s121 00:12:47.120 --> 00:12:47.920 Okay, cool. s122 00:12:47.920 --> 00:12:48.480 Awesome. s123 00:12:48.480 --> 00:12:51.600 So, I think this is going to this going to speak to you very much. s124 00:12:51.600 --> 00:12:58.639 Um, so basically what I have here is, you know, a typical, you know, banking assistant. s125 00:12:58.639 --> 00:13:05.440 Everybody, anybody of us that has a banking services probably have seen this in each of of your banks, right? s126 00:13:05.440 --> 00:13:19.519 However, the all the stuff that happens behind is you know very important for the bank itself or for the institution to provide you with very accurate information because there is like financial liability, there's legal liability. s127 00:13:19.519 --> 00:13:30.240 There's a bunch of you know risk for the bank itself to put out these type of solutions when they are powered by LLMs which as we all know are inherently non-deterministic. s128 00:13:30.240 --> 00:13:48.079 So we want to make sure that we reduce the unpredictability and that we can then reproduce the outcomes as much as we can and figure out where are the failure modes and where is the agent giving us the results that we don't want so that then we can you know improve upon those. s129 00:13:48.079 --> 00:13:51.519 In this example I have a few things that I I would like to show you. s130 00:13:51.519 --> 00:14:02.000 The first thing is that I can impersonate, you know, a few different type of clients because obviously a bank is always going to have customers that are like, you know, your everyday Joe like myself. s131 00:14:02.000 --> 00:14:07.760 I only have a couple of bucks in the bank, but then you're going to have your ultra high net worth individuals and all of that. s132 00:14:07.760 --> 00:14:10.959 So, we can have here impersonation of different kinds. s133 00:14:10.959 --> 00:14:21.519 And this is important because we want to be able to know if the agent is able to respect any guard rails that we have put in place when it comes to those type of uh personas that we have. s134 00:14:21.519 --> 00:14:24.959 The other thing is if you notice here maybe let me blow it up. s135 00:14:24.959 --> 00:14:37.519 We also have PII identification uh enabled and any sort of uh data that is deem as PII is automatically masked before it gets persisted into Langfuse. s136 00:14:37.519 --> 00:14:52.160 Again, this is another very big requirement for spec specifically financials institutions that they don't want to, you know, persist in an observability system their credit card information or social security numbers or their clients. s137 00:14:52.160 --> 00:14:58.160 They want to keep that on the systems of record and as much as possible in their systems uh that they own. s138 00:14:58.160 --> 00:15:03.360 So, that is also enabled and I'll show you a couple examples on how we can uh get that to work. s139 00:15:03.360 --> 00:15:15.120 And then we're also going to see if the agent itself is respecting some of the controls that we have put in place and how is being scored and if the quality gates that we have defined s140 00:15:15.120 --> 00:15:16.800 are also being enforced. s141 00:15:16.800 --> 00:15:18.959 So we're going to me measure all of that. s142 00:15:18.959 --> 00:15:28.959 I'm going to show you a few examples and then take you behind the scenes in the product itself to show you how you can set that up and then how an end toend flow looks like. s143 00:15:28.959 --> 00:15:33.440 So let's say that I am, you know, in this case I'm coming in as Anna. s144 00:15:33.440 --> 00:15:36.240 I'm asking about international wire fees. s145 00:15:36.240 --> 00:15:38.639 Very simple question, just a knowledge base. s146 00:15:38.639 --> 00:15:43.279 And then in a few seconds, it's going to come back and give me the results from our knowledge base. s147 00:15:43.279 --> 00:15:44.480 And here you go. s148 00:15:44.480 --> 00:15:52.320 You see in here, I'm surfacing information that obviously I wouldn't surface to a to a customer, but I'm doing it here for the purposes of the demo. s149 00:15:52.320 --> 00:16:02.399 But I'm calling out here the tools that are being called, the the data sources that are being called as well and some of the details as well that make up the configuration s150 00:16:02.399 --> 00:16:05.519 of this agent when it comes to giving back this answer. s151 00:16:05.519 --> 00:16:09.600 If I were to then say, okay, look, I I lost my card. s152 00:16:09.600 --> 00:16:10.480 This is my number. s153 00:16:10.480 --> 00:16:11.839 Is it blocked? s154 00:16:11.839 --> 00:16:14.639 I'm putting my credit card number in clear text, right? s155 00:16:14.639 --> 00:16:17.120 So that is obviously that's not not great. s156 00:16:17.120 --> 00:16:28.880 So you want to make sure that even though that the user you cannot control that you cannot control what your uses your users input in the system you want to make sure that you catch that whenever you are instrumenting s157 00:16:28.880 --> 00:16:40.160 and whenever you are uh building your evals so now we'll see in the trace itself that the credit card will be indeed redacted if I come here to langfuse and I open my trace s158 00:16:40.160 --> 00:16:57.199 you'll see that now the question is my car number is and then it landed in LA completely redacted That is because I have a masking function that automatically any sort of PI information that I have defined will be immediately masked and then is not persisted in plain text in my system. s159 00:16:57.199 --> 00:17:13.839 So just to show you uh in terms of what a a typical interaction looks like and I can see from the top here this was the the prompt that the user sent but then as you all know behind the scenes there's a lot of stuff going on that we don't necessarily surface to our end users but are s160 00:17:13.839 --> 00:17:22.959 critical for us as the platform teams as the developers to understand how the agent is is behaving performing and the quality of the outputs as well. s161 00:17:22.959 --> 00:17:27.120 I can see the entire tree of all the steps that he took. s162 00:17:27.120 --> 00:17:32.880 You know, here the the the anthropic LLM call that he made, the tools that are available. s163 00:17:32.880 --> 00:17:34.880 Here it called a list of tools. s164 00:17:34.880 --> 00:17:37.200 It listed the accounts and and so forth. s165 00:17:37.200 --> 00:17:38.799 And these are the outputs. s166 00:17:38.799 --> 00:17:46.799 But then more importantly here on the top, I can see all the scores that I have defined for this particular use case. s167 00:17:46.799 --> 00:17:57.600 You see that they're they're automatically calculated whenever there's a a new interaction or a new trace being captured by the system and I can I can capture things if you know down from s168 00:17:57.600 --> 00:18:07.840 if the information that I gave to the customer is compliant with banking information meaning that the customer is asking about the weather I'm not going to give that information it's not compliant s169 00:18:07.840 --> 00:18:17.520 um if there's any sort of failure modes um if there's any guard rails that we're that we're blocking any any answers that we're giving back to the customer. s170 00:18:17.520 --> 00:18:28.880 One example of that is for instance I can come someone can might come and ask can I put my savings in bitcoin right so that is something that the bank itself shouldn't be providing s171 00:18:28.880 --> 00:18:40.480 information about that because that is you know that has some ramifications and the agent you know correctly does that but at the same time it is flagging it as investment advice and s172 00:18:40.480 --> 00:18:48.559 this requires you know for the person to either talk to an advisor or or something else and we are using a score to do that. s173 00:18:48.559 --> 00:18:51.840 And if I go again to my trace, this is how I see it. s174 00:18:51.840 --> 00:18:57.840 I have the input guard rail here that tells me that this is the primary risk is investment advice. s175 00:18:57.840 --> 00:19:06.240 And in the score here, I can see that the intent of the customer on this score was indeed investment advice. s176 00:19:06.240 --> 00:19:10.880 So that way I can very easily go out and filter for those in particular. s177 00:19:10.880 --> 00:19:14.640 But anyways, I wanted to show you kind of a simple flow of this. s178 00:19:14.640 --> 00:19:24.720 The last one before getting into the end to end on how you build a data set and how you build an evaluation flow is that the fact that we can also support multimodal. s179 00:19:24.720 --> 00:19:29.919 So let's say that instead of you having the customers writing via chatbot they're calling your contact center. s180 00:19:29.919 --> 00:19:30.960 We can come here. s181 00:19:30.960 --> 00:19:41.760 We can say I want to process this call and this going to do a speech to text and then the text is going to be evaluated to to figure out if there's any violation s182 00:19:41.760 --> 00:19:54.799 on on here and I can come then in here it says that the customer was asking about blocking his card and I can indeed see the trace here and I can even see the actual link to the media s183 00:19:54.799 --> 00:20:04.559 that the customer was sent or in terms of that audio uh or that call that the customer made and also the transcript and what the assistant gave back. s184 00:20:04.559 --> 00:20:08.799 So that is in terms of the tracing how can I capture all of those activities. s185 00:20:08.799 --> 00:20:22.640 Now when I am trying to then think about okay how do I then put together or put in place a framework on how to evaluate this interactions that my customers are going to be sending through my system s186 00:20:22.640 --> 00:20:25.840 because as we all know customers are unpredictable. s187 00:20:25.840 --> 00:20:35.200 We might design a system or a solution with a a specific intention, but our customers are always going to find very clever ways to use a system in a different way. s188 00:20:35.200 --> 00:20:48.000 And we also cannot anticipate every single outcome specifically with LLM power solutions that are going to happen when we we release this into the wild for customers to use. s189 00:20:48.000 --> 00:20:58.240 So that is when we start relying on building our evaluation criteria and for for us the anchor of that is building data sets. s190 00:20:58.240 --> 00:20:59.440 So what is a data set? s191 00:20:59.440 --> 00:21:03.360 A data set is basically and let's go to this one that is the the generic one. s192 00:21:03.360 --> 00:21:14.400 A data set is basically your set of questions and expected outputs that allow you to then measure the quality of the outputs of your LLM or AI solution. s193 00:21:14.400 --> 00:21:20.960 If you come from a software development background, the way that I like to think about this is that these are your test cases. s194 00:21:20.960 --> 00:21:33.120 And I don't think anybody here that has shipped software into production will ship anything without passing into QA unless you merge into main or prod, which I'm pretty sure most of you have done it, but it's not a best practice. s195 00:21:33.120 --> 00:21:36.320 Um, I know I know I have, but whatever. s196 00:21:36.320 --> 00:21:47.440 Um, but at the same time, this is the way that you can then have the evidence that you pass the agent through a series of steps or requirements that measure s197 00:21:47.440 --> 00:21:49.120 all the things that you want to capture. s198 00:21:49.120 --> 00:21:58.000 And one thing to note here that the expected output doesn't necessarily need to be the answer that the the agent is going to give. s199 00:21:58.000 --> 00:22:01.760 It could also you want to measure maybe the intent of the agent. s200 00:22:01.760 --> 00:22:04.240 You want to measure what the agent is supposed to do. s201 00:22:04.240 --> 00:22:09.360 If you want to measure behavior, there are different ways that you can measure the quality of the output. s202 00:22:09.360 --> 00:22:20.720 And that is why you typically end up with different sets of data sets to measure a specific things or or specific areas of quality in your agent fleet. s203 00:22:20.720 --> 00:22:38.159 You see here how I have one that is like basically red teaming my agent and looking for prompt injection attacks and any sort of uh attack that will allow someone to come in and circumvent the guardrails that we have put in place to then provide information that they shouldn't get. s204 00:22:38.159 --> 00:22:51.360 The other one I have here is like very specifically for a use case that I want to know which uh which cases have been disputed and where the dis the that dispute has not been solved. s205 00:22:51.360 --> 00:23:00.880 So the point here is that I can come and and curate a specific data sets that are very specific to not only the use case but also the technical requirements s206 00:23:00.880 --> 00:23:03.039 that I have for that agent. s207 00:23:03.039 --> 00:23:09.440 This is where I would say that not unfortunately because I think I still think humans we are very important. s208 00:23:09.440 --> 00:23:17.280 So this is where you still need your domain experts in the loop working with the development teams defining what are these. s209 00:23:17.280 --> 00:23:22.000 There are two main ways that you can basically create your golden data sets. s210 00:23:22.000 --> 00:23:25.440 You either have a team and I've seen variations of both. s211 00:23:25.440 --> 00:23:28.320 You either have a team that already has a spreadsheet somewhere. s212 00:23:28.320 --> 00:23:33.200 these are all the things that we want to test on my agent and just upload it and that's pretty easy. s213 00:23:33.200 --> 00:23:45.280 You can come here and say okay perfect I want to create a test data set and then I want to upload a CSV and you can upload the CSV and it will exactly bring up everything that you just had before. s214 00:23:45.280 --> 00:23:52.720 The other thing I like to mention here everything that I'm showing you can do it as well identically through our CLI or NCP in our APIs. s215 00:23:52.720 --> 00:23:57.840 It's just that I think it's very it's better for me to show it in a UI rather than telling claude to do it end to end. s216 00:23:57.840 --> 00:23:58.960 Make no mistakes. s217 00:23:58.960 --> 00:23:59.679 Okay. s218 00:23:59.679 --> 00:24:12.400 Um so that is that on that but then I could also come here and say that I want to add um you know items into my data set for instance s219 00:24:12.400 --> 00:24:17.600 that are already as part of my production traces. s220 00:24:17.600 --> 00:24:22.720 So let's say that you saw how I was already collecting a lot of activity from my application. s221 00:24:22.720 --> 00:24:30.720 I can come here and say I want to add all of these into my data set and then select the data set that I want to select. s222 00:24:30.720 --> 00:24:40.799 So this way you can kind of dog food in a way the data set that you're going to use by looking at the production traces that your agent is producing s223 00:24:40.799 --> 00:24:49.600 as long as you have determined that yes these are good examples of what I want future outputs or traces to look like. s224 00:24:49.600 --> 00:24:58.080 Now an important thing here to do is a process that I always like to call out which is error analysis. s225 00:24:58.080 --> 00:24:58.799 Right? s226 00:24:58.799 --> 00:25:10.400 So error analysis is basically this idea that I want to look at my traces and I think Annabelle kind of talked about this uh very eloquently as well when she was on the other side as a user s227 00:25:10.400 --> 00:25:14.720 of of Langfuse that she had to like and we were talking earlier today. s228 00:25:14.720 --> 00:25:23.840 She spent like days like literally looking at traces like that was her job like looking at the trace the structure what was what were the outputs and all of that. s229 00:25:23.840 --> 00:25:40.880 And that is still a very important process because when you look at the trace you you first start understanding how the agent is behaving if it's taking all the steps that you needed to take if it's calling all the tools that it needs to call and if it's giving you the output that you want. s230 00:25:40.880 --> 00:25:43.200 That's you know very very important. s231 00:25:43.200 --> 00:25:46.000 But then how do you simplify that process? s232 00:25:46.000 --> 00:25:54.080 because obviously it is not feasible for me as a human to go out and look at thousands of traces once the system is in production. s233 00:25:54.080 --> 00:26:05.679 So what we typically do is we want to curate a sample that is representative of our production system on of traces for a human expert to come look and give me feedback. s234 00:26:05.679 --> 00:26:06.320 Right? s235 00:26:06.320 --> 00:26:10.080 It's the same as like hey I have all this activity. s236 00:26:10.080 --> 00:26:19.679 I'm just going to take 10% or 20% and I can say I just want to focus on the ones that the customer is talking about, you know, disputes or credit card information, s237 00:26:19.679 --> 00:26:22.720 whatever filter that you want, you can do that. s238 00:26:22.720 --> 00:26:27.840 And then you can build for them what we call an annotation queue, right? s239 00:26:27.840 --> 00:26:36.799 In this annotation queue, effectively what you're doing, you're telling a domain expert, in this case, let's say me, that, hey, you are the risk expert. s240 00:26:36.799 --> 00:26:44.880 I need you to take a look at this output that the agent is giving us or is giving our clients and give me your input. s241 00:26:44.880 --> 00:26:52.400 And the thing is here you want to make it very very simple for the domain expert because obviously they have their day job. s242 00:26:52.400 --> 00:26:55.279 They cannot spend just hours looking at this. s243 00:26:55.279 --> 00:27:09.279 So the the typical things that I recommend is you can give them like a couple scores that they can give you, you know, yes, no, true, false, a scale 1 to 10, whatever that is. s244 00:27:09.279 --> 00:27:12.799 And then they can also come here and give you a comment. s245 00:27:12.799 --> 00:27:20.799 So let's say here failure mode, let's just say for for for the sake of argument that in this case is missing a citation. s246 00:27:20.799 --> 00:27:24.480 And then I'm going to I'm going to put a comment. s247 00:27:24.480 --> 00:27:27.360 Citation is not there. s248 00:27:27.360 --> 00:27:34.559 Needs to be included in all these cases, right? s249 00:27:34.559 --> 00:27:38.159 You're going to go through this process and and look how you know it was simple for me. s250 00:27:38.159 --> 00:27:40.080 It took me what 10 seconds maybe. s251 00:27:40.080 --> 00:27:44.559 So let's say I have I don't know a hundred of these and it takes me 30 seconds each. s252 00:27:44.559 --> 00:27:46.480 That is what 50 minutes. s253 00:27:46.480 --> 00:27:47.600 No, no, not even. s254 00:27:47.600 --> 00:27:47.919 Right. s255 00:27:47.919 --> 00:27:49.440 I'm I'm I'm bad at math today. s256 00:27:49.440 --> 00:27:51.120 But anyways, you get the idea, right? s257 00:27:51.120 --> 00:27:55.679 So you you you can tell your domain expert, hey, give me an hour of your time. s258 00:27:55.679 --> 00:27:58.559 I'm going to curate a bunch of activity for you. s259 00:27:58.559 --> 00:28:00.720 Just give me your feedback. s260 00:28:00.720 --> 00:28:02.080 Just give me a few comments. s261 00:28:02.080 --> 00:28:05.120 And now with whisper flow or something, they can just dictate it. s262 00:28:05.120 --> 00:28:06.320 They don't even need to type it. s263 00:28:06.320 --> 00:28:06.799 Right? s264 00:28:06.799 --> 00:28:08.480 So make it very simple. s265 00:28:08.480 --> 00:28:12.480 The second thing that you're going to do with this, I'm going to mark this as completed. s266 00:28:12.480 --> 00:28:16.480 Now this human annotation lives with the trace as evidence. s267 00:28:16.480 --> 00:28:25.840 But then the second step that I want to do is take all this information, all this annotation that they gave us and then actually use an LLM to find s268 00:28:25.840 --> 00:28:32.480 all the failure modes or categories that are patterns across all of their comments. s269 00:28:32.480 --> 00:28:33.279 Right? s270 00:28:33.279 --> 00:28:38.960 So it there might be you're not asking the the the theme to invent categories. s271 00:28:38.960 --> 00:28:51.039 You're just telling them, give me your unfiltered raw commentary on these traces and then you're going to use an LLM to then figure out the categories because that's what they are great at. s272 00:28:51.039 --> 00:28:52.720 They are great at pattern matching. s273 00:28:52.720 --> 00:28:54.880 They are great at extracting information from text. s274 00:28:54.880 --> 00:29:00.399 So that way you keep it simple for the domain expert and use LLM for what they are great at. s275 00:29:00.399 --> 00:29:13.279 Now once you have all those categories that is where we can then get into our failure modes and those failure modes if I come here and look at my experiments s276 00:29:13.279 --> 00:29:20.960 that is what you're going to see whenever I run an evaluation you see all of these things that I'm measuring against. s277 00:29:20.960 --> 00:29:22.559 Maybe let me brought it up here. s278 00:29:22.559 --> 00:29:34.320 So now after I have my domain expert give me the the the inputs I extracted what are the most common categories that appear in my corpus of data based on my domain expert. s279 00:29:34.320 --> 00:29:41.360 I can then define what are my vectors or dimensions of you know quality. s280 00:29:41.360 --> 00:29:47.200 Let's just call it we just call it scores right but I like to come up with fancy words to to say things. s281 00:29:47.200 --> 00:30:00.880 But you see here how one of them was sites experts expected source correctness for formal register meaning that you know in some languages you want to address your customers with a more formal tone s282 00:30:00.880 --> 00:30:07.919 uh if the language matches what the customer came in with u must include non solicited and so forth. s283 00:30:07.919 --> 00:30:17.039 So now that I have that foundation of the categories that I'm going to measure on the quality of my outputs, then I can decide okay cool. s284 00:30:17.039 --> 00:30:20.000 So now how am I going to grade that right? s285 00:30:20.000 --> 00:30:21.600 What type of score is that? s286 00:30:21.600 --> 00:30:24.080 What type of evaluator is that? s287 00:30:24.080 --> 00:30:31.520 And for that I can either do come here and build my kind of library of evaluators, right? s288 00:30:31.520 --> 00:30:43.679 So that those as as you know traces coming in they're automatically calculating those scores for me but at the same time I can have the scores being calculated directly s289 00:30:43.679 --> 00:30:52.000 from the client application whenever something happens online in real time so that then I can surface it back to the system. s290 00:30:52.000 --> 00:31:06.799 But basically, I want to have a a way of me measuring the outputs of the agent across both LLM as a judge scores or evaluators, but also deterministic evaluators. s291 00:31:06.799 --> 00:31:18.000 Another lesson that we've that we've seen working with these clients is that the more that you can have a mix um a balance mix that is depending on the type of categories that you identify s292 00:31:18.000 --> 00:31:30.000 between LLM as a judge evaluators and deterministic evaluators the better because what that's going to do is if you rely 100% just on LLM as a judge there's a nonzero s293 00:31:30.000 --> 00:31:37.440 chance that some of the outputs are going are not going to be accurate because that's just the nature of the LLMs. s294 00:31:37.440 --> 00:31:49.919 But if you factor in some deterministic checks that are always going to be exactly the same for the same inputs irrespective of what happens, then you can have a blended s295 00:31:49.919 --> 00:31:53.360 score or a blended assessment on the quality of your agent. s296 00:31:53.360 --> 00:31:56.480 So that's what we typically would like to see. s297 00:31:56.480 --> 00:32:05.519 And you see here when I'm def trying to define a new evaluator I can come and say I want it to be a new LLM as a judge evaluator s298 00:32:05.519 --> 00:32:17.200 or I want it to be a code evaluator and I can define all the details on as to how I'm going to evaluate that particular agent activity. s299 00:32:17.200 --> 00:32:17.919 So that's that. s300 00:32:17.919 --> 00:32:21.760 And then if I want to have a code evaluator, you see this is just code. s301 00:32:21.760 --> 00:32:22.880 It's a function. s302 00:32:22.880 --> 00:32:23.919 It's code. s303 00:32:23.919 --> 00:32:24.640 This is very simple. s304 00:32:24.640 --> 00:32:25.679 It's just a reax. s305 00:32:25.679 --> 00:32:27.679 This is doing an exact match. s306 00:32:27.679 --> 00:32:32.720 But this is going to give me the same output 100% of the time. s307 00:32:32.720 --> 00:32:36.080 This is not going to be well 95% or 80%. s308 00:32:36.080 --> 00:32:39.679 It's always going to be 100% the same output for the same input. s309 00:32:39.679 --> 00:32:43.760 So again, the lesson here, you want to have a mix of both. s310 00:32:43.760 --> 00:32:45.919 So that's on the evaluation side. s311 00:32:45.919 --> 00:32:52.480 So now we've talked about like Annabelle said and maybe let me go back here to the AI engineering loop. s312 00:32:52.480 --> 00:32:55.679 So we've talked about here building the data sets. s313 00:32:55.679 --> 00:33:04.480 We've we touched on how we can then operationalize the measurement of the quality of those agents running experiments. s314 00:33:04.480 --> 00:33:07.440 And then maybe let me go back here. s315 00:33:07.440 --> 00:33:11.679 So basically an experiment is anchored on a data set. s316 00:33:11.679 --> 00:33:13.200 That's my test cases. s317 00:33:13.200 --> 00:33:16.320 Those are my tasks that I'm going to run against my agent. s318 00:33:16.320 --> 00:33:23.679 And now I'm going to say, you know what, this agent is running today with this system prompt, this model, these tools. s319 00:33:23.679 --> 00:33:26.799 What happens if I change the model? s320 00:33:26.799 --> 00:33:29.200 What happens if I make a change on the system prompt? s321 00:33:29.200 --> 00:33:34.480 What happens if I now all of a sudden I give it access to new tools to access new data? s322 00:33:34.480 --> 00:33:45.440 So you want to be able to kind of prototype that before, you know, promoting into production and just running on Vibes, which is pretty much what all we've all been doing, right? s323 00:33:45.440 --> 00:33:47.200 It's like, ah, yeah, it looks kind of good. s324 00:33:47.200 --> 00:33:49.120 Let me just push it into prod. s325 00:33:49.120 --> 00:33:52.240 But this is a a systematic way of you doing that. s326 00:33:52.240 --> 00:33:59.360 And in a few minutes, I'll show you how you can incorporate this as part of your CI/CD pipeline, so you don't have to be in the loop. s327 00:33:59.360 --> 00:34:08.079 So that anytime a change hits on your agent, then a CI job is going to kick off to run the experiment against the data set. s328 00:34:08.079 --> 00:34:17.839 It's going to evaluate your agent with a new configuration and it's going to give you a pass fail gate effectively across all of the scores and evaluators that I mentioned before. s329 00:34:17.839 --> 00:34:23.599 So you have a consistent way on how you evaluate and measure the outputs of these agents. s330 00:34:23.599 --> 00:34:24.480 Make sense? s331 00:34:24.480 --> 00:34:25.359 Okay, perfect. s332 00:34:25.359 --> 00:34:26.960 So that's experiments. s333 00:34:26.960 --> 00:34:27.520 That's great. s334 00:34:27.520 --> 00:34:28.879 So we touched on experiments. s335 00:34:28.879 --> 00:34:36.320 We touched on data sets and human annotation to bring those humans and their expertise into the loop. s336 00:34:36.320 --> 00:34:39.359 So now how do I bring it all together? s337 00:34:39.359 --> 00:34:40.399 Right? s338 00:34:40.399 --> 00:34:47.440 If I come here and I have my prompt this my system prompts that I'm versioning here in Langfuse. s339 00:34:47.440 --> 00:34:49.760 You can do it either here in Langfuse. s340 00:34:49.760 --> 00:34:51.679 You know we don't we don't have it's okay. s341 00:34:51.679 --> 00:34:53.119 We don't we don't even charge for this. s342 00:34:53.119 --> 00:34:53.839 So that's fine. s343 00:34:53.839 --> 00:34:57.119 A lot of our customers what they do is they have another version in system. s344 00:34:57.119 --> 00:35:00.720 they have a version in GitHub, GitLab, whatever that is. s345 00:35:00.720 --> 00:35:09.520 But you should look at this as your trigger for then kicking off the automation to do your eval completely automatically. s346 00:35:09.520 --> 00:35:13.839 So the way that this happens is I already have defined an automation here. s347 00:35:13.839 --> 00:35:20.079 And if you see the configuration is triggering on any prompt changes that I have here. s348 00:35:20.079 --> 00:35:30.560 And then the action is it's going to launch a CI job in my GitHub, which is this guy right here. s349 00:35:30.640 --> 00:35:35.359 Hopefully I don't I'm not get I'm not getting the unicorn from from GitHub. s350 00:35:35.359 --> 00:35:36.320 So it's this guy right here. s351 00:35:36.320 --> 00:35:42.720 So I have already my CI set it up set up waiting for any changes in the config of my agent. s352 00:35:42.720 --> 00:35:45.359 Let me bring it closer here. s353 00:35:45.359 --> 00:35:48.000 And then I can come and see my prompt. s354 00:35:48.000 --> 00:35:49.599 Let's just see this one. s355 00:35:49.599 --> 00:35:51.440 I have five versions of the prompt. s356 00:35:51.440 --> 00:35:55.040 You see how the prompt has evolved over time. s357 00:35:55.040 --> 00:36:00.400 Version one, it used to have this and then version two, a slight change. s358 00:36:00.400 --> 00:36:03.680 Then version three, it was, you know, we tried something different. s359 00:36:03.680 --> 00:36:05.040 This is for development. s360 00:36:05.040 --> 00:36:11.359 Then version four, we added more details and also passing different parameters into the configuration of the agent. s361 00:36:11.359 --> 00:36:17.839 But what I can do now is I can say, you know what, this is the production system prompt that I have right now. s362 00:36:17.839 --> 00:36:22.720 I want to test again with version 4 because I saw some regressions. s363 00:36:22.720 --> 00:36:26.079 So I can come here and do promote into production. s364 00:36:26.079 --> 00:36:30.560 And again, I'm showing you this as a human, you know, like a caveman like clicking through. s365 00:36:30.560 --> 00:36:33.599 All of this can be done via the API on the CLI. s366 00:36:33.599 --> 00:36:35.839 So you don't even have to do this by hand. s367 00:36:35.839 --> 00:36:42.960 You can just call the API and say promote this prompt version with a production uh tag and it will kick it off for you. s368 00:36:42.960 --> 00:36:44.400 I have the production tag. s369 00:36:44.400 --> 00:36:48.960 I'm going to save and promote promote to production which is the new configuration. s370 00:36:48.960 --> 00:37:00.720 In a few seconds we should have a new CI job that is going is going to launch here that then is going to kick off the experiment to run my evaluation run. s371 00:37:00.720 --> 00:37:02.240 Okay, there you go. s372 00:37:02.240 --> 00:37:03.200 Thank you GitHub. s373 00:37:03.200 --> 00:37:04.480 Thank you so much. s374 00:37:04.480 --> 00:37:05.040 So perfect. s375 00:37:05.040 --> 00:37:08.560 So now I have this job that is running automatically. s376 00:37:08.560 --> 00:37:14.880 So any time that I push any changes in the config of my agent, it's going to hit the CI. s377 00:37:14.880 --> 00:37:22.960 It's going to run the eval gates to tell me if I'm passing or failing any of the evaluators that we have defined from before. s378 00:37:22.960 --> 00:37:31.280 And then with this way, I can have the evidence and also the assurance if I can go into production, you know, safely with this new configuration. s379 00:37:31.280 --> 00:37:38.640 if it's actually the agent is degrading or improving in quality or maybe even violating any sort of guard rails that I have put in place. s380 00:37:38.640 --> 00:37:42.800 That's going to run for a few minutes, but let me show you one that I run earlier. s381 00:37:42.800 --> 00:37:47.599 And to show you the result, I can come here and see what the output is. s382 00:37:47.599 --> 00:37:56.079 And it's telling me that this was the prompt that was sent, the version, the labels, and then it's telling me that they gave passed, right? s383 00:37:56.079 --> 00:37:57.119 So, perfect. s384 00:37:57.119 --> 00:38:02.240 And these are the evaluators that I measured my agent against. s385 00:38:02.240 --> 00:38:09.520 I have here the most include, the source recall, the language match, the correctness and not unsolicited upsell. s386 00:38:09.520 --> 00:38:11.200 And I can also define minimums. s387 00:38:11.200 --> 00:38:15.760 I can say this is the minimum threshold I have for this specific score. s388 00:38:15.760 --> 00:38:22.240 If it doesn't clear that bar, it's not going to be considered as a pass uh to go into production. s389 00:38:22.240 --> 00:38:26.560 Now, obviously, in this case, I'm not crazy enough to just say, okay, cool. s390 00:38:26.560 --> 00:38:30.320 So you pass, go merge it and push into into main. s391 00:38:30.320 --> 00:38:31.920 I'm just letting it sit there. s392 00:38:31.920 --> 00:38:50.480 But admittedly, you could have a full end to end process that it will just create the PR for you and it will just wait for someone to review the PR or maybe if you are, you know, like like everybody's doing today, just pushing the PR into main straight up uh if it's clearing the bar, but at least s393 00:38:50.480 --> 00:38:52.640 you have the evidence here, right? s394 00:38:52.640 --> 00:38:59.680 So you're not pushing a PR just running on vibes or just running on what you think that the agent is good at at doing. s395 00:38:59.680 --> 00:39:09.040 You're actually have the the paper trail so to speak and the evidence on why you make that decision for this new configuration of the agent. s396 00:39:09.040 --> 00:39:09.760 Perfect. s397 00:39:09.760 --> 00:39:17.599 So now I want to show you one couple more things as an actual like use case, right? s398 00:39:17.599 --> 00:39:40.000 because we've been talking a lot about the technical aspects of of langu and more you know organizations are focused a lot on tokconomics right I think 2025 in beginning 2026 everybody was token maxing s399 00:39:40.000 --> 00:39:50.079 now everybody is like token valuing or whatever they're calling it now so everybody's like trying to extract as much value as they can of their tokens. s400 00:39:50.079 --> 00:40:00.320 So now let me show you a couple of ways that you can then show your bosses or your leaders or whoever or maybe you want to build this for your business stakeholders s401 00:40:00.320 --> 00:40:07.760 for them to feel a little bit more reassured that whatever agent solution that you're building is actually bringing benefits to the business. s402 00:40:07.760 --> 00:40:10.320 This is not just like oh the cool thing that we're building. s403 00:40:10.320 --> 00:40:22.480 Now I have here a dashboard that I'm trying to put together a few metrics that I can report on to the business based on the data that I'm being collecting from my agent. s404 00:40:22.480 --> 00:40:23.359 Right? s405 00:40:23.359 --> 00:40:32.000 The first one is and this is where you might argue with me in the logic behind this but you know every organization is different. s406 00:40:32.000 --> 00:40:44.160 But what I'm trying to show here is that I've had 1,400 interactions of this agent over the last seven days that didn't need to go to a human agent, s407 00:40:44.160 --> 00:40:44.880 right? s408 00:40:44.880 --> 00:40:46.160 So that's great. s409 00:40:46.160 --> 00:40:54.640 So that means that I'm saving, you know, human capital from spending time to talking to customers in in and do the math then. s410 00:40:54.640 --> 00:40:56.560 How much has that cost me? s411 00:40:56.560 --> 00:40:59.760 It has cost me less than $7, right? s412 00:40:59.760 --> 00:41:01.440 Over the last seven days. s413 00:41:01.440 --> 00:41:16.800 If you take any contact I used to be a contact center employee back in the day 20 years ago you know what you gonna pay that person $15$20 an hour right so that person is not going to handle 1400 calls in one hour so already you're showing s414 00:41:16.800 --> 00:41:28.640 at least some directional business benefit by having this solution in place right so that's the first thing the second thing is you could also start pinpointing what are the areas s415 00:41:28.640 --> 00:41:31.920 where your agent is tripping Right? s416 00:41:31.920 --> 00:41:43.119 If you remember, we define the categories or the failure modes as we call them in the industry where the agent is not giving me the results or the answers that I want. s417 00:41:43.119 --> 00:41:50.079 And these are the ones that I'm capturing in order of occurrence over the last seven days. s418 00:41:50.079 --> 00:41:59.920 You see here that now out of scope, informal register, and dispute not resolved are are my top three and they've been creeping up over time. s419 00:41:59.920 --> 00:42:13.839 So now, instead of me having to go through thousands or hundreds of thousands of traces trying to figure out which one is the activity from the agent that is causing the most errors or the most trouble, s420 00:42:13.839 --> 00:42:16.560 I can come here and scope it down. s421 00:42:16.560 --> 00:42:24.079 And I know already that this failure mode is my top one, the autoscope, the foremost registered and the dispute not resolved. s422 00:42:24.079 --> 00:42:32.960 So now I can come here in my tracing and do a very simple filter and say dispute not resolved. s423 00:42:32.960 --> 00:42:34.079 I can save it as a view. s424 00:42:34.079 --> 00:42:36.720 So I don't even need to every time filter it. s425 00:42:36.720 --> 00:42:39.040 You see how it's filtering up top for me. s426 00:42:39.040 --> 00:42:44.000 And even on the side I can come here and say dispute not resolved equals zero. s427 00:42:44.000 --> 00:42:54.560 So that shows all the instances where the customer had a dispute or was asking for a dispute of a product and it hasn't been resolved by the agent. s428 00:42:54.560 --> 00:43:00.880 So it either had to go to a human agent or it just lingered and the session closed and nothing happened. s429 00:43:00.880 --> 00:43:08.720 But now I can then spend my time instead of going through thousands, I can scope it down to maybe dozens or hundreds. s430 00:43:08.720 --> 00:43:12.160 And now I can do um an investigation that is more targeted. s431 00:43:12.160 --> 00:43:17.920 And now I can also find where are the areas where my agent is actually uh tripping itself up. s432 00:43:17.920 --> 00:43:19.920 So that's that's one. s433 00:43:19.920 --> 00:43:34.640 The second one is that if I want to come here in the dashboards, I want to then be able to say okay what are the number of times where my customers are giving me either thumbs up, thumbs down. s434 00:43:34.640 --> 00:43:38.000 I can I can capture that as a score very easily. s435 00:43:38.000 --> 00:43:54.960 You see here every time I can come here thumbs up thumbs down and I can just show it here in a dashboard very easy to consume any business person any user any human actually is going to be able to see this and say okay most of them are thumbs up that's cool but what about the thumbs down s436 00:43:54.960 --> 00:44:09.440 let's focus on that you know 18% of them what's been happening and then the other one is around the security mix more in here good news most of our interactions from the agent is not catching s437 00:44:09.440 --> 00:44:13.760 any sort of risk in the interaction that is having with the customers. s438 00:44:13.760 --> 00:44:26.640 But we see here that in 89 cases over the last seven days, it actually the customer was asking for investment advice and the agent might have given or might have tried to give investment advice. s439 00:44:26.640 --> 00:44:29.359 So we can then go out and focus on those. s440 00:44:29.359 --> 00:44:30.960 And again, same thing. s441 00:44:30.960 --> 00:44:36.480 I can come here and say, let me go back to my default view. s442 00:44:36.480 --> 00:44:44.560 And now I can say instead of the numeric score, I can come and say the categorical scores add a condition. s443 00:44:44.560 --> 00:44:52.240 And then say what is the intent or the security risk and then the security risk, let's say investment advice. s444 00:44:52.240 --> 00:45:06.400 And now I can see what are the specific examples where investment advice was identified as a reason for being flagged and that the that the agent shouldn't engage on that with the customer. s445 00:45:06.400 --> 00:45:11.280 And you see here even from the previous example um that I was showing you. s446 00:45:11.280 --> 00:45:21.280 So that's kind of what I wanted to show in terms of first the the nature of going end to end on a solution that you're putting customerf facing first. s447 00:45:21.280 --> 00:45:32.960 Obviously, which has a lot of critical implications in terms of liability, you know, financial implications for and reputational as well because you don't want to be putting out information that is not accurate s448 00:45:32.960 --> 00:45:48.640 or is hallucinated in front of your customers and how you can follow a systematic process so that then you can show evidence and validate that the outputs of these agents is actually following the guardrails and the controls that you have put in place. s449 00:45:48.640 --> 00:45:58.319 And with that I would like to leave you with Annabelle again so we can close and then maybe we can take a couple of questions at the end. s450 00:45:58.319 --> 00:45:59.920 Thank you. s451 00:45:59.920 --> 00:46:05.280 So what we see some companies trying to do or some teams trying to do is this. s452 00:46:05.280 --> 00:46:06.319 Please fix my agent. s453 00:46:06.319 --> 00:46:08.480 Make no mistakes. s454 00:46:08.480 --> 00:46:12.640 And there's probably a few things that we can recommend less than this. s455 00:46:12.640 --> 00:46:19.200 Even though your agent might be hooked up via MCP or CLI with your data in Lenfuse or in whichever tool you might be using. s456 00:46:19.200 --> 00:46:25.680 Um this is something that we just see producing rather slop than anything um that is actually helping you. s457 00:46:25.680 --> 00:46:32.720 But of course we're also very much in favor of working with your coding agents to continuously improve your applications. s458 00:46:32.720 --> 00:46:37.440 Um so we wanted to share this just as a picture how you can maybe memorize it. s459 00:46:37.440 --> 00:46:39.200 So we have two of those a here. s460 00:46:39.200 --> 00:46:41.200 We have one uh being the time invest. s461 00:46:41.200 --> 00:46:53.040 So, a lot of time spent or very little time spent and then also a low quality system or a high quality system and you don't want to be down there. s462 00:46:53.040 --> 00:46:59.839 If you fully automate it and just let your agent go for it and just be like fix everything, do it for me. s463 00:46:59.839 --> 00:47:01.440 Um, this is not going to turn out well. s464 00:47:01.440 --> 00:47:07.520 You will end up with low quality and yes, you don't have to do anything, but also your customers will not be super satisfied. s465 00:47:07.520 --> 00:47:11.359 At the same time, you also don't want to go go the full manual way anymore. s466 00:47:11.359 --> 00:47:13.760 You don't want to instrument your app yourself anymore. s467 00:47:13.760 --> 00:47:15.280 This is something that can be automated. s468 00:47:15.280 --> 00:47:19.599 You don't want to catch all the failure modes yourself and re review all the data yourself. s469 00:47:19.599 --> 00:47:21.119 This can be automated. s470 00:47:21.119 --> 00:47:24.640 Um but at the same time, for some things, you should get your taste in. s471 00:47:24.640 --> 00:47:34.640 You should get your domain experts in the room to build out the data sets to curate what is actually um what actually is the problem that your application should be solving and in which direction you want it to go. s472 00:47:34.640 --> 00:47:37.520 And for this um you want to land somewhere here. s473 00:47:37.520 --> 00:47:50.560 So where the ideal balance is between like um you reviewing data, bringing in your taste, bringing in your domain expertise, but also working with your coding agent for all the things that you can automate like instrumenting it, finding failure modes, s474 00:47:50.560 --> 00:48:06.400 running the evals, maybe making also 20 experiments instead of only one um that you would come up with and yeah, just like avoid getting down there in the left uh down corner, but try to be the the tasteful AI engineer that is bringing in the taste, s475 00:48:06.400 --> 00:48:12.880 making sure the the emails are great and then you can also more and more automate this loop over time. s476 00:48:12.880 --> 00:48:17.359 If we were you and we would have to get started tomorrow, h this is what we would be doing. s477 00:48:17.359 --> 00:48:21.200 We would install the lenuse skill which is a yeah it's an agent skill. s478 00:48:21.200 --> 00:48:28.800 So it's a set of markdown files that is helping your coding agent to wrap its head around how to work with lenfuse and how to implement those best practices. s479 00:48:28.800 --> 00:48:32.640 It is hooked up with our documentation with all our integration pages. s480 00:48:32.640 --> 00:48:39.920 um it is hooked up with our academy where we ingrain the best practices around data sets and how to instrument and how to write evaluators and all of this. s481 00:48:39.920 --> 00:48:51.280 So it's a a great starting point um to do the things uh automatically um that you should be doing automatically but also to maybe curate a set um of data that you should look manually at. s482 00:48:51.280 --> 00:49:02.319 uh you should then start to collect um also ground truth that you can evaluate your agent against maybe from historical data also from uh production or from some samples that you're running with your domain experts s483 00:49:02.319 --> 00:49:16.720 and if you're in a regulated space uh define your hard limits in a way to make sure these are the things that definitely shouldn't happen for financial services for example it should never give investment advice so make sure that you're ingraining this also in your evaluation set s484 00:49:16.720 --> 00:49:25.280 um that ideally never happening but at least you will know about it um when it's going to happen and we have a few resources for you. s485 00:49:25.280 --> 00:49:30.160 So the repository Donelli showed you and used um is also available online. s486 00:49:30.160 --> 00:49:31.599 Uh you can check it out yourself. s487 00:49:31.599 --> 00:49:35.119 You can take everything that you want from it and also use it for your own use cases. s488 00:49:35.119 --> 00:49:44.720 It's of course fictional data and not um not something that actually um was taken from our customers and I think also this session will be recorded so you can take a look again. s489 00:49:44.720 --> 00:49:57.040 uh we then also recommend the langu academy which is like a larger writeup of uh how to go about AI engineering um the whole steps um described again and we also have a self-s serve workshop s490 00:49:57.040 --> 00:50:08.400 that is basically guiding you through all the steps so you understood it once and then um yeah can go ahead with your own application there's some opportunities to bump into us over the next days s491 00:50:08.400 --> 00:50:20.880 uh one is um Donelli is giving a talk also on building deployment gates for AI agents on Wednesday so Check it out if you want to understand how you can make sure uh that AI models that go to production are actually um clearing quality gates. s492 00:50:20.880 --> 00:50:26.079 Uh we're also having a booth, a booth 9. Um I haven't checked out where it is, but I'm sure you will find it. s493 00:50:26.079 --> 00:50:29.359 Um you can meet the team there, but also you can get custom stickers. s494 00:50:29.359 --> 00:50:37.599 So you take a picture of your face and then it AI generates stickers of your face and length use with a maybe some jokes also included. s495 00:50:37.599 --> 00:50:40.960 Um and you can take them home and sticker it on everything you have. s496 00:50:40.960 --> 00:50:42.800 And then finally, we also have a community run. s497 00:50:42.800 --> 00:50:50.880 So if you want to step out the conference center and meet the team, we also have two of our founders here um and a few more team members and we're all big runners. s498 00:50:50.880 --> 00:50:55.280 So we thought we just might invite everyone else as well to join us um for this. s499 00:50:55.280 --> 00:50:57.280 So yeah, please come for this one as well. s500 00:50:57.280 --> 00:51:00.720 And we're super happy to use the remaining minutes for any kind of questions. s501 00:51:00.720 --> 00:51:03.280 They also have some non-custom stickers here. s502 00:51:03.280 --> 00:51:06.000 It's just a lengthy sticker, but I think it's still pretty cool. s503 00:51:06.000 --> 00:51:08.480 So if you want some, you can come up uh and get them. s504 00:51:08.480 --> 00:51:09.839 We're happy to take any questions. s505 00:51:09.839 --> 00:51:12.559 and ple please meet us and don't be a stranger. s506 00:51:12.559 --> 00:51:14.160 I'm handing this to Delli. s507 00:51:14.160 --> 00:51:14.480 Yeah. s508 00:51:14.480 --> 00:51:19.520 Well, the Langfuse per se has a hard dependency for their stores with click house. s509 00:51:19.520 --> 00:51:24.400 Um that is just because you know in the evolution of the product it started with Postgress. s510 00:51:24.400 --> 00:51:30.480 Uh but then Postgress couldn't handle the scale that we achieved and like billions of billions of events per day. s511 00:51:30.480 --> 00:51:32.480 That's when we moved over to Clickhouse. s512 00:51:32.480 --> 00:51:44.400 So in terms of that data plane yes in clickout so but maybe the followup for you what is the need for you in terms of that data stores because we also offer the ability for you to export the data you can export it to S3 s513 00:51:44.400 --> 00:51:52.160 you can use the a yeah you can use the API the CLI or the MCP if you want to consume the data from langfuse so it's totally up to you yeah s514 00:51:52.160 --> 00:51:54.079 I want to host it on s515 00:51:54.079 --> 00:52:07.040 yeah you you can host it 100% on Amazon um it's you know we have a self-hosted deployment as well um you can either use our cloud offering that is fully SAS turnkey you don't need to do anything it runs on Amazon s516 00:52:07.040 --> 00:52:18.319 but if you want to run the infra yourself because you have I don't know some you know EC2 instances lying around there you want to use we also have a deployment model for that and it comes prepackaged with all the components s517 00:52:18.319 --> 00:52:29.359 yeah super important question so the question was about interator reliability um and also probably just consistency of LLM as a judge evaluators to make sure it's actually catching um what you want it to catch s518 00:52:29.359 --> 00:52:36.160 um there's actually also part of this language skill is handling this as well and helping you to um h there's a term for it. s519 00:52:36.160 --> 00:52:46.160 Um it fell out of my head but it's in the end about the process of making sure the LLM is judging like a human would judge and uh what you do basically is you judge a sample s520 00:52:46.160 --> 00:52:59.920 um ideally a representative sample yourself and then you run your LLM as a judge and you then optimize for the LLM as a judge being coherent with what your human human raiders do and ideally you have like a few human raiders and that's academic approach to it. s521 00:52:59.920 --> 00:53:10.319 So probably the practical approach to it is have one human label some data and then optimize against this ground truth and then ideally you just check it over time to make sure your set stays representative s522 00:53:10.319 --> 00:53:19.920 and especially if you are using it for um running on production data it's in the end ideally you just start with having a a few too many false positives s523 00:53:19.920 --> 00:53:26.640 so you catch a little more than necessary but don't miss anything and then you just tune it down over time.