WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=3z2uT5aDx_Y NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/3z2uT5aDx_Y.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:05.840 --> 00:00:07.360 Hi everyone. s2 00:00:07.360 --> 00:00:13.880 My name's Nick and I'm here with Akshay to give a talk about eval. s3 00:00:13.880 --> 00:00:25.120 We are from we're from Lyft and we've been building Lyft customer support AI agent for a year two now and gave a lot of a lot of thoughts about s4 00:00:25.120 --> 00:00:33.600 how to build eval that actually matters and scale our AI agent multi-agent AI agent system. s5 00:00:33.600 --> 00:00:35.920 Just a little bit of quick introductions. s6 00:00:35.920 --> 00:00:37.720 My name is Nick. s7 00:00:37.720 --> 00:00:39.120 I'm a data science manager. s8 00:00:39.120 --> 00:00:42.760 I've been at Lyft for 6 years, a long time Lyster. s9 00:00:42.760 --> 00:00:48.000 Uh really excited to talk to you a little bit more about eval. s10 00:00:48.000 --> 00:00:50.480 And I'll pass to Akshay. s11 00:00:50.480 --> 00:00:51.800 Hi everyone. s12 00:00:51.800 --> 00:00:52.760 I'm Akshay. s13 00:00:52.760 --> 00:01:04.720 I'm in Nick's team and we've been working together uh on customer support agents for Lyft, uh improving the hardness, improving the evals, things like that. s14 00:01:04.720 --> 00:01:07.560 And I've been at Lyft for almost 4 years now. s15 00:01:07.560 --> 00:01:14.040 I'm very excited to be here and talk about uh building evals that actually matter. s16 00:01:16.720 --> 00:01:25.120 Super excited to be here and super honored to be to be on the online track for AI engineer warfare. s17 00:01:25.120 --> 00:01:27.840 Uh yeah, then let's dive in. s18 00:01:27.840 --> 00:01:34.080 Uh for the agenda of today, uh our talk will primarily focus on eval. s19 00:01:34.080 --> 00:01:45.720 Uh we will start by sharing how we think about the end-to-end pipeline for our evaluations for building support customer support AI agent system. s20 00:01:45.720 --> 00:01:51.000 Uh we'll go into deep dive into each component more deeply as we go. s21 00:01:51.000 --> 00:02:01.560 We'll start by talking about offline evaluations, online evaluations, eval hardness, as well as what we are planning to build going forward. s22 00:02:02.600 --> 00:02:04.760 All right, let's dive in. s23 00:02:04.760 --> 00:02:15.440 I want to quickly explain the you know, high-level system of how we think about evaluation system for AI agents. s24 00:02:15.440 --> 00:02:21.360 So here you can see we have the development phase and the production phase. s25 00:02:21.360 --> 00:02:39.360 So during development if you're building agents, you should be very familiar with you know, managing contacts, building web pipeline to give your agents educational contacts, defining your tool, building your agentic graph as well as writing a system prompt. s26 00:02:39.360 --> 00:02:46.320 So once all of that engine agent engineering process is done, you have an you have an AI agents. s27 00:02:46.320 --> 00:02:58.120 The way we think about this is before we launch this AI agents to productions, we want to go through a rigorous offline evaluation process to make sure that this agent s28 00:02:58.120 --> 00:03:04.720 actually has sufficient performance before we launch this to a live users. s29 00:03:04.720 --> 00:03:18.760 So coming from you know, data science and machine learning background, we've been building model machine learning model for for for for a while and I think the way that we think about agent development is very similar to s30 00:03:18.760 --> 00:03:21.360 building machine learning models as well. s31 00:03:21.360 --> 00:03:35.480 If we are running offline evaluations for our machine learning model before that goes to productions, I think we should do the same for AI agents as well any other agentic platform agentic applications. s32 00:03:35.720 --> 00:03:47.880 But what I think you know, offline evaluation how that how that is different than traditional machine learning model is that you know, we typically we're building specifically for customer support s33 00:03:47.880 --> 00:03:51.840 AI use case, we're building an agent that's multi-turn. s34 00:03:51.840 --> 00:03:58.560 So for offline evaluation, there will be a component of simulated conversations. s35 00:03:58.560 --> 00:04:15.160 So you want you typically want to have a data set synthetic data set that's representative of your production traffic have our user and that plays plays out the complete multi-turn simulated conversation s36 00:04:15.160 --> 00:04:23.800 as far as having a grader such as a LNN as a judge to be able to evaluate how good that interaction was. s37 00:04:23.800 --> 00:04:26.840 And then we have a launch gate, right? s38 00:04:26.840 --> 00:04:38.520 We want to make sure that we have certain criteria on our offline eval and we're meeting that criteria before we decide to launch this AI agent to productions. s39 00:04:38.520 --> 00:04:47.880 And so the the real imperative here really is that it we don't want to use our live user as, you know, test data for our AI agents. s40 00:04:47.880 --> 00:04:52.440 And I think in any any cases that that is not not a good practice. s41 00:04:52.440 --> 00:04:58.040 So we really want to emphasize the importance of having an offline evaluation process. s42 00:04:58.320 --> 00:05:04.600 So once the agent hit productions, we also have our online evaluation pipeline as well. s43 00:05:04.600 --> 00:05:17.840 We have our own favorite tracing tools to trace all the executions and contacts that the AI agent used to respond to a real user in in production's environment. s44 00:05:17.840 --> 00:05:23.800 We have our online graders as well that grades how well our AI agent is doing in productions. s45 00:05:23.800 --> 00:05:38.240 And as far as having a human in the loop pipeline to do error analysis, identify failure mode, and feed back that insights to the development teams to continuously improve our AI agents. s46 00:05:39.440 --> 00:05:49.880 I want to quickly go over I think three of the most common reasons why we think evaluation typically fail in for for different teams. s47 00:05:49.880 --> 00:05:58.800 So, the first reason is that, you know, the the grader that we create, the scores that we create needs to be meaningfully gating something. s48 00:05:58.800 --> 00:06:05.720 Uh this is what we really emphasize on in the previous slide that we need to have a launch gate. s49 00:06:05.720 --> 00:06:17.400 If your LM as a judge is just floating out there, that there's a score, but no one is really using that score as a meaningful gate uh for your development and production s50 00:06:17.400 --> 00:06:22.000 environment, then that LM as a judge is not not available. s51 00:06:22.040 --> 00:06:30.880 Uh we have also seen a lot of a lot of mishaps people have when they're creating the LM as a judge. s52 00:06:30.880 --> 00:06:42.760 Uh there's a lot of different opinion out there on in terms of how do you create uh a good LM as a judge, and typically uh and unfortunately also very early on in our journey, s53 00:06:42.760 --> 00:06:48.840 the LM as a the LM judge that we created are uh very noisy, too generic. s54 00:06:48.840 --> 00:06:59.560 Uh it will output a score, and but people don't really believe in uh in what the LM judge is doing, or they don't think the LM judge insight is actionable. s55 00:06:59.960 --> 00:07:13.960 And finally, I think when something regresses in production, we need to have clear mechanism to be able to to to catch that regression, as well as identify clear owners to be able to take actions on s56 00:07:13.960 --> 00:07:18.960 uh the insights of our graders and regression gates. s57 00:07:20.520 --> 00:07:21.120 Very cool. s58 00:07:21.120 --> 00:07:26.280 I want to sequence into talking about our offline evaluation system. s59 00:07:26.280 --> 00:07:35.720 And uh as we as I touched on earlier on, I think this is the most critical piece uh going from development cycle to to production. s60 00:07:35.720 --> 00:07:45.400 We really want to have a robust offline evaluation system to be able to get more confidence in uh in the AI agents that we are shipping to productions. s61 00:07:45.400 --> 00:08:01.480 We took a lot of inspiration from this paper called TauBench, which is developed by the wonderful people at Sierra AI, and this is specifically for customer support AI agent, but we also think this is applicable for s62 00:08:01.480 --> 00:08:04.880 any user-facing agentic applications. s63 00:08:05.240 --> 00:08:19.680 So, here, as you can see, this is a offline offline simulations where you have the AI agent as well as a user LLM that are interacting with each others to produce the s64 00:08:19.680 --> 00:08:22.480 multi-turn traces. s65 00:08:22.480 --> 00:08:36.479 You have agent domain policy, which is essentially what the you know, for each customer support use case, there is instructions, policy on how to handle different different customer support issues. s66 00:08:38.000 --> 00:08:46.400 So, taking inspiration from TauBench, this is sort of a high-level approach that we have in creating our offline simulator. s67 00:08:46.400 --> 00:08:52.720 As we mentioned earlier, we have our LangGraph agents that we've built. s68 00:08:52.720 --> 00:09:00.440 And we have defined an instructions for our user LLMs, right? s69 00:09:00.440 --> 00:09:06.400 As you can see, for simulation-wise, we define the user intent. s70 00:09:06.400 --> 00:09:12.160 We define you know, what this intent is supposed to to represent. s71 00:09:12.160 --> 00:09:17.760 And we define the user data point or the world state of the of our user. s72 00:09:17.760 --> 00:09:27.560 For example, you know, the driver that are coming to us might be a luxury driver that might have been driving for us for for a couple years, and so and so forth. s73 00:09:27.560 --> 00:09:37.680 And finally, to define the user user behavior or user personas, as we would typically see with our real you know, real-life users well. s74 00:09:37.680 --> 00:09:41.440 We also created these different persona for our user. s75 00:09:41.440 --> 00:09:51.600 Uh, one example here is, uh, you know, this can be a loyal long-time Lyft customer, but they are frustrated with, uh, with our Lyft earning systems. s76 00:09:51.640 --> 00:10:04.200 So, in our offline simulator, we have this language agent that are, you know, interacting with our user LLM model and generating this multi-turn trajectories. s77 00:10:04.200 --> 00:10:09.520 Uh, of uh, of this multi-turn agent take uh, agent trajectory. s78 00:10:09.520 --> 00:10:15.880 And we also built our offline offline grader or offline evaluator. s79 00:10:15.880 --> 00:10:18.320 Uh, LLM judge is a big component of that. s80 00:10:18.320 --> 00:10:24.560 We will dive dive a little a little a lot more deeper into how to build a great LLM as a judge. s81 00:10:24.560 --> 00:10:31.240 Uh, and apart from, you know, LLM as a judge, we also have more deterministic, uh, evaluator as well. s82 00:10:31.240 --> 00:10:37.240 And these usually looks like a code assertion, as you see in traditional unit test. s83 00:10:37.240 --> 00:10:45.240 Uh, and for example, here some some of the deterministic criteria that we have created so far uh, looks something like this, right? s84 00:10:45.240 --> 00:10:57.640 If, you know, if in this in this specific interactions, the AI agent is supposed to grant concession, we'll write like a rules like this whether or not the AI agent has indeed grant concessions. s85 00:10:57.640 --> 00:11:11.760 Uh, and we'll compare the agent uh, agent two calls with the expected outcome to make sure that uh, we we can measure the accuracy of the agent instruction following behavior. s86 00:11:16.360 --> 00:11:31.920 So, one of the big gotcha that we face with running our offline evaluation is the creating synthetic data is is one of the key challenges in making sure our offline data set is representative s87 00:11:31.920 --> 00:11:35.040 of our production and data. s88 00:11:35.040 --> 00:11:40.560 So, uh here we, you know, uh here we have a meme here. s89 00:11:40.560 --> 00:11:51.600 Uh What ideally what you don't want to be doing is just to simply prompt an LM model to generate 50 different test query for your offline data sets. s90 00:11:51.600 --> 00:12:03.800 Uh so, here's a couple suggestions where you can approach this, you know, much more much more realistically and uh assemble a data set uh that closely resemble to your production data. s91 00:12:03.800 --> 00:12:09.880 So, this is also something that we would try to do for customer support AI agent as well. s92 00:12:09.880 --> 00:12:16.640 We take uh we we we we take some sample from our production data. s93 00:12:16.640 --> 00:12:22.880 Uh so, uh Lyft user has been reaching out to support for a long time. s94 00:12:22.880 --> 00:12:29.960 We take a sample of our real production example and uh supplement our offline data set with that. s95 00:12:29.960 --> 00:12:44.640 And you And the second thing that we can we do is we mutate uh different criterias for from our offline um data sets to be able to cover different golden path and edge cases. s96 00:12:48.640 --> 00:13:07.800 So, one problem with our offline evaluator is that as as we mentioned in the previous slide, if you were simply uh sampling different 50 50 test use uh test query from s97 00:13:07.800 --> 00:13:20.920 uh by using an LM model, uh another thing an another big gotcha that we we face in in building this offline simulator was that we were using a Frontier Lab LM s98 00:13:20.920 --> 00:13:25.960 uh model to role-play Lyft user in our offline evaluations. s99 00:13:25.960 --> 00:13:38.800 Um and for the most part, you know, our frontier lab model are trained to be helpful assistant rather than a uh live user that might not uh that might not sound s100 00:13:38.800 --> 00:13:41.560 as nice as uh as always. s101 00:13:41.560 --> 00:13:52.520 So, I I think uh for everyone that have tried customer support before, you know, you don't typically reach out to customer support agent uh with a very nice verbatim. s102 00:13:52.520 --> 00:14:03.240 Um, so in our first pass at running our offline evaluation, what we noticed is that our LM user sounds almost too nice. s103 00:14:03.240 --> 00:14:10.120 And as you can see on on the right here, these verbatim are very very very very complete. s104 00:14:10.120 --> 00:14:16.640 They use the LM user are very patiently explaining the issues that they're facing productions. s105 00:14:16.640 --> 00:14:25.000 And our first attempt at our offline evaluation gave us 90 plus pass rate or accuracy rate, right? s106 00:14:25.000 --> 00:14:31.240 Uh this almost sounds too good to be true, and I think it indeed is the too good to be true. s107 00:14:31.240 --> 00:14:37.080 So, in reality, these are the real user verbatim that we get in productions. s108 00:14:37.080 --> 00:14:44.839 Uh as you can see here, most user, they are they're impatient, they're already frustrated. s109 00:14:44.839 --> 00:14:52.120 So, the verbatim they they they they don't want to explain their issues like a l- LM user well. s110 00:14:52.120 --> 00:14:56.880 Uh so, typically what we see in production is is something like this. s111 00:14:56.880 --> 00:15:05.240 Uh and in in fact, in reality, the the the these makes, you know, AI agents uh much more difficult to evaluate. s112 00:15:06.000 --> 00:15:08.400 So, what can we do here? s113 00:15:08.400 --> 00:15:20.240 How how can we make sure our live user LM uh simulate this real-life live user much closely, right? s114 00:15:20.240 --> 00:15:31.560 Uh so, what we what we how we approach this is we fine-tune a LLM model with Lyft user verbatim. s115 00:15:31.560 --> 00:15:46.320 So, instead of speaking like this, very verbose and very nicely and patiently explaining their issues, uh, our LLM user will produce verbatim that resembles this much closely. s116 00:15:46.320 --> 00:15:56.480 And the benefit of this is, while, you know, we we did see our evaluation score goes down after we have fine-tuned a LLM model that speaks more like a Lyft user, s117 00:15:56.480 --> 00:16:05.680 and therefore making our evaluation more difficult, uh, but in reality, this is really what you want uh, when you're building this user simulator, right? s118 00:16:05.680 --> 00:16:15.160 If you have an eval that's too easy, that doesn't give you any real uh, production insights into how your AI agent is actually going to perform. s119 00:16:15.160 --> 00:16:23.240 Um, and this also give you a lot more room to be able to tweak your AI agents to deal with, quote-unquote, difficult user. s120 00:16:23.240 --> 00:16:32.680 And as I as I gave a little bit of sneak peek earlier as well, we also did a lot of work to define Lyft user personas. s121 00:16:32.680 --> 00:16:42.600 Uh, this can really ground our LLM user to adopt a specific Lyft user persona, and therefore, uh, simulate our real-life user much more closely. s122 00:16:42.600 --> 00:16:53.480 Uh, a couple of different user persona that we have defined here, uh, are, you know, bypasser user who just want to escalate to agent, regardless of and not giving AI a chance, s123 00:16:53.480 --> 00:16:57.200 refund seeker, uh, AI skeptics. s124 00:17:00.320 --> 00:17:15.600 So, the we we really also took inspiration from this paper, uh, from Microsoft user LLM Microsoft paper, uh, and I think they adopt a very similar similar approach as well. s125 00:17:15.600 --> 00:17:22.000 They fine-tune a uh, user LLM model until evaluation score goes down. s126 00:17:22.000 --> 00:17:30.480 Um but I think in reality, this is really what's expected and and good for for your applications. s127 00:17:31.680 --> 00:17:37.080 All right, I I'll hand it off to Akshay to talk more about LLM judge. s128 00:17:37.520 --> 00:17:38.960 All right. s129 00:17:38.960 --> 00:17:39.600 Hi everyone. s130 00:17:39.600 --> 00:17:41.600 So, I'm going to take from here. s131 00:17:41.600 --> 00:17:48.040 I'll talk about second problem which we usually face when we are doing evals with LLM as a judge. s132 00:17:48.040 --> 00:17:55.360 And here you can see uh this is how pretty much everyone is using LLM as a judge to evaluate their agents. s133 00:17:55.360 --> 00:17:58.400 The focus can be slightly different based on the use case. s134 00:17:58.400 --> 00:18:10.600 Someone can focus more on safety, someone can focus more on cost and latency, someone can focus more on quality, but people are going to measure these sort of metrics more or less. s135 00:18:10.600 --> 00:18:16.880 So, uh we want to detect leaks, we want to detect safety issues, uh and and things like that. s136 00:18:16.880 --> 00:18:24.800 So, some parts are deterministic, which can be evaluated by code, and some are not, which are evaluated by the LLM. s137 00:18:24.800 --> 00:18:34.200 Now, the problem with this approach is uh that these metrics are too generic and not actionable. s138 00:18:34.200 --> 00:18:50.400 So, for example, we also started with our eval journey using pre-built metrics from DeepEval, which was uh which were measuring tool usage appropriateness, uh response helpfulness, conversation naturalness, completeness, and things like that. s139 00:18:50.400 --> 00:18:56.160 And we did see those metrics, but the problem was these metrics were not actionable. s140 00:18:56.160 --> 00:18:58.680 They were not giving us any actionable insights. s141 00:18:58.680 --> 00:19:03.840 If something if let's say a response helpfulness is 0.5, then what do we do with it? s142 00:19:03.840 --> 00:19:18.840 So, uh things like and other other scores like toxicity score, bias, fairness, conciseness, all these are kind of relevant, but if the metrics are just scores, we we don't know what to do with them. s143 00:19:18.840 --> 00:19:34.760 So, we can use these pre-built eval metrics as a baseline, but we shouldn't use them as our core eval metrics because we want eval metrics to be actionable and tied to the business outcome or the product which we are focusing on. s144 00:19:34.760 --> 00:19:42.600 Uh Okay, so what LLM as judge should be. s145 00:19:42.640 --> 00:19:50.280 Uh we collaborate very closely with um domain experts and utilize their insights. s146 00:19:50.280 --> 00:19:54.120 So, eval should be framed around a task success or failure. s147 00:19:54.120 --> 00:20:04.080 And a binary outcome is very easy to calibrate and train um LLM judge that can consistently score your agent trajectory. s148 00:20:04.080 --> 00:20:16.360 When we partner with domain experts and data scientists to build metrics which are actionable and aligned with business goals, we can see much more meaningful results uh and actionable insights. s149 00:20:16.360 --> 00:20:29.040 Not only this is more consistent, uh but when an agent fails an interaction, we can systematically analyze the error pattern and have and actually know what what we can do to fix this. s150 00:20:29.040 --> 00:20:36.000 Here is an example of an uh actionable metric for our use case, uh which is called education rubric. s151 00:20:36.000 --> 00:20:43.280 Here the AI we define the metric how it should be, and we define success and fail criteria for LLM judge. s152 00:20:43.280 --> 00:20:52.840 So, for example, the AI agent tries too many times to educate the user um if it could have escalated the the issue. s153 00:20:52.840 --> 00:20:57.960 Or if it's escalating too soon without giving any chance to educate the user. s154 00:20:57.960 --> 00:21:08.880 So, things like that comes under failure category, so we we mark this as fail, uh but if it's an expected behavior, then we mark it as as pass. s155 00:21:09.520 --> 00:21:20.760 Uh okay, so with this, how do we validate if our LLM judge is working as expected? s156 00:21:20.880 --> 00:21:23.680 First thing is we need to treat it as a classifier. s157 00:21:23.680 --> 00:21:37.880 So, how we train our classification models in machine traditional machine learning, we can also treat our evaluation judges as as those traditional binary classifiers with binary outputs. s158 00:21:37.880 --> 00:21:53.720 Once we have binary outputs for every metric or a task based on our business goals and functional requirements, we can hand label around 100 examples with pass fail labels and then split the data into s159 00:21:53.720 --> 00:21:59.480 train, dev, and validation sets like how we used to do with machine learning models. s160 00:21:59.480 --> 00:22:11.320 Uh then we score precision and recall for our judge based on human label ground truths, which will give us an actual report on how good our judge is performing. s161 00:22:11.320 --> 00:22:22.520 Uh and how do we split our data to do to calculate precision and recall is this. s162 00:22:22.520 --> 00:22:31.640 So, we split the data similar to how we used to do in training machine learning models, but the difference is the percentage of splits. s163 00:22:31.640 --> 00:22:35.240 Uh and here we are not actually training models. s164 00:22:35.240 --> 00:22:38.600 So, we are just using the data to inform judges prompt. s165 00:22:38.600 --> 00:22:40.840 So, the percentages are little bit different. s166 00:22:40.840 --> 00:22:49.200 For example, in training data, we we pick a few short examples from that from this set for the judges prompt. s167 00:22:49.200 --> 00:22:55.960 And then we iterate the prompt against the dev set and improve our harness or our prompt. s168 00:22:55.960 --> 00:23:04.600 And then finally, we validate against the test set uh to see that we didn't overfit on the dev examples. s169 00:23:04.720 --> 00:23:20.080 This is the practical way of like splitting the data and then calculating precision and recall scores for your judge to actually know that the judge is working as expected. s170 00:23:23.120 --> 00:23:25.880 All right, next slide. s171 00:23:25.880 --> 00:23:41.040 Okay, so this is another thing which This is another thing which most of us ignore when we are doing evaluations, which is criteria drift and validating the validators. s172 00:23:41.040 --> 00:23:50.000 The key idea is that we actually discover what our evaluation criteria is by looking at the data and grading our outputs. s173 00:23:50.000 --> 00:24:00.880 Our and our sense of quality will also evolve with with new data we see and more examples we grade. s174 00:24:00.880 --> 00:24:07.080 So the evaluation should not be decoupled from model observations. s175 00:24:07.080 --> 00:24:18.840 In fact, they should be developed They should be co-developed with the model when we are testing the evaluator and calculating the precision and recall scores. s176 00:24:18.840 --> 00:24:25.720 So uh the There's always a gap when we when we talk about LLM as a judge. s177 00:24:25.720 --> 00:24:31.480 We cannot define the criteria beforehand and then evaluate agents against them. s178 00:24:31.480 --> 00:24:44.280 Our criteria should also be evolving as and when we see more examples and then we should refine our metrics for our judges and then evaluate judges on top of those metrics. s179 00:24:47.720 --> 00:24:49.720 Okay. s180 00:24:50.160 --> 00:25:03.200 So the next step One of the things which which can make the numbers we are reporting more meaningful is to add some statistical rigor to it, right? s181 00:25:03.200 --> 00:25:08.040 Uh, we report alignment rates as their point estimates. s182 00:25:08.040 --> 00:25:17.480 So, if we add confidence intervals, we do some calibration and proper sampling, uh, the same numbers can become more meaningful. s183 00:25:17.480 --> 00:25:31.400 We should definitely reserve the expensive rigor for the moments uh, number actually gets something or like like a shipping decision or we are reporting numbers to uh, company leaders. s184 00:25:31.400 --> 00:25:40.360 But, depending on the use case, we should definitely have uh, confidence intervals for the for the numbers we report. s185 00:25:40.360 --> 00:25:43.520 Because every score needs an interval. s186 00:25:45.840 --> 00:25:52.320 So, this is just a small example to give you uh, an insight on what this actually means. s187 00:25:52.320 --> 00:26:05.520 Uh, let's say we have two evaluators and one scores 84% and one and the other scores 88% and the number of samples we number [clears throat] of traces which we have s188 00:26:05.520 --> 00:26:08.240 uh, used is, let's say, 50. s189 00:26:08.240 --> 00:26:20.520 So, to show that like this is this is a very small gain and we we need much more than 50 examples to actually uh, show that this gain gain is real. s190 00:26:20.520 --> 00:26:30.920 Uh, with just 50 examples and only 4% 0.1% gain gain, we we don't actually know if this gain is real or not. s191 00:26:30.920 --> 00:26:39.760 So, bigger gains and bigger gains and paired designs need far less uh, and we can reserve the rigor. s192 00:26:39.840 --> 00:26:45.000 Statistical rigor for for things which matter the most. s193 00:26:46.160 --> 00:26:53.600 Okay, so this is uh, non-exhaustive list of observed eval anti-patterns. s194 00:26:53.600 --> 00:26:59.880 I'm not going to read all of them, but uh these definitely contain some low-hanging fruits. s195 00:26:59.880 --> 00:27:04.520 We need to put in time and effort if we if we need meaningful evals. s196 00:27:04.520 --> 00:27:08.800 So, we cannot rely on LLMs for everything yet. s197 00:27:08.800 --> 00:27:24.320 Uh and what I can say is ignoring the data is one of the most important uh things which we shouldn't do uh and which we sometimes don't focus on due to lack of time or resourcing, but it acts as the foundation for meaningful s198 00:27:24.320 --> 00:27:25.440 evaluations. s199 00:27:25.440 --> 00:27:32.520 If you don't look at the data, you won't be able to create meaningful criteria uh or labels. s200 00:27:32.520 --> 00:27:35.720 And if you don't have labels, you won't be able to evaluate your judges. s201 00:27:35.720 --> 00:27:43.120 And if you're not evaluating your judges, you don't know if your uh agentic pipeline is working as as expected. s202 00:27:43.120 --> 00:27:44.640 So, this acts as a base. s203 00:27:44.640 --> 00:27:50.600 So, uh one of the most important things we should not ignore. s204 00:27:50.840 --> 00:27:51.320 Okay. s205 00:27:51.320 --> 00:28:01.200 Uh we we said that we want to make metrics more actionable and standardize the pipeline, but how actually we should do it? s206 00:28:01.200 --> 00:28:07.200 So, this this gives you like a template to to do an error analysis loop. s207 00:28:07.200 --> 00:28:11.960 And it is important to know that this loop is something which runs continuously. s208 00:28:11.960 --> 00:28:14.720 It's not uh not a one-off audit. s209 00:28:14.720 --> 00:28:19.240 Uh so, we deep dive into raw traces. s210 00:28:19.240 --> 00:28:30.880 So, once we have logged our traces for our agentic flows, multi-agent systems, or whatever we have in our uh use case, then we pinpoint failure modes. s211 00:28:30.880 --> 00:28:35.520 So, basically we try to identify what exactly is failing. s212 00:28:35.520 --> 00:28:39.080 And we only keep the metrics that change our decision. s213 00:28:39.080 --> 00:28:40.880 We remove all the noise. s214 00:28:40.880 --> 00:28:55.200 We only prioritize on the metrics which are tied to business use cases, functional uh requirements, and actually something which changes a decision, then we form a fresh premise to reevaluate s215 00:28:55.200 --> 00:28:56.920 and then we repeat it. s216 00:28:56.920 --> 00:29:00.160 So, we can have like a regular cadence of doing this pipeline. s217 00:29:00.160 --> 00:29:10.400 It can be weekly, it can be bi-weekly, but this is something which needs to run uh continuously and it's it's not a one-off audit. s218 00:29:14.120 --> 00:29:14.440 Okay. s219 00:29:14.440 --> 00:29:15.720 So, tracing. s220 00:29:15.720 --> 00:29:23.040 Tracing, as we said, is uh one of the most important things and everything kind of depends on it. s221 00:29:23.040 --> 00:29:27.400 For diving deep into raw traces, we definitely need to log them first. s222 00:29:27.400 --> 00:29:32.920 So, we can use tools like LangSmith, LangFuse, etc. to log and view the traces. s223 00:29:32.920 --> 00:29:45.720 Uh and here each trace captures the full graph execution, which nodes ran, what LLM saw, which tools were called, uh what was the token usage, what was the latency for every call, and things like that. s224 00:29:45.720 --> 00:29:53.880 We can also add its traces with metadata if you want, which gives you more insights than uh than the actual data. s225 00:29:54.360 --> 00:29:54.680 Okay. s226 00:29:54.680 --> 00:29:56.680 And we also have annotation queues. s227 00:29:56.680 --> 00:30:10.600 Annotation queues are nothing but uh an interface which is very helpful for domain experts to label or give feedback to evaluators in in an easy-to-understand UI. s228 00:30:10.600 --> 00:30:17.160 So, they don't have to look at the raw traces, uh JSONs, and stuff stuff like that to figure out what to focus on. s229 00:30:17.160 --> 00:30:23.280 They can use this annotation queue and they can give feedback uh or label examples easily. s230 00:30:23.280 --> 00:30:34.520 We can then add these traces to datasets for offline evaluation or we can use this for uh calculating our precision and recall for our judges. s231 00:30:34.520 --> 00:30:39.840 So, this forms the basis to validate the evaluation with ground truth labels. s232 00:30:43.360 --> 00:30:43.840 Okay. s233 00:30:43.840 --> 00:30:45.040 All right. s234 00:30:45.040 --> 00:30:51.800 Now I'll hand it over to Nick to to close the evaluation loop. s235 00:30:51.800 --> 00:30:53.160 Thank you, Akshay. s236 00:30:53.160 --> 00:31:05.560 Uh and I think the the goal of having eval is to be able to feed uh our evaluation insights back into improving the model performance or our agent's performance. s237 00:31:05.560 --> 00:31:14.760 So, here uh I'll introduce a couple of ways that we can think about continual learning for our AI agent and closing the evaluation loop. s238 00:31:14.760 --> 00:31:19.960 Um and here you have model learning, context learning, and harness learning. s239 00:31:19.960 --> 00:31:27.960 A model learning is really about, you know, post-training, updating the underlying model weights, and training a custom LLM models. s240 00:31:27.960 --> 00:31:35.640 Uh context learning and harness learning is really more about improving uh everything else other than the models. s241 00:31:35.640 --> 00:31:40.840 Uh so, context is improving what information the agent is receives. s242 00:31:40.840 --> 00:31:48.000 This can be documents, this can be uh the stored memories of the user, tool outputs, and so and so forth. s243 00:31:48.000 --> 00:32:00.600 Hardest, uh that means updating the model system prompt, um tool schemas, control flow, routing, retries, and and so and so forth. s244 00:32:01.480 --> 00:32:13.040 So, I think from, you know, the error analysis that Akshay had shared earlier, I think that has really helped us to understand how can we improve uh our agent prompt s245 00:32:13.040 --> 00:32:22.200 as well as updating uh updating our knowledge space, and tune the context management strategy that we have our AI agent. s246 00:32:22.200 --> 00:32:31.080 Um the identified seller mode really help us uh be able to feed that insights into actual improvement for the agent AI agents. s247 00:32:31.640 --> 00:32:39.960 So, I want to quickly talk about what's next for us uh in our journey of building customer support AI agent at Lyft. s248 00:32:39.960 --> 00:32:53.520 Uh we have a We we we have We have some some level of ability to run our offline uh simulator, but in fact, I think, you know, it's not repeatable. s249 00:32:53.520 --> 00:33:00.800 These are currently stored as uh scattered script across different notebooks and different analysis repo. s250 00:33:00.800 --> 00:33:15.200 Uh I think one thing that we're looking really really looking into investing is a systematic eval harness and having a uh a a harness system that can help run our offline evaluation s251 00:33:15.200 --> 00:33:28.920 in in a systematic and standardized manner and allowed uh different people to contribute to uh our evaluation suite with uh very we we we predefine uh primitive and conflict-based s252 00:33:28.920 --> 00:33:30.760 uh workflow. s253 00:33:30.760 --> 00:33:36.200 Uh another thing that we've been talking thinking a lot about is uh post training. s254 00:33:36.200 --> 00:33:52.400 Uh as we mentioned earlier, you know, identifying model failure mode has really helped us tune the agent context and as far as the agent harness, uh but over the years, we have gathered a lot of real user signals on our agent performance as well. s255 00:33:52.400 --> 00:34:05.760 Uh so, we we're really starting to think about how do we fine-tune our model that does different tasks for our customer support AI agent uh as far as framing a reward modeling problem s256 00:34:05.760 --> 00:34:09.840 to to enable reinforcement learning. s257 00:34:10.879 --> 00:34:23.720 I want to quickly share a little bit about the work that we're doing with eval harness and how we think about building uh building building eval harness for uh agent-facing s258 00:34:23.720 --> 00:34:26.760 uh user-facing agent deep applications. s259 00:34:26.760 --> 00:34:42.240 Uh so, again, eval is really that scaffolding that you that we need to be able to run eval efficiently uh in a standardized format across all the different agents and sub agents that we have for our customer support s260 00:34:42.240 --> 00:34:44.000 AI agents. s261 00:34:44.000 --> 00:34:59.320 And and as you can see from my offline simulator slide earlier, our eval harness is config driven and these are typically stored as YAML file that's easily editable by different contributor and not just by engineers. s262 00:34:59.320 --> 00:35:05.560 Analysts and data scientists can can contribute to this evaluation suite as well. s263 00:35:05.560 --> 00:35:22.160 And you know, with thousands if not tens of thousands of examples in our evaluation suite, we need, you know, parallelism and throughput to be able to run our offline evaluation in a reasonable amount of times. s264 00:35:22.160 --> 00:35:33.560 And to enable user to be different user to be able to to be able to contribute to our eval evaluation suite, we also define, you know, primitive around our eval s265 00:35:33.560 --> 00:35:35.400 eval harness. s266 00:35:35.400 --> 00:35:44.320 These are high-level things like task, data sets, personas, LM adapter, and and evaluator. s267 00:35:44.480 --> 00:35:56.640 And after all, you know, the benefit of having an eval harness is that you can define the config once and run these eval indefinitely many times across different at different touch points s268 00:35:56.640 --> 00:35:59.840 or different gates of your agent development process. s269 00:35:59.840 --> 00:36:04.080 This can be, you know, locally when you're developing this agent. s270 00:36:04.080 --> 00:36:17.560 Uh at any point when you tune a prompt, you can run the evaluation suite and get an immediate in immediate feedback on how your agent is doing compared to the previous versions. s271 00:36:17.560 --> 00:36:28.720 We can run these at pre-commit hook to make sure that our performance doesn't degrade before we push a change uh uh agent service. s272 00:36:28.720 --> 00:36:42.480 Uh another area that we are looking at is also a CI/CD and how we can use our eval harness to uh build our regression test suite, uh acceptance acceptance test suite as well. s273 00:36:43.000 --> 00:36:46.840 So, that wraps up our presentation today. s274 00:36:46.840 --> 00:36:56.960 Uh we've gone through a lot of We've gone through a lot of uh a lot of different topics uh for evaluations. s275 00:36:56.960 --> 00:37:09.880 Uh really, I think this is a an end-to-end journey uh for building a evaluation pipeline that works for customer support AI agent or any user-facing agent tech applications. s276 00:37:09.880 --> 00:37:18.480 Uh Akshat and I, we are very interested to hear about, you know, what you all have been working on and share any learnings that you have for eval. s277 00:37:18.480 --> 00:37:28.960 So, feel free to contact us uh if you have any questions or you just want to share some uh brainstorm about how to improve your evaluations. s278 00:37:30.120 --> 00:37:35.760 All right, thank you and I hope you all enjoy the AI Engineer World Fair. s279 00:37:36.760 --> 00:37:38.080 Thank you. s280 00:37:38.080 --> 00:37:40.360 Thanks, all.