WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=zLc8zeGeNaY NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/zLc8zeGeNaY.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.680 --> 00:00:14.360 All right, well, hi everybody. s2 00:00:14.360 --> 00:00:15.320 I'm Cameron Wolfe. s3 00:00:15.320 --> 00:00:17.520 I'm a staff research scientist at Netflix. s4 00:00:17.520 --> 00:00:19.800 I also write the deep learning focused newsletter. s5 00:00:19.800 --> 00:00:21.040 Hopefully you read it. s6 00:00:21.040 --> 00:00:29.280 And today I'm going to be talking about evaluations and more specifically trying to evaluate highly subjective tasks. s7 00:00:29.280 --> 00:00:31.840 And we're going to go over a lot of information. s8 00:00:31.840 --> 00:00:34.280 Uh it's supposed to be kind of like an overview. s9 00:00:34.280 --> 00:00:37.880 If there's anything that you want to look into in more depth, you could scan this QR code. s10 00:00:37.880 --> 00:00:39.080 It'll take you to the slides. s11 00:00:39.080 --> 00:00:42.960 I've also written a lot of stuff that outlines all of these different concepts in detail. s12 00:00:42.960 --> 00:00:50.360 So plenty opportunity for further reading if there's anything that you're interested in throughout the talk. s13 00:00:50.760 --> 00:00:53.360 So, we all know evaluations are difficult. s14 00:00:53.360 --> 00:00:56.360 We've seen a lot of presentations on this during AIEE. s15 00:00:56.360 --> 00:01:02.880 And if we think about why this is the case, there's just there's evidence of this everywhere within the AI research community. s16 00:01:02.880 --> 00:01:05.480 People are talking about benchmarking constantly. s17 00:01:05.480 --> 00:01:09.400 There's new evals being released like every day in different revisions the evals. s18 00:01:09.400 --> 00:01:13.280 So, we all know that evaluations are difficult. s19 00:01:13.280 --> 00:01:17.400 But if we step back and think about why this is the case, there are a couple of different reasons. s20 00:01:17.400 --> 00:01:22.600 And one of the primary reasons and the one that we're going to focus on a lot today is noise. s21 00:01:22.600 --> 00:01:27.560 We're working with several non-deterministic components when we're trying to evaluate models. s22 00:01:27.560 --> 00:01:35.520 This creates a lot of fluctuations in behavior and it just makes it really difficult to draw reliable conclusions from eval results. s23 00:01:35.680 --> 00:01:39.960 And to understand why this is the case, we can just look at the structure of an evaluation trial. s24 00:01:39.960 --> 00:01:42.360 So, we have our evaluation suite here. s25 00:01:42.360 --> 00:01:45.640 It's composed of a bunch of different tasks that we want to try and solve. s26 00:01:45.640 --> 00:01:51.720 We pass these of those tasks to a non-deterministic agent that gives us a rollout and eventually a solution. s27 00:01:51.720 --> 00:01:56.800 And then we take this solution and pass it to a grader which will give us just some some kind of pass fail result. s28 00:01:56.800 --> 00:02:03.412 So, at the end of the day we're just trying to get our agents to solve these tasks and seeing if they're passing them or not. s29 00:02:03.412 --> 00:02:10.332 And because of the cost of evaluation, you know, some of these recent benchmarks will cost like, you know, 50, 100,000 dollars to run them. s30 00:02:10.332 --> 00:02:12.332 Uh we can't evaluate over that many tasks. s31 00:02:12.332 --> 00:02:16.972 So, the set of tasks within our evaluation benchmark typically is going to be relatively small. s32 00:02:16.972 --> 00:02:26.012 And this combined with the fact that we're already working with a non-deterministic agent, it already just makes it difficult to have any sort of like reliance on the eval result. s33 00:02:26.012 --> 00:02:32.812 And on top of this, if we look at most of the stuff that people are working on in terms of evals, we're primarily focusing on verifiable tasks. s34 00:02:32.812 --> 00:02:42.812 So, this grader that we see at the bottom of the screen here, it's implemented as some sort of like deterministic check, whether it be string matching, symbolic verification, test cases, and so on. s35 00:02:42.812 --> 00:02:50.132 Uh so, we're mostly focusing on these verifiable tasks where pass-fail is something that we can evaluate with a function. s36 00:02:50.132 --> 00:02:56.172 But if we look at the different things that an agent will try and do in the real world, it kind of lies on a spectrum of verifiability. s37 00:02:56.172 --> 00:02:58.612 Not all tasks are verifiable. s38 00:02:58.612 --> 00:03:06.812 So, on one end of the spectrum, we have things like math where we can verify solutions deterministically with things like string matching or symbolic verification. s39 00:03:06.812 --> 00:03:11.212 But on the other end of the spectrum, we have super subjective tasks like creative writing. s40 00:03:11.212 --> 00:03:17.732 And here, the definition of pass-fail is extremely subjective and open-ended, and it's hard even for humans to evaluate reliably. s41 00:03:17.732 --> 00:03:23.812 So, usually we're going to have to draw upon some type of like LM judge or rubric to help us with this. s42 00:03:23.812 --> 00:03:32.892 And what we end up with is that we're taking a component that was previously deterministic, this grader, and we're replacing it with another non-deterministic LM-based component. s43 00:03:32.892 --> 00:03:39.732 So, we just end up with a lot of noise and subjectivity within an evaluation process that was already difficult to begin with. s44 00:03:39.732 --> 00:03:43.172 Um so, what we end up with with this is just something that looks like this. s45 00:03:43.172 --> 00:03:45.252 We're drowning in all this noise. s46 00:03:45.252 --> 00:03:48.412 Uh we're completely blind to if our agent is like doing well or not. s47 00:03:48.412 --> 00:03:54.252 So, the goal of the talk today is to try and figure out how to not become this person. s48 00:03:54.252 --> 00:04:00.452 So, what we're going to do is we're going to outline a repeatable three-part framework for evaluating super subjective tasks. s49 00:04:00.452 --> 00:04:04.894 This will consist of offline evaluation, human evaluation, and production monitoring. s50 00:04:04.894 --> 00:04:12.214 And then we're going to outline the best practices for each of these components, which applies both to subjective tasks and just to evaluation in general. s51 00:04:12.214 --> 00:04:19.654 And then along the way we're going to cover a bunch of different tricks that we can use to reduce noise, have more statistical like confidence in these evaluation results, s52 00:04:19.654 --> 00:04:25.734 and just dry the most dry dry the most reliable conclusions that we can from our evaluation results. s53 00:04:25.734 --> 00:04:27.894 So, let's get into it. s54 00:04:27.894 --> 00:04:33.654 So, when we're working on a very subjective or open-ended task, the first step is to set up a human evaluation process. s55 00:04:33.654 --> 00:04:39.454 And this is going to be the North Star for defining, measuring, and improving the quality of our task over time. s56 00:04:39.454 --> 00:04:45.174 But, in order to set up human evaluation, we need to first create a clear definition of what quality means. s57 00:04:45.174 --> 00:04:52.054 And usually the way that we do this is by partnering with some small number of subject matter experts to create quality guidelines. s58 00:04:52.054 --> 00:04:59.454 And within these quality guidelines for our task, we define a small set of criteria that we care about in terms of what should be in the final solution. s59 00:04:59.454 --> 00:05:04.454 We define how to rate these criteria, usually using some type of like binary or Likert score. s60 00:05:04.454 --> 00:05:11.134 And then we just provide a ton of details defining what each of these criteria are, specific examples of how to rate them, and things like this. s61 00:05:11.134 --> 00:05:19.414 So, we want to provide humans as much information possible so that we can minimize the subjectivity of this human evaluation process. s62 00:05:19.414 --> 00:05:26.134 But, even if we write the best possible human evaluation guidelines, we can't just assume that the human evaluation process is going to be reliable. s63 00:05:26.134 --> 00:05:28.734 So, we need to calibrate it extensively. s64 00:05:28.734 --> 00:05:39.454 And what this means concretely is that we take multiple human evaluation samples for the same data, look at the agreement between them, and then update our guidelines based upon the sources of disagreement that we see. s65 00:05:39.454 --> 00:05:47.934 So, this is kind of an iterative process where you're literally just talking with humans and trying to see, you know, is there some issue in my guideline that makes it unclear how to evaluate this? s66 00:05:47.934 --> 00:05:55.614 And over time, we want to make our quality guidelines as clear as possible so we can get high agreement between all of these different humans. s67 00:05:55.974 --> 00:06:02.094 So, if we do this properly, then we end up with a calibrated human evaluation process and very clear quality guidelines. s68 00:06:02.094 --> 00:06:05.414 And we can already use this to start performing evaluations. s69 00:06:05.414 --> 00:06:11.975 But, if we want to draw some sort of reliable conclusion from these human evaluations, we also need to think about how we sample data. s70 00:06:11.975 --> 00:06:19.815 So, we need to think about things like the amount of data that we're going to put into the evaluation process, the type of data that we will evaluate on and so forth. s71 00:06:19.815 --> 00:06:29.775 And for the amount of data, we can draw upon a statistical concept called power analysis, which will help us understand the amount of data that we need to pass into the evaluation process. s72 00:06:29.775 --> 00:06:42.375 And power is a really interesting area of statistics, but for the purposes of this presentation, the high-level idea is that power analysis allows us to estimate the volume of data that we need to put into the evaluation process s73 00:06:42.375 --> 00:06:47.055 to make a meaningful conclusion about the difference in performance between two different models. s74 00:06:47.055 --> 00:06:53.695 And we make this estimation by looking at things like the effect size that we want to detect, the level of noise that we observe in our eval results. s75 00:06:53.695 --> 00:06:59.015 You can estimate this with like resampling model completions and so forth, and the desired significance level. s76 00:06:59.015 --> 00:07:04.775 So, how confident we want to be that a difference in two models' performance is actually legitimate and not due to noise. s77 00:07:04.775 --> 00:07:11.335 So, we can use power analysis with these factors to estimate the size of our eval that we need for a reliable conclusion. s78 00:07:11.335 --> 00:07:19.495 And then in terms of the type of data that we want to sample, there are a bunch of different factors, but at a high level, if you read a bunch of different benchmark reports and see what people do, s79 00:07:19.495 --> 00:07:25.015 generally they're going to focus upon diversity and the realism of the evaluation. s80 00:07:25.015 --> 00:07:32.055 So, we want all of the data that we evaluate on to accurately reflect the types of tasks that the agent is solving in production. s81 00:07:32.055 --> 00:07:37.535 We want it to be difficult and discriminative, meaning that it differentiates between the capabilities of different models. s82 00:07:37.535 --> 00:07:45.415 Usually, this is assessed via some form of like profiling, just sampling rollouts and seeing what types of problems a model can actually solve versus not. s83 00:07:45.415 --> 00:07:53.695 And then we want it to be as accurate as possible, which is usually accomplished via some combination of like model and human-based filtering over time. s84 00:07:53.695 --> 00:07:58.055 And these concepts for sampling data properly, they're not only applicable to human evaluation. s85 00:07:58.055 --> 00:07:59.855 You can kind of use these anywhere. s86 00:07:59.855 --> 00:08:07.455 So, we'll also see again in the future when we talk about offline evals, we'll probably use like very similar data sampling techniques. s87 00:08:07.615 --> 00:08:15.697 So, at this point, we have a reliable human evaluation process, but the problem is if we want to human evaluate every new agent or model that we're looking into, s88 00:08:15.697 --> 00:08:20.017 uh it's quickly going to become a bottleneck, and we can't iterate fast enough. s89 00:08:20.017 --> 00:08:24.137 So, we need to invest into creating offline or automatic evaluations. s90 00:08:24.137 --> 00:08:35.097 And the key idea here is that instead of verifying the solution quality with a human, we need to find some sort of automatic proxy for this, which in this case is usually going to be some LM judge or rubric. s91 00:08:35.097 --> 00:08:46.977 And we won't talk deeply about the differences between LM judges and rubrics in this talk, but at a high level, basically LM judges tend to be given high-level quality guidelines, and it's very similar to how humans evaluate tasks. s92 00:08:46.977 --> 00:08:51.657 We give them our quality guidelines, and they're going to evaluate whether the solution adheres to that. s93 00:08:51.657 --> 00:08:57.577 And it's the same set of quality guidelines that we look at across all different tasks that we're trying to evaluate. s94 00:08:57.577 --> 00:09:00.057 Rubrics, on the other hand, are a little bit more specific. s95 00:09:00.057 --> 00:09:07.257 So, they define a task-specific checklist of individual items that are each separately verified with with an LM judge. s96 00:09:07.257 --> 00:09:18.897 So, one example of this from the Dr. Tulu paper up here, I like this example cuz it's both super subjective, and it's also talking about trying to culturally adapt a book into a next Netflix film. s97 00:09:18.897 --> 00:09:25.057 But if we look at all the rubric checks that the model is looking at here, we're not looking at some high-level like quality guideline. s98 00:09:25.057 --> 00:09:30.657 We're outlining a checklist of specific things that the LM judge needs to verify within the solution that it's looking at. s99 00:09:30.657 --> 00:09:37.457 So, it's task-specific, and we're looking and verifying each of them independently to make the evaluation way more objective. s100 00:09:37.457 --> 00:09:47.217 And what we end up with here is automatic automatic metrics that we can use to verify solution for quality in a similar way to how a human would evaluate a solution. s101 00:09:47.217 --> 00:09:49.977 But the problem with this is that these are all automatic metrics. s102 00:09:49.977 --> 00:09:53.177 They're just a proxy, so they're going to be imperfect. s103 00:09:53.177 --> 00:09:58.137 So, to solve this, we need to do some sort of meta-evaluation to ensure that our judge is accurate. s104 00:09:58.137 --> 00:10:02.657 And what this means is that we'll take data from the human evaluation process, golden data. s105 00:10:02.657 --> 00:10:06.417 These are going to be evaluation examples from humans that we're confident in their quality. s106 00:10:06.417 --> 00:10:09.177 We know these are good evaluation examples. s107 00:10:09.177 --> 00:10:14.618 And then we want to measure the agreement between what our judge or rubric is telling us and this golden data set. s108 00:10:14.618 --> 00:10:21.178 So, we basically just want to look at basic things like correlation metrics, binary accuracy, and so forth. s109 00:10:21.178 --> 00:10:26.498 But meta-evaluation is like a pretty wide space and there's a bunch of different ways that you could approach this. s110 00:10:26.498 --> 00:10:37.138 One example that I like to pull on from Netflix, a project that I worked on, we actually took an LLM judge and showed that the outputs of this LLM judge were causally correlated with key streaming metrics like abandonment. s111 00:10:37.138 --> 00:10:42.978 So, if your LLM judge tells you something is bad, it's likely that somebody will abandon the show after trying to watch it. s112 00:10:42.978 --> 00:10:50.778 So, there's a bunch of different possibilities for how to evaluate judges and I'd encourage you to look into this blog if you're interested in them. s113 00:10:50.778 --> 00:10:58.058 And then finally, before moving on from from meta-evaluation, one common question is how to figure out what agreement level is good. s114 00:10:58.058 --> 00:10:59.898 And for this, it depends a lot on the task. s115 00:10:59.898 --> 00:11:04.498 Like more subjective tasks, the level of agreement that you can achieve is going to be a lot lower. s116 00:11:04.498 --> 00:11:07.498 But what we can do is just use human evaluation as a baseline. s117 00:11:07.498 --> 00:11:15.778 So, we can look at the amount of agreement between different humans who are trying to evaluate this task, and we can shoot for a similar level of agreement of our judge with the humans. s118 00:11:15.778 --> 00:11:22.338 So, similar levels of agreement both between judges and humans and mutually between humans. s119 00:11:22.778 --> 00:11:25.938 And then from there, we have automatic verifiers that we know work well. s120 00:11:25.938 --> 00:11:29.938 So, we can just start to create benchmark benchmarks around all of these verifiers. s121 00:11:29.938 --> 00:11:32.658 To do this, it's pretty similar to human evaluation. s122 00:11:32.658 --> 00:11:45.018 We just sample data properly using the same techniques we just talked about, and then we use LLM judges or rubrics to verify these outputs and then just try and continually revise and improve the data that we're evaluating on over time. s123 00:11:45.018 --> 00:11:50.738 The only point that we still need to talk about is that we now have multiple non-deterministic components that we're working with. s124 00:11:50.738 --> 00:11:55.458 In human evaluate evaluations, we only had a non-deterministic LLM. s125 00:11:55.458 --> 00:11:59.178 Now we have a non-deterministic model and a non-deterministic evaluator. s126 00:11:59.178 --> 00:12:02.898 So, we have a lot of noise that we need to figure out how to handle. s127 00:12:02.898 --> 00:12:09.418 And in order to do this, we can just apply the kitchen sink of different like statistical methods to try and put confidence intervals on all of this. s128 00:12:09.418 --> 00:12:15.859 So, similarly to human evaluation, we can use power analysis to tell us what the size of our benchmark should be. s129 00:12:15.859 --> 00:12:28.059 We can report error bars along with all of our evaluation results and then use techniques like resampling, which basically just means taking multiple rollouts from your LLM or from the judges well to reduce variance. s130 00:12:28.059 --> 00:12:31.499 And what we end up with is we don't just report evaluation results. s131 00:12:31.499 --> 00:12:40.539 We report our evaluation results with clear confidence signals and then we take interventions to try and make all of these confidence signals as tight as possible. s132 00:12:40.539 --> 00:12:46.499 And if we report those together, we can basically more meaningfully detect capability improvements in our model. s133 00:12:46.499 --> 00:12:53.739 We know if one score on an eval is better, it's because the model is actually better, not because of noise or something like this. s134 00:12:54.259 --> 00:12:57.659 So now we have a calibrated human evaluation process. s135 00:12:57.659 --> 00:13:03.179 We've created offline evaluations and calibrated judges that we can use to run this automatically. s136 00:13:03.179 --> 00:13:09.539 So we can just run this a bunch of different times and test out tons of different ideas to scale the experimentation process. s137 00:13:09.539 --> 00:13:17.459 And then from there, we can take whatever is working well and pass it back to human evaluation to have humans evaluate whether this is actually going well. s138 00:13:17.459 --> 00:13:22.139 And at this point, we might be asking ourselves whether we're ready to deploy this model or not. s139 00:13:22.139 --> 00:13:31.859 And the answer could be yes, but if we if we say yes at this point, we'd actually be skipping a super important part of the evaluation process, which is production monitoring. s140 00:13:31.859 --> 00:13:35.659 And the idea here is that we can define and track production metrics for our agent. s141 00:13:35.659 --> 00:13:42.339 So we're using how the agent is used in production as another mechanism for evaluating whether it's working properly or not. s142 00:13:42.339 --> 00:13:45.019 And to do this, you can look at a variety of different metrics. s143 00:13:45.019 --> 00:13:49.659 Some of them are explicit like thumbs up, thumbs down feedback, preference feedback and so forth. s144 00:13:49.659 --> 00:13:52.339 You'll see this done in like chatbots all the time. s145 00:13:52.339 --> 00:13:56.659 But it could also be implicit metrics like the length of a session, abandonment rate and so on. s146 00:13:56.659 --> 00:14:04.219 So you can define these different metrics that are some somehow correlated to whether the agent is performing well or not. s147 00:14:04.219 --> 00:14:09.659 And these are basically just quality signals from production that we could use to see if our agent is doing good. s148 00:14:09.659 --> 00:14:16.621 And what we could do is we can take these production metrics and measure them in an AB test to get gain a relative sense of how our agent is doing. s149 00:14:16.621 --> 00:14:21.461 So, seeing compared to what we already have in place, is the current agent doing better than that? s150 00:14:21.461 --> 00:14:27.181 And again, we can use a lot of statistical techniques to tell us how long we need to run this AB test and so forth. s151 00:14:27.181 --> 00:14:31.421 This is related to power analysis, which we talked about before. s152 00:14:31.421 --> 00:14:37.901 And the nice thing about AB testing is that what we could use what we could do is basically treat this as a soft model deployment process. s153 00:14:37.901 --> 00:14:47.381 So, instead of just sending the model out to everybody, we can vet the quality of this model in production with an AB test and make sure everything's going well before pursuing a full rollout. s154 00:14:47.381 --> 00:14:52.501 And even after we roll out the model, there's a lot of opportunities to just continually production monitor. s155 00:14:52.501 --> 00:15:02.541 We can look for any concerning trends in performance or even find cases where the agent is failing, new use cases and so forth, and use production to ground the other evaluation steps in the real world. s156 00:15:02.541 --> 00:15:09.101 So, production monitoring is incredibly useful for extending evals, finding new failure cases, all of these things. s157 00:15:09.101 --> 00:15:14.901 We want to find all of these cases and then pass them back to the other parts of the evaluation process. s158 00:15:14.901 --> 00:15:20.421 So, in this way, we just have to continually work towards improving all of the evaluation components over time. s159 00:15:20.421 --> 00:15:25.981 Uh we've seen all of the different components that we're going to cover in this talk, uh but we can always make all of them better. s160 00:15:25.981 --> 00:15:37.341 So, we could define new or better quality criteria, we can improve calibration of humans or judges, or we could look at production to find new error examples that we can work into offline evals and so forth. s161 00:15:37.341 --> 00:15:41.061 Or we could just take this entire process and repeat it in a new new domain. s162 00:15:41.061 --> 00:15:44.541 So, just build an entirely different eval. s163 00:15:45.141 --> 00:15:54.021 So, to sum everything up, evaluation is difficult, but for subjective tasks, it's especially difficult because we're dealing with more sources of noise. s164 00:15:54.021 --> 00:16:02.701 And we've seen a roadmap and all of these different components and tricks that we can use to reduce the amount of noise as much as possible to draw more confident conclusions. s165 00:16:02.701 --> 00:16:06.661 And in this roadmap, human evaluation serves as our North Star for quality. s166 00:16:06.661 --> 00:16:09.981 It provides a clear quality definition that humans agree on. s167 00:16:09.981 --> 00:16:20.022 And then from this, we could use this definition to calibrate LLM judges or rubrics as a proxy to human evaluation, and this allows us to ex-scale the experimentation process. s168 00:16:20.022 --> 00:16:24.702 Then anything that goes well, we can pass it to human evaluation and eventually deploy it. s169 00:16:24.702 --> 00:16:32.102 And when we deploy, we can use AB testing, production monitoring is another evaluation mechanism to make sure everything's going well in the real world. s170 00:16:32.102 --> 00:16:35.382 And this basically is something where we have to fight noise at every step. s171 00:16:35.382 --> 00:16:39.622 So because we're working with subjective evaluations, we know that they're going to be noisy. s172 00:16:39.622 --> 00:16:46.982 We need to be very skeptical of evaluation results and try and use statistics to give us like the best picture of how our engine is performing. s173 00:16:46.982 --> 00:16:57.422 So we can use techniques like power analysis, confidence intervals, resampling and more to reduce the variance of the evaluation process as much as possible and provide clearer confidence signals. s174 00:16:57.422 --> 00:17:02.342 And if we put all of this together, we basically just end up with a multi-step evaluation process. s175 00:17:02.342 --> 00:17:06.142 We run tons of offline evals to give us an idea of what's going to go well. s176 00:17:06.142 --> 00:17:11.222 We pass the best ones to human evaluation to give us a higher fidelity view of how something is doing. s177 00:17:11.222 --> 00:17:18.182 And then we AB test it to kind of test the waters in terms of well whether it's doing well in production and eventually deploy it. s178 00:17:18.182 --> 00:17:25.422 And once we deploy, we can continue to monitor for failure cases that we then work back into all of the other steps of the process. s179 00:17:25.702 --> 00:17:26.102 All right. s180 00:17:26.102 --> 00:17:33.662 So I know that's a lot of information, but for any areas that were confusing or you want to read more deeply into, I created this index here. s181 00:17:33.662 --> 00:17:39.422 So this links out to further reading on pretty much everything that we talked about both from myself and other people. s182 00:17:39.422 --> 00:17:43.622 So I encourage you to look into all of these details if it's interesting to you. s183 00:17:43.622 --> 00:17:46.822 Thank you so much and I hope it was helpful.