How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix

AI Engineer · 17 min · 183 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12All right, well, hi everybody.
00:14I'm Cameron Wolfe.
00:15I'm a staff research scientist at Netflix.
00:17I also write the deep learning focused newsletter.
00:19Hopefully you read it.
00:21And today I'm going to be talking about evaluations and more specifically trying to evaluate highly subjective tasks.
00:29And we're going to go over a lot of information.
00:31Uh it's supposed to be kind of like an overview.
00:34If there's anything that you want to look into in more depth, you could scan this QR code.
00:37It'll take you to the slides.
00:39I've also written a lot of stuff that outlines all of these different concepts in detail.
00:42So plenty opportunity for further reading if there's anything that you're interested in throughout the talk.
00:50So, we all know evaluations are difficult.
00:53We've seen a lot of presentations on this during AIEE.
00:56And if we think about why this is the case, there's just there's evidence of this everywhere within the AI research community.
01:02People are talking about benchmarking constantly.
01:05There's new evals being released like every day in different revisions the evals.
01:09So, we all know that evaluations are difficult.
01:13But if we step back and think about why this is the case, there are a couple of different reasons.
01:17And one of the primary reasons and the one that we're going to focus on a lot today is noise.
01:22We're working with several non-deterministic components when we're trying to evaluate models.
01:27This creates a lot of fluctuations in behavior and it just makes it really difficult to draw reliable conclusions from eval results.
01:35And to understand why this is the case, we can just look at the structure of an evaluation trial.
01:39So, we have our evaluation suite here.
01:42It's composed of a bunch of different tasks that we want to try and solve.
01:45We pass these of those tasks to a non-deterministic agent that gives us a rollout and eventually a solution.
01:51And then we take this solution and pass it to a grader which will give us just some some kind of pass fail result.
01:56So, at the end of the day we're just trying to get our agents to solve these tasks and seeing if they're passing them or not.
02:03And because of the cost of evaluation, you know, some of these recent benchmarks will cost like, you know, 50, 100,000 dollars to run them.
02:10Uh we can't evaluate over that many tasks.
02:12So, the set of tasks within our evaluation benchmark typically is going to be relatively small.
02:16And this combined with the fact that we're already working with a non-deterministic agent, it already just makes it difficult to have any sort of like reliance on the eval result.
02:26And on top of this, if we look at most of the stuff that people are working on in terms of evals, we're primarily focusing on verifiable tasks.
02:32So, this grader that we see at the bottom of the screen here, it's implemented as some sort of like deterministic check, whether it be string matching, symbolic verification, test cases, and so on.
02:42Uh so, we're mostly focusing on these verifiable tasks where pass-fail is something that we can evaluate with a function.
02:50But if we look at the different things that an agent will try and do in the real world, it kind of lies on a spectrum of verifiability.
02:56Not all tasks are verifiable.
02:58So, on one end of the spectrum, we have things like math where we can verify solutions deterministically with things like string matching or symbolic verification.
03:06But on the other end of the spectrum, we have super subjective tasks like creative writing.
03:11And here, the definition of pass-fail is extremely subjective and open-ended, and it's hard even for humans to evaluate reliably.
03:17So, usually we're going to have to draw upon some type of like LM judge or rubric to help us with this.
03:23And what we end up with is that we're taking a component that was previously deterministic, this grader, and we're replacing it with another non-deterministic LM-based component.
03:32So, we just end up with a lot of noise and subjectivity within an evaluation process that was already difficult to begin with.
03:39Um so, what we end up with with this is just something that looks like this.
03:43We're drowning in all this noise.
03:45Uh we're completely blind to if our agent is like doing well or not.
03:48So, the goal of the talk today is to try and figure out how to not become this person.
03:54So, what we're going to do is we're going to outline a repeatable three-part framework for evaluating super subjective tasks.
04:00This will consist of offline evaluation, human evaluation, and production monitoring.
04:04And then we're going to outline the best practices for each of these components, which applies both to subjective tasks and just to evaluation in general.
04:12And then along the way we're going to cover a bunch of different tricks that we can use to reduce noise, have more statistical like confidence in these evaluation results,
04:19and just dry the most dry dry the most reliable conclusions that we can from our evaluation results.
04:25So, let's get into it.
04:27So, when we're working on a very subjective or open-ended task, the first step is to set up a human evaluation process.
04:33And this is going to be the North Star for defining, measuring, and improving the quality of our task over time.
04:39But, in order to set up human evaluation, we need to first create a clear definition of what quality means.
04:45And usually the way that we do this is by partnering with some small number of subject matter experts to create quality guidelines.
04:52And within these quality guidelines for our task, we define a small set of criteria that we care about in terms of what should be in the final solution.
04:59We define how to rate these criteria, usually using some type of like binary or Likert score.
05:04And then we just provide a ton of details defining what each of these criteria are, specific examples of how to rate them, and things like this.
05:11So, we want to provide humans as much information possible so that we can minimize the subjectivity of this human evaluation process.
05:19But, even if we write the best possible human evaluation guidelines, we can't just assume that the human evaluation process is going to be reliable.
05:26So, we need to calibrate it extensively.
05:28And what this means concretely is that we take multiple human evaluation samples for the same data, look at the agreement between them, and then update our guidelines based upon the sources of disagreement that we see.
05:39So, this is kind of an iterative process where you're literally just talking with humans and trying to see, you know, is there some issue in my guideline that makes it unclear how to evaluate this?
05:47And over time, we want to make our quality guidelines as clear as possible so we can get high agreement between all of these different humans.
05:55So, if we do this properly, then we end up with a calibrated human evaluation process and very clear quality guidelines.
06:02And we can already use this to start performing evaluations.
06:05But, if we want to draw some sort of reliable conclusion from these human evaluations, we also need to think about how we sample data.
06:11So, we need to think about things like the amount of data that we're going to put into the evaluation process, the type of data that we will evaluate on and so forth.
06:19And for the amount of data, we can draw upon a statistical concept called power analysis, which will help us understand the amount of data that we need to pass into the evaluation process.
06:29And power is a really interesting area of statistics, but for the purposes of this presentation, the high-level idea is that power analysis allows us to estimate the volume of data that we need to put into the evaluation process
06:42to make a meaningful conclusion about the difference in performance between two different models.
06:47And we make this estimation by looking at things like the effect size that we want to detect, the level of noise that we observe in our eval results.
06:53You can estimate this with like resampling model completions and so forth, and the desired significance level.
06:59So, how confident we want to be that a difference in two models' performance is actually legitimate and not due to noise.
07:04So, we can use power analysis with these factors to estimate the size of our eval that we need for a reliable conclusion.
07:11And then in terms of the type of data that we want to sample, there are a bunch of different factors, but at a high level, if you read a bunch of different benchmark reports and see what people do,
07:19generally they're going to focus upon diversity and the realism of the evaluation.
07:25So, we want all of the data that we evaluate on to accurately reflect the types of tasks that the agent is solving in production.
07:32We want it to be difficult and discriminative, meaning that it differentiates between the capabilities of different models.
07:37Usually, this is assessed via some form of like profiling, just sampling rollouts and seeing what types of problems a model can actually solve versus not.
07:45And then we want it to be as accurate as possible, which is usually accomplished via some combination of like model and human-based filtering over time.
07:53And these concepts for sampling data properly, they're not only applicable to human evaluation.
07:58You can kind of use these anywhere.
07:59So, we'll also see again in the future when we talk about offline evals, we'll probably use like very similar data sampling techniques.
08:07So, at this point, we have a reliable human evaluation process, but the problem is if we want to human evaluate every new agent or model that we're looking into,
08:15uh it's quickly going to become a bottleneck, and we can't iterate fast enough.
08:20So, we need to invest into creating offline or automatic evaluations.
08:24And the key idea here is that instead of verifying the solution quality with a human, we need to find some sort of automatic proxy for this, which in this case is usually going to be some LM judge or rubric.
08:35And we won't talk deeply about the differences between LM judges and rubrics in this talk, but at a high level, basically LM judges tend to be given high-level quality guidelines, and it's very similar to how humans evaluate tasks.
08:46We give them our quality guidelines, and they're going to evaluate whether the solution adheres to that.
08:51And it's the same set of quality guidelines that we look at across all different tasks that we're trying to evaluate.
08:57Rubrics, on the other hand, are a little bit more specific.
09:00So, they define a task-specific checklist of individual items that are each separately verified with with an LM judge.
09:07So, one example of this from the Dr. Tulu paper up here, I like this example cuz it's both super subjective, and it's also talking about trying to culturally adapt a book into a next Netflix film.
09:18But if we look at all the rubric checks that the model is looking at here, we're not looking at some high-level like quality guideline.
09:25We're outlining a checklist of specific things that the LM judge needs to verify within the solution that it's looking at.
09:30So, it's task-specific, and we're looking and verifying each of them independently to make the evaluation way more objective.
09:37And what we end up with here is automatic automatic metrics that we can use to verify solution for quality in a similar way to how a human would evaluate a solution.
09:47But the problem with this is that these are all automatic metrics.
09:49They're just a proxy, so they're going to be imperfect.
09:53So, to solve this, we need to do some sort of meta-evaluation to ensure that our judge is accurate.
09:58And what this means is that we'll take data from the human evaluation process, golden data.
10:02These are going to be evaluation examples from humans that we're confident in their quality.
10:06We know these are good evaluation examples.
10:09And then we want to measure the agreement between what our judge or rubric is telling us and this golden data set.
10:14So, we basically just want to look at basic things like correlation metrics, binary accuracy, and so forth.
10:21But meta-evaluation is like a pretty wide space and there's a bunch of different ways that you could approach this.
10:26One example that I like to pull on from Netflix, a project that I worked on, we actually took an LLM judge and showed that the outputs of this LLM judge were causally correlated with key streaming metrics like abandonment.
10:37So, if your LLM judge tells you something is bad, it's likely that somebody will abandon the show after trying to watch it.
10:42So, there's a bunch of different possibilities for how to evaluate judges and I'd encourage you to look into this blog if you're interested in them.
10:50And then finally, before moving on from from meta-evaluation, one common question is how to figure out what agreement level is good.
10:58And for this, it depends a lot on the task.
10:59Like more subjective tasks, the level of agreement that you can achieve is going to be a lot lower.
11:04But what we can do is just use human evaluation as a baseline.
11:07So, we can look at the amount of agreement between different humans who are trying to evaluate this task, and we can shoot for a similar level of agreement of our judge with the humans.
11:15So, similar levels of agreement both between judges and humans and mutually between humans.
11:22And then from there, we have automatic verifiers that we know work well.
11:25So, we can just start to create benchmark benchmarks around all of these verifiers.
11:29To do this, it's pretty similar to human evaluation.
11:32We just sample data properly using the same techniques we just talked about, and then we use LLM judges or rubrics to verify these outputs and then just try and continually revise and improve the data that we're evaluating on over time.
11:45The only point that we still need to talk about is that we now have multiple non-deterministic components that we're working with.
11:50In human evaluate evaluations, we only had a non-deterministic LLM.
11:55Now we have a non-deterministic model and a non-deterministic evaluator.
11:59So, we have a lot of noise that we need to figure out how to handle.
12:02And in order to do this, we can just apply the kitchen sink of different like statistical methods to try and put confidence intervals on all of this.
12:09So, similarly to human evaluation, we can use power analysis to tell us what the size of our benchmark should be.
12:15We can report error bars along with all of our evaluation results and then use techniques like resampling, which basically just means taking multiple rollouts from your LLM or from the judges well to reduce variance.
12:28And what we end up with is we don't just report evaluation results.
12:31We report our evaluation results with clear confidence signals and then we take interventions to try and make all of these confidence signals as tight as possible.
12:40And if we report those together, we can basically more meaningfully detect capability improvements in our model.
12:46We know if one score on an eval is better, it's because the model is actually better, not because of noise or something like this.
12:54So now we have a calibrated human evaluation process.
12:57We've created offline evaluations and calibrated judges that we can use to run this automatically.
13:03So we can just run this a bunch of different times and test out tons of different ideas to scale the experimentation process.
13:09And then from there, we can take whatever is working well and pass it back to human evaluation to have humans evaluate whether this is actually going well.
13:17And at this point, we might be asking ourselves whether we're ready to deploy this model or not.
13:22And the answer could be yes, but if we if we say yes at this point, we'd actually be skipping a super important part of the evaluation process, which is production monitoring.
13:31And the idea here is that we can define and track production metrics for our agent.
13:35So we're using how the agent is used in production as another mechanism for evaluating whether it's working properly or not.
13:42And to do this, you can look at a variety of different metrics.
13:45Some of them are explicit like thumbs up, thumbs down feedback, preference feedback and so forth.
13:49You'll see this done in like chatbots all the time.
13:52But it could also be implicit metrics like the length of a session, abandonment rate and so on.
13:56So you can define these different metrics that are some somehow correlated to whether the agent is performing well or not.
14:04And these are basically just quality signals from production that we could use to see if our agent is doing good.
14:09And what we could do is we can take these production metrics and measure them in an AB test to get gain a relative sense of how our agent is doing.
14:16So, seeing compared to what we already have in place, is the current agent doing better than that?
14:21And again, we can use a lot of statistical techniques to tell us how long we need to run this AB test and so forth.
14:27This is related to power analysis, which we talked about before.
14:31And the nice thing about AB testing is that what we could use what we could do is basically treat this as a soft model deployment process.
14:37So, instead of just sending the model out to everybody, we can vet the quality of this model in production with an AB test and make sure everything's going well before pursuing a full rollout.
14:47And even after we roll out the model, there's a lot of opportunities to just continually production monitor.
14:52We can look for any concerning trends in performance or even find cases where the agent is failing, new use cases and so forth, and use production to ground the other evaluation steps in the real world.
15:02So, production monitoring is incredibly useful for extending evals, finding new failure cases, all of these things.
15:09We want to find all of these cases and then pass them back to the other parts of the evaluation process.
15:14So, in this way, we just have to continually work towards improving all of the evaluation components over time.
15:20Uh we've seen all of the different components that we're going to cover in this talk, uh but we can always make all of them better.
15:25So, we could define new or better quality criteria, we can improve calibration of humans or judges, or we could look at production to find new error examples that we can work into offline evals and so forth.
15:37Or we could just take this entire process and repeat it in a new new domain.
15:41So, just build an entirely different eval.
15:45So, to sum everything up, evaluation is difficult, but for subjective tasks, it's especially difficult because we're dealing with more sources of noise.
15:54And we've seen a roadmap and all of these different components and tricks that we can use to reduce the amount of noise as much as possible to draw more confident conclusions.
16:02And in this roadmap, human evaluation serves as our North Star for quality.
16:06It provides a clear quality definition that humans agree on.
16:09And then from this, we could use this definition to calibrate LLM judges or rubrics as a proxy to human evaluation, and this allows us to ex-scale the experimentation process.
16:20Then anything that goes well, we can pass it to human evaluation and eventually deploy it.
16:24And when we deploy, we can use AB testing, production monitoring is another evaluation mechanism to make sure everything's going well in the real world.
16:32And this basically is something where we have to fight noise at every step.
16:35So because we're working with subjective evaluations, we know that they're going to be noisy.
16:39We need to be very skeptical of evaluation results and try and use statistics to give us like the best picture of how our engine is performing.
16:46So we can use techniques like power analysis, confidence intervals, resampling and more to reduce the variance of the evaluation process as much as possible and provide clearer confidence signals.
16:57And if we put all of this together, we basically just end up with a multi-step evaluation process.
17:02We run tons of offline evals to give us an idea of what's going to go well.
17:06We pass the best ones to human evaluation to give us a higher fidelity view of how something is doing.
17:11And then we AB test it to kind of test the waters in terms of well whether it's doing well in production and eventually deploy it.
17:18And once we deploy, we can continue to monitor for failure cases that we then work back into all of the other steps of the process.
17:25All right.
17:26So I know that's a lot of information, but for any areas that were confusing or you want to read more deeply into, I created this index here.
17:33So this links out to further reading on pretty much everything that we talked about both from myself and other people.
17:39So I encourage you to look into all of these details if it's interesting to you.
17:43Thank you so much and I hope it was helpful.