WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=xyL2Ltkh-SA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/xyL2Ltkh-SA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.719 --> 00:00:14.240
Hi everyone.

s3
00:00:14.240 --> 00:00:16.960
Uh sounds like everybody came back from lunch.

s4
00:00:16.960 --> 00:00:20.160
So hope everybody is recharged and not sleepy at all.

s5
00:00:20.160 --> 00:00:23.199
It's always interesting to do a talk right after lunch because you never know.

s6
00:00:23.199 --> 00:00:24.400
It's a mixed crowd.

s7
00:00:24.400 --> 00:00:26.000
Uh but we're very happy to be here.

s8
00:00:26.000 --> 00:00:27.119
Happy to see you all.

s9
00:00:27.119 --> 00:00:29.039
Our talk is going to be about eval.

s10
00:00:29.039 --> 00:00:30.560
Of course, we're in the eval track.

s11
00:00:30.560 --> 00:00:36.800
We're going to talk you through what are some things that worked for us while we were building eval uh especially for YouTube ads.

s12
00:00:36.800 --> 00:00:42.640
We work on the YouTube ads team as part of the we do image and video models for YouTube ads.

s13
00:00:42.640 --> 00:00:44.879
So, building an agent is hard.

s14
00:00:44.879 --> 00:00:50.399
I think anybody who's here in the audience probably has built an agent as a side project or as part of production systems.

s15
00:00:50.399 --> 00:00:52.239
It's a very hard thing to do.

s16
00:00:52.239 --> 00:00:52.960
It's laborious.

s17
00:00:52.960 --> 00:00:54.559
It takes a lot of time.

s18
00:00:54.559 --> 00:00:58.000
Uh making it reliable is harder.

s19
00:00:58.000 --> 00:01:07.840
So having it do things that you actually want it to do in production, understanding the different kind of things that it can play with, how it's going to react when you launch it to your end users,

s20
00:01:07.840 --> 00:01:14.320
that's always a very hard thing to do, which is why eval are pretty handy way to manage that.

s21
00:01:14.799 --> 00:01:23.119
Uh yeah, and then um basically uh the first step when you're doing this is of course you need to have your agent foundation.

s22
00:01:23.119 --> 00:01:34.640
So um when you're building uh your agent, you know, you will want to have a focused and strong set of LLM friendly tools to give your agent a very good foundation.

s23
00:01:34.640 --> 00:01:46.000
Um so yeah, I would say it's important to first optimize these tools and make sure they're the best they can be before just jumping onto um larger agent evals.

s24
00:01:46.000 --> 00:02:01.280
Um so once your tools are optimized uh you can also take some other steps like making an independent critique agent right with a remediation loop and this can fill more gaps as far as um having a self-correction

s25
00:02:01.280 --> 00:02:08.719
mechanism and filling those gaps where uh maybe your base tool set has limitations.

s26
00:02:08.959 --> 00:02:27.040
Uh and then once your base structure is defined uh you can have you can then go to um having an eval and having a strong eval is very important as this gives you um like a way of proving the value of changes you make as well as running ablation experiments on any changes you make.

s27
00:02:27.040 --> 00:02:30.959
So I would say this is a very essential tool for climbing the quality ladder.

s28
00:02:30.959 --> 00:02:37.120
Uh but again it's very important to have that uh good foundation to begin with.

s29
00:02:38.480 --> 00:02:51.920
Um so yeah the the reliability of your agent is basically a function of the capabilities of the agent uh the guard rails and the evals.

s30
00:02:51.920 --> 00:02:57.200
Um so understanding uh what your agents do in the real world.

s31
00:02:57.200 --> 00:03:17.680
uh basically uh generative AI outputs as I'm sure you're all familiar are not exactly deterministic right so it can often fail in certain areas or uh one time it can succeed one time it can fail so we can't really guarantee how it will behave in the wild and for some use cases this is extremely important right

s32
00:03:17.680 --> 00:03:28.879
and we need a way to uh measure at scale and make sure that it is uh getting the output we want uh despite the non-determinism of these models So

s33
00:03:28.879 --> 00:03:43.040
we need to define um what's good here and eval behave in the real world by defining what good looks like.

s34
00:03:43.040 --> 00:03:49.360
Uh it's basically just setting this is uh you know our target output.

s35
00:03:49.360 --> 00:03:57.040
Uh so to build evals that actually scale they really need to be strict and measurable.

s36
00:03:58.720 --> 00:04:08.000
And uh so an interesting uh thing here that I think might be somewhat counterintuitive is that early on vibing can actually be kind of good for you.

s37
00:04:08.000 --> 00:04:15.200
Uh and what I mean here by vibing is basically um doing things that are not exactly scalable to begin with.

s38
00:04:15.200 --> 00:04:27.120
Um so when you're first uh starting out it may be that uh you you know you could uh take a track of basically just going ahead and making the super comprehensive eval

s39
00:04:27.120 --> 00:04:41.759
right um but we found it actually works better to first do intuition based approach where you kind of um first see the capabilities and look at the outputs and at this stage it's pretty easy to tell what the issues actually are

s40
00:04:41.759 --> 00:04:54.560
right uh so even though this is non-scalable it will still give you like a very good idea of when you change this what happens um and like uh it allows you to more quickly iterate as well.

s41
00:04:54.560 --> 00:05:04.320
So at this stage prompt tweaks can also have like large performance gains you can make a radical change to the architecture um and your eval is not kind of like hindering you in this way.

s42
00:05:04.320 --> 00:05:13.520
So it's like a very good way kind of you know like an early stage company of just like first you know doing something making more radical changes quickly.

s43
00:05:13.520 --> 00:05:30.639
Um so yeah this way I think you can also get very familiar with what you're building what the failure patterns are and uh it gives you more of a a sense of depth and understanding it uh which allows you to hill climb in a targeted way and these will basically be very useful learnings

s44
00:05:30.639 --> 00:05:34.639
when you're actually like building the more comprehensive EVO.

s45
00:05:34.639 --> 00:05:50.400
Uh yeah, and then there's a chart here showing kind of uh you know if if you uh jump to scale to um these scaled raiders like too early uh it can cause you to kind of have like very big ups and downs as you might be iterating and calibrating the eval

s46
00:05:50.400 --> 00:05:55.360
as you are struggling with uh changing the model radically.

s47
00:05:57.360 --> 00:06:03.680
Um so another thing here is um you can you should start early and start small.

s48
00:06:03.680 --> 00:06:10.479
So um you don't as I said before you don't need to have like a massive golden set on day one.

s49
00:06:10.479 --> 00:06:13.759
Um you can just kind of start with a few core tasks.

s50
00:06:13.759 --> 00:06:19.440
So you can look through your agent and define what are the primary things that you want to target, right?

s51
00:06:19.440 --> 00:06:28.720
Um and just uh basically start with those like highle things and can slowly come to a more detailed representation as you move on.

s52
00:06:28.720 --> 00:06:32.479
Um, and so here it's important to also test the negatives.

s53
00:06:32.479 --> 00:06:41.680
Checking if the model like didn't do something as bad, uh, something bad is just as critical as checking if it did the task.

s54
00:06:42.000 --> 00:06:57.759
Um, so yeah, and uh, yeah, there's a funny visual here about writing the evals can be a very small point and humans arguing over what the rubric should be is, uh, is kind of like a very large task here.

s55
00:07:00.240 --> 00:07:00.960
Uh cool.

s56
00:07:00.960 --> 00:07:04.960
So we've established that we want to start small and we want to wipe code at the start.

s57
00:07:04.960 --> 00:07:10.880
Not wipe code but like why eval at the start and try to get a sense of like what our models doing good where the agents are falling.

s58
00:07:10.880 --> 00:07:12.400
Find out all the patterns.

s59
00:07:12.400 --> 00:07:15.520
Eventually you'll get at a stage where you will try to involve more teams.

s60
00:07:15.520 --> 00:07:20.240
So if it's just a core team of like PM and GX working at the start then you'll bring in more team.

s61
00:07:20.240 --> 00:07:23.680
You'll have a bigger golden set a bigger data set that you want to test out.

s62
00:07:23.680 --> 00:07:27.520
So you'll think about scale raiders, LLM raiders, all of that.

s63
00:07:27.520 --> 00:07:30.639
So we'll get a little bit more into what that looks like.

s64
00:07:30.639 --> 00:07:35.599
So just a couple of things on like working with scale raiders and things that worked for us.

s65
00:07:35.599 --> 00:07:43.919
Uh one was that providing them with a clear rubric of what they were actually rating with very clear examples.

s66
00:07:43.919 --> 00:07:48.639
So we had a lot of situations, especially early on when you're building things.

s67
00:07:48.639 --> 00:07:53.840
of course like there are so many edge cases and difficult cases that we've not tested out that a raider might encounter.

s68
00:07:53.840 --> 00:08:03.759
So they're coming back to you saying oh what what what should I do in this case and then sometimes we as a team are like disagreeing on like should this be a pass should be should this be a fail things like that.

s69
00:08:03.759 --> 00:08:10.879
So I think that's very important to do early on as much as clarity and examples you can give the raers that would be super helpful.

s70
00:08:10.879 --> 00:08:18.800
So yeah to that point like human human agreement should be strong within your team of what you consider a good use case and a good past case for an eval.

s71
00:08:18.800 --> 00:08:24.240
Uh the second things that we noticed that helped us a lot was getting explanations from raider.

s72
00:08:24.240 --> 00:08:37.440
So when you do have your teams or other scale members rate eval if it's a pass or a fail, that doesn't really tell you much about where should the agent improve, what was the thinking that went behind

s73
00:08:37.440 --> 00:08:38.719
coming to that conclusion.

s74
00:08:38.719 --> 00:08:44.480
So it's helpful to get explanations of why they're rating something a certain way.

s75
00:08:44.480 --> 00:08:55.040
And this is true for like if you do single side evals or sideby-side eval like when you're testing two models at the same time having explanations of why one thing failed or one thing worked can be super helpful.

s76
00:08:55.040 --> 00:09:00.880
Uh other things to keep in mind is that you could also do like in our case it was multi output.

s77
00:09:00.880 --> 00:09:07.760
So we were asking scale raiders um when we were building ads like are these ads accurate like did we do the right things for it?

s78
00:09:07.760 --> 00:09:08.959
Is it brand safe?

s79
00:09:08.959 --> 00:09:10.959
Is it like something that we expected it to be?

s80
00:09:10.959 --> 00:09:11.839
Things of that nature.

s81
00:09:11.839 --> 00:09:15.279
So we had like almost like a multi-turn eval system.

s82
00:09:15.279 --> 00:09:20.880
If you're building those kind of cases, it can get a little tricky because it's not exactly a pass failure.

s83
00:09:20.880 --> 00:09:28.880
Your raiders could be like, "Oh, well, it does very well in well in brand safety, but it does not do really good in like accuracy or things of that nature."

s84
00:09:28.880 --> 00:09:34.480
So explanations really help you like get to the bottom of like where is it that the agent's actually like missing things.

s85
00:09:34.480 --> 00:09:38.720
And then you can also use that input to train your agent better.

s86
00:09:38.720 --> 00:09:47.760
Okay, so now we talked a little bit more about involving cross functional teams and human agents, but what about if you're using LLM raiders, auto raiders, LMS judges.

s87
00:09:47.760 --> 00:09:50.480
So we ended up going down that path.

s88
00:09:50.480 --> 00:10:02.959
Also few things that we tried to do to kind of set this set a more uh comprehensive structure for ourselves is one we we monitored disagreements or in some cases teams monitored agreements.

s89
00:10:02.959 --> 00:10:15.120
So basically if you can have a sample pipeline of sorts that is monitoring how a human raider or some expert would rate an eval versus how an LLM would rate it.

s90
00:10:15.120 --> 00:10:21.760
You can get a sense of like how it's trending and if the agreement rates are in the ballpark that you would expect it to be.

s91
00:10:21.760 --> 00:10:25.839
Uh the second thing was we went a little bit beyond pass and fail.

s92
00:10:25.839 --> 00:10:29.279
So we also looked at agent traces which we'll get to later.

s93
00:10:29.279 --> 00:10:38.800
But when we were doing pass fail eval and trying to understand how things were getting rated, we did a couple of spot checks to understand the reasoning behind those logic

s94
00:10:38.800 --> 00:10:41.120
so we could really see what was going on.

s95
00:10:41.120 --> 00:10:45.440
How did it come to the conclusion that something was a final pass or no?

s96
00:10:45.440 --> 00:10:50.720
Uh and again like just reinforcing the high quality ground truth point that Daniel made earlier.

s97
00:10:50.720 --> 00:10:55.200
So we want to give a golden set that's like super expansive.

s98
00:10:55.200 --> 00:11:02.000
It covers a broad range of use cases and it also has very high human human agreement within your team.

s99
00:11:02.000 --> 00:11:02.399
Okay.

s100
00:11:02.399 --> 00:11:06.640
So this is a quick example of what we had seen in the agent.

s101
00:11:06.640 --> 00:11:08.000
I'll walk you folks through it.

s102
00:11:08.000 --> 00:11:14.000
So uh it basically says if you if you want to know what it's doing look at it at its thinking.

s103
00:11:14.000 --> 00:11:20.560
Uh we had given the agent one of the prompts and it was that for legal reasons disclaimers can never be removed.

s104
00:11:20.560 --> 00:11:23.519
And we had mentioned that to the agent a couple of times in the prompt.

s105
00:11:23.519 --> 00:11:26.320
We had like trained it on that and it was all going fine.

s106
00:11:26.320 --> 00:11:36.079
But then we started seeing that there were edge cases in which the agent was seeing the prompt and it was seeing that there's a disclaimer present in the ad and then still removing it.

s107
00:11:36.079 --> 00:11:41.920
And we could not find that if we were just doing a categorical like the this x% pass rate or not.

s108
00:11:41.920 --> 00:11:44.880
So we really had to look at the traces to see what was going on.

s109
00:11:44.880 --> 00:11:57.600
And in this example, you can see in the initial trace, it actually detects that there is a disclaimer in what it's searching for and it says, okay, I found a disclaimer and now I'm going to go ahead and remove it, which was not what we asked it to do.

s110
00:11:57.600 --> 00:12:00.720
Uh, this is a sample image that I created that I ran through the agent.

s111
00:12:00.720 --> 00:12:02.240
It says, "America, we can do better.

s112
00:12:02.240 --> 00:12:03.600
It's a public parks ad."

s113
00:12:03.600 --> 00:12:10.399
And if you'll see at the bottom right, it says paid by the community of parks of keep parks clean.

s114
00:12:10.399 --> 00:12:14.720
and we sent it to the agent and it just removed it when we told the explicitly not to.

s115
00:12:14.720 --> 00:12:16.160
So those kind of things will happen.

s116
00:12:16.160 --> 00:12:21.760
So it's really important to like check the reasoning and how it's like getting to the things that you care about.

s117
00:12:21.760 --> 00:12:23.680
Uh cool one other thing.

s118
00:12:23.680 --> 00:12:28.639
So like all ML systems some things still are applicable which is my favorite part.

s119
00:12:28.639 --> 00:12:35.360
So agents will not generalize very well depending on the kind of like data sets that you've trained on.

s120
00:12:35.360 --> 00:12:43.360
It's usually a good idea to have some sort of data set to test for like edge cases like broader capabilities.

s121
00:12:43.360 --> 00:12:46.720
Uh also a very good practice to have a test set of sorts.

s122
00:12:46.720 --> 00:12:52.399
If some of you have worked in like traditional ML systems, it was always good to have test validation sets of that sort.

s123
00:12:52.399 --> 00:12:54.160
Same concept applies here.

s124
00:12:54.160 --> 00:12:59.600
Uh if you have a test set, use it sparingly and also refresh it with prod data.

s125
00:12:59.600 --> 00:13:01.279
Daniel.

s126
00:13:01.279 --> 00:13:02.320
Yes.

s127
00:13:02.320 --> 00:13:11.120
Um so then um yeah this slide is basically uh talking about hill climbing and how it can be quite rewarding.

s128
00:13:11.120 --> 00:13:23.760
So um yeah you might find that uh after human eval right uh you can uh this this diagram basically shows like the uh kind of stack of what you can do.

s129
00:13:23.760 --> 00:13:34.639
So after um human eval if it meets the bar obviously you can finish there but um it probably won't and then uh when it doesn't meet the bar uh you can first review your eval set

s130
00:13:34.639 --> 00:13:47.440
uh and find numbers like precision and recall um and then you can um iterate and basically uh you know make changes to the eval or adjust your uh rating guide

s131
00:13:47.440 --> 00:13:54.160
things like this uh or just adjusting the model itself or the agent uh adjusting its tooling.

s132
00:13:54.160 --> 00:14:00.959
Um, and then through all of these things, you can both iterate on your eval as along with um, iterating on the agent.

s133
00:14:00.959 --> 00:14:12.160
And this loop, uh, once you have a very good eval defined, it works quite well for iterating on your agent um, and doing quality hill climbing as I mentioned before, doing ablations

s134
00:14:12.160 --> 00:14:14.320
and whatnot.

s135
00:14:16.160 --> 00:14:24.000
And then um so as far as uh launch readiness you uh basically need to understand um regressions.

s136
00:14:24.000 --> 00:14:34.160
So identify where and why the model performance is uh degrading so that you can distinguish between acceptable trade-offs and critical failures.

s137
00:14:34.160 --> 00:14:45.600
Um so yeah this is basically just saying like it's very important to understand from these evals right what is the exact issue that you're having and figure out um the trade-offs here

s138
00:14:45.600 --> 00:15:04.160
and then um you should also this is a very important point so you should focus on patterns rather than isolated runs so a tempting thing is to hyperfixate on very small examples from the model right so you might have uh one run that you do with your agent and then you find uh it fails on this case

s139
00:15:04.160 --> 00:15:12.240
and you might think to yourself, okay, well, I should update the prompt based on my eval and uh the the trace of the agent, right?

s140
00:15:12.240 --> 00:15:17.920
But um if you do this, this is kind of a trap because as I said before, these are non-deterministic systems.

s141
00:15:17.920 --> 00:15:21.760
So what's more important is that you actually rely on patterns.

s142
00:15:21.760 --> 00:15:29.360
So in your golden set, it's important to have multiple examples that can cover these kind of uh patterns that you might want to see.

s143
00:15:29.360 --> 00:15:38.560
And you basically want to look at the entire picture of how often is it failing on that pattern, not that specific individual example.

s144
00:15:38.560 --> 00:15:49.920
Um, and yeah, it's important here also to invest in online evals uh and and making sure that your data is matching the real world representation.

s145
00:15:51.519 --> 00:15:52.480
Okay, awesome.

s146
00:15:52.480 --> 00:15:56.320
So, we talked a bunch about like what worked for us while we were building evals.

s147
00:15:56.320 --> 00:15:59.279
Of course, your mileage may vary depending on your application.

s148
00:15:59.279 --> 00:16:02.160
Things can uh things can differ.

s149
00:16:02.160 --> 00:16:13.279
Uh some of the things that we wanted to recap here was uh it's like what what we think makes a good eval system generally is like it should be representative of what you want your product to be great at.

s150
00:16:13.279 --> 00:16:20.480
And that will differ depending on the state at which your agent is when you're building MVP cases, it would look differently versus when you're doing production rollout.

s151
00:16:20.480 --> 00:16:21.600
So that would differ.

s152
00:16:21.600 --> 00:16:28.639
Uh but it still needs to be very much uh centered around what do you want your product to be good at and optimizing for that.

s153
00:16:28.639 --> 00:16:30.720
Uh important to of course keep it evolving.

s154
00:16:30.720 --> 00:16:38.079
That's why we talked about having your online eval having test sets that are refreshed with production data, having sampling pipelines, all sorts of things.

s155
00:16:38.079 --> 00:16:43.839
Uh highly curated golden sets which will also evolve as your use cases evolve.

s156
00:16:43.839 --> 00:16:53.759
So training teams whether it's scale raers or your cross functional teams on how to rate things how what are you expecting out of them that's also very important that's

s157
00:16:53.759 --> 00:17:07.360
uh I think now it's getting more mainstream so hopefully it's less less controversial but like six months ago our teams were like still figuring out okay how do we like do this what's expected out of it so I think investing in those trainings can be helpful

s158
00:17:07.360 --> 00:17:15.520
uh and then raider templates and rubrics with clear set of examples so you don't have scale raers coming back to you saying I'm not sure how to rate this.

s159
00:17:15.520 --> 00:17:19.760
Um lots of like things getting marked as like unknowns or I don't know things like that.

s160
00:17:19.760 --> 00:17:21.600
Also choosing the right launch metrics.

s161
00:17:21.600 --> 00:17:24.959
So some of the launch metrics launch slides that we showed you.

s162
00:17:24.959 --> 00:17:28.240
This is a very high level of generally how you would do a launch readiness.

s163
00:17:28.240 --> 00:17:29.760
You would like check it.

s164
00:17:29.760 --> 00:17:31.919
You'll do bunch of iterations on the model.

s165
00:17:31.919 --> 00:17:36.080
You'll do an AB diff or ablation and you'll try to see okay where is the regression happening?

s166
00:17:36.080 --> 00:17:38.000
What's an acceptable regression versus not?

s167
00:17:38.000 --> 00:17:39.120
Things like that.

s168
00:17:39.120 --> 00:17:47.039
As you're doing these systems, it's important to like uh get some clarity early on on what is your gatekeeping rule like what's your launch criteria.

s169
00:17:47.039 --> 00:17:49.760
Is there a certain precision recall number that you're looking at?

s170
00:17:49.760 --> 00:17:51.360
Is there some other metric that you look at?

s171
00:17:51.360 --> 00:17:55.760
If you're doing a model eval, then probably that metric looks different than just the usual precision recall.

s172
00:17:55.760 --> 00:17:58.480
So those things can also be important to keep in mind.

s173
00:17:58.480 --> 00:18:02.799
Uh yeah, those are all the tips that we have to build production grade eval.

s174
00:18:02.799 --> 00:18:04.692
Thank you.

s175
00:18:04.692 --> 00:18:06.692
[applause]

s176
00:18:09.760 --> 00:18:12.240
We're on time.

s177
00:18:14.160 --> 00:18:14.720
Awesome.

s178
00:18:14.720 --> 00:18:15.840
Thank you very much.

s179
00:18:15.840 --> 00:18:18.000
Uh do we have time for questions?

s180
00:18:18.000 --> 00:18:20.720
Staff one.

s181
00:18:20.720 --> 00:18:21.760
Do we have time for questions?

s182
00:18:21.760 --> 00:18:22.240
Just one.

s183
00:18:22.240 --> 00:18:22.640
All right.

s184
00:18:22.640 --> 00:18:23.760
You went up first, sir.

s185
00:18:23.760 --> 00:18:25.840
Go ahead.

s186
00:18:25.840 --> 00:18:32.559
Are uh all your eval judgments being performed by humans or are you also using LLM as a judge?

s187
00:18:32.559 --> 00:18:40.960
Um, and if so, what's your calibration process look like for calibrating that judge to provide good evaluations?

s188
00:18:40.960 --> 00:18:42.160
Yeah, I think that's a good question.

s189
00:18:42.160 --> 00:18:43.919
I think I wouldn't say all.

s190
00:18:43.919 --> 00:18:48.480
I think it depends on very use cases like depending on like what kind of systems you're trying to build.

s191
00:18:48.480 --> 00:18:50.880
We have of course like a plethora of use cases.

s192
00:18:50.880 --> 00:18:52.400
So, I won't say all.

s193
00:18:52.400 --> 00:19:02.880
I co can't go into details about what the benchmarking and all of that system looks like, but some of the things that we talked about in terms of disagreement rates and ma monitoring like sampling pipelines, those things hold true generally.

s194
00:19:02.880 --> 00:19:03.760
Yeah,

s195
00:19:03.760 --> 00:19:04.880
happy to chat more offline.

s196
00:19:04.880 --> 00:19:05.600
Yeah.

s197
00:19:05.600 --> 00:19:06.240
Awesome.

s198
00:19:06.240 --> 00:19:06.640
Great.

s199
00:19:06.640 --> 00:19:07.600
Thank you very much.

s200
00:19:07.600 --> 00:19:09.919
Uh Daniel and Pratika, please give them a round of applause.

s201
00:19:09.919 --> 00:19:11.760
You can always ask them questions in the back.

s202
00:19:11.760 --> 00:19:13.919
Thank you.

s203
00:19:25.258 --> 00:19:27.258
[music]