WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=NdrSPm6NCdk

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/NdrSPm6NCdk.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.120 --> 00:00:15.920
Good afternoon everybody.

s2
00:00:15.920 --> 00:00:16.720
Hi.

s3
00:00:16.720 --> 00:00:17.119
Thanks.

s4
00:00:17.119 --> 00:00:18.000
One person's awake.

s5
00:00:18.000 --> 00:00:20.640
I was just I was just going to wait until somebody said hi.

s6
00:00:20.640 --> 00:00:21.439
Uh good to see you.

s7
00:00:21.439 --> 00:00:22.720
I'm I'm so glad to be here.

s8
00:00:22.720 --> 00:00:23.920
Thanks so much for coming out.

s9
00:00:23.920 --> 00:00:26.560
Uh this is going to be a somewhat of a long session, but not so long.

s10
00:00:26.560 --> 00:00:27.359
It's an hour.

s11
00:00:27.359 --> 00:00:29.840
Uh, and I'm very excited to be here with you today in San Francisco.

s12
00:00:29.840 --> 00:00:31.679
Fun fact, I flew 18 hours to be here.

s13
00:00:31.679 --> 00:00:32.079
So cool.

s14
00:00:32.079 --> 00:00:32.800
Anyway, thank you.

s15
00:00:32.800 --> 00:00:33.120
Thank you.

s16
00:00:33.120 --> 00:00:35.280
Yeah, she's uh, someone's awake.

s17
00:00:35.280 --> 00:00:35.520
Thank you.

s18
00:00:35.520 --> 00:00:36.000
I need that.

s19
00:00:36.000 --> 00:00:37.520
I'm gonna keep looking at you now.

s20
00:00:37.520 --> 00:00:38.960
Uh, my name is Tis.

s21
00:00:38.960 --> 00:00:41.120
That's pronounced like contagious.

s22
00:00:41.120 --> 00:00:42.160
Don't worry, I'm not.

s23
00:00:42.160 --> 00:00:43.520
Uh, hopefully.

s24
00:00:43.520 --> 00:00:48.960
Uh, and over the years, I've had the privilege of working at a number of different places in various capacities.

s25
00:00:48.960 --> 00:00:51.760
Uh, learning from really incredible people, right?

s26
00:00:51.760 --> 00:00:51.840
Right.

s27
00:00:51.840 --> 00:00:57.039
And and the reason I share this is because a lot of what we're going to talk about today is not like my opinion, which is probably worth a penny.

s28
00:00:57.039 --> 00:01:03.440
Uh but but more um facts and and figures from from established uh people in the industry and peer-reviewed research.

s29
00:01:03.440 --> 00:01:03.760
Okay.

s30
00:01:03.760 --> 00:01:07.520
I uh work today as a AI engineer at IBM.

s31
00:01:07.520 --> 00:01:09.439
Uh anyone use IBM technology here?

s32
00:01:09.439 --> 00:01:09.920
No. Okay.

s33
00:01:09.920 --> 00:01:10.560
Yeah, that's what I thought.

s34
00:01:10.560 --> 00:01:15.760
Anyway, um I I uh I I work on the Watson X data team.

s35
00:01:15.760 --> 00:01:16.960
Uh we we do a lot of AI.

s36
00:01:16.960 --> 00:01:19.119
We we um train our own models.

s37
00:01:19.119 --> 00:01:19.520
Granite.

s38
00:01:19.520 --> 00:01:20.159
Anyone using that?

s39
00:01:20.159 --> 00:01:21.200
That's what I thought.

s40
00:01:21.200 --> 00:01:37.600
um soon soon open router has it anyway um and uh and and we build things like harnesses and eval pipelines and rag pipelines and all of this and so my job is is to support teams with AI and and a lot of this what what we'll talk about today is based in uh realw world experience okay but

s41
00:01:37.600 --> 00:01:51.920
we're not here to talk about any of that so to speak we're here to talk about eval in AI uh and it's it's a deep dive uh and and it's really it's going to be a fun time uh we're going to go through a lot of examples we're going to look at a few architect aritectures and techniques.

s42
00:01:51.920 --> 00:01:57.759
But before we get to this at all, we need to start by answering the question like why, right?

s43
00:01:57.759 --> 00:01:59.759
Why do we even need this?

s44
00:01:59.759 --> 00:02:02.399
Um, and I think this is kind we'll talk about what they are and how you build them.

s45
00:02:02.399 --> 00:02:04.960
But if we don't know why it's important, then we've kind of missed the boat.

s46
00:02:04.960 --> 00:02:06.159
And so I'd love to start here.

s47
00:02:06.159 --> 00:02:07.040
Why evouts?

s48
00:02:07.040 --> 00:02:12.400
Really, there's two reasons I want to highlight today to set the tone for the rest of our conversation.

s49
00:02:12.400 --> 00:02:14.160
And it will be somewhat of a long conversation.

s50
00:02:14.160 --> 00:02:16.160
So this is like foundational stuff.

s51
00:02:16.160 --> 00:02:16.560
Okay.

s52
00:02:16.560 --> 00:02:19.920
Uh, reason number one is EVAs provide reliability.

s53
00:02:19.920 --> 00:02:23.680
Uh but not just any reliability, ahead of time reliability.

s54
00:02:23.680 --> 00:02:24.959
Um what does that mean?

s55
00:02:24.959 --> 00:02:30.160
I recently I did a talk at this conference um in April at AI engineer Europe.

s56
00:02:30.160 --> 00:02:33.040
Uh the talk was titled harnesses in AI a deep dive.

s57
00:02:33.040 --> 00:02:36.319
It was actually exactly the same title except instead of eval harnesses.

s58
00:02:36.319 --> 00:02:38.400
Uh and it was a it was a pretty fun talk.

s59
00:02:38.400 --> 00:02:40.319
We we talked about exactly the same format.

s60
00:02:40.319 --> 00:02:41.280
Why harnesses?

s61
00:02:41.280 --> 00:02:42.000
What are harnesses?

s62
00:02:42.000 --> 00:02:43.200
And how do you build one?

s63
00:02:43.200 --> 00:02:45.519
And we built one live on stage.

s64
00:02:45.519 --> 00:03:00.800
uh just a like a baby's first harness where we demonstrated uh using a very cheap model GPT 3.5 Turbo at this point is basically free right using that we created a computer use agent that could go and perform tasks on a user's behalf deterministically

s65
00:03:00.800 --> 00:03:15.840
like or almost deterministically or very deterministically like it if it met with some type of authentication gate the harness itself would log in and then hand off to the agent and so we built this thing which was really wonderful and the thesis of that talk is that

s66
00:03:15.840 --> 00:03:18.239
harnesses allow for reliability.

s67
00:03:18.239 --> 00:03:20.640
They're a reliability measure also.

s68
00:03:20.640 --> 00:03:27.200
But what I want to share in in the context of eval is that harnesses tend to be more just in time reliability.

s69
00:03:27.200 --> 00:03:27.840
Does this make sense?

s70
00:03:27.840 --> 00:03:31.440
Like a harness runs is is actually run by the user.

s71
00:03:31.440 --> 00:03:34.720
Like a user will prompt a harness and the harness will do work, right?

s72
00:03:34.720 --> 00:03:37.599
Examples of harnesses in the wild claude codecs.

s73
00:03:37.599 --> 00:03:39.200
A lot of them are coding harnesses.

s74
00:03:39.200 --> 00:03:43.360
Um sometimes they they leak details about their harness.

s75
00:03:43.360 --> 00:03:44.239
Has anyone seen this?

s76
00:03:44.239 --> 00:03:50.879
like I um I used claude code and I wanted to get information from a conference website's schedule.

s77
00:03:50.879 --> 00:04:05.519
I wanted to know when a few talks were and so I who browses the internet anymore you know I I I told claude code I was like go to this website tell me when the talk is um and so it used its tool the the harness used the web search tool found the website parsed the content

s78
00:04:05.519 --> 00:04:16.239
and then responded to me in cloud code saying here's your information and then it added like by the way um the source code of this HTML web page tried to do prompt injection

s79
00:04:16.239 --> 00:04:24.880
uh in in the HTML source code was commented HTML that said ignore all previous instructions and like do other stuff, right?

s80
00:04:24.880 --> 00:04:35.600
And it told cloud code was like my harness caught it said the words my harness caught this um because I'm supposed to treat all tool call results as um non-instructional.

s81
00:04:35.600 --> 00:04:36.639
It's just data, right?

s82
00:04:36.639 --> 00:04:38.960
But it leaked that it said my harness did that.

s83
00:04:38.960 --> 00:04:44.880
And so that just makes the case that harnesses are a measure for just in time reliability.

s84
00:04:44.880 --> 00:04:49.919
And what I'm making today, the case I'm making about eval is that they allow for ahead of time reliability.

s85
00:04:49.919 --> 00:04:52.880
And really these two work as two pedals on a bicycle.

s86
00:04:52.880 --> 00:05:05.520
If you can lock in on your harnesses and you lock in on your evals, you can go very very far with AI often for very low cost as well because a lot of times you use less capable models with a great harness and a great eval set and you go very very far.

s87
00:05:05.520 --> 00:05:08.080
And so that's why eval on on the first level.

s88
00:05:08.080 --> 00:05:15.600
Um I draw a comparison here between ahead of time just in time as an eval harness uh with programming language.

s89
00:05:15.600 --> 00:05:17.280
I'm sure all of us here write code.

s90
00:05:17.280 --> 00:05:21.280
I hope uh or or used to write code before the agents, right?

s91
00:05:21.280 --> 00:05:22.720
But we kind of get how code works.

s92
00:05:22.720 --> 00:05:26.880
And um we see this same paradigm in the world of programming.

s93
00:05:26.880 --> 00:05:31.120
We've got just in time compiled languages like like JavaScript, right?

s94
00:05:31.120 --> 00:05:33.759
That that are unsafe because they're just in time, right?

s95
00:05:33.759 --> 00:05:42.560
Like if you try to do something unsafe in JavaScript like um a string two in double quotes plus an integer one, it's not three, it's 21, right?

s96
00:05:42.560 --> 00:05:45.280
because JavaScript is like that and it does that just in time.

s97
00:05:45.280 --> 00:05:51.759
Whereas if you use an ahead of time compiled language like TypeScript, string 2 plus integer one will just be like what are you doing?

s98
00:05:51.759 --> 00:05:52.479
This is wrong.

s99
00:05:52.479 --> 00:05:57.759
Your build will probably fail unless you escape out of it with like TS uh ignore or something like that, right?

s100
00:05:57.759 --> 00:06:01.840
And so that paradigm makes its way to AI with eval harnesses.

s101
00:06:01.840 --> 00:06:05.600
We get ahead of time protections with a really good um set of eval.

s102
00:06:05.600 --> 00:06:09.440
Number two, it's it also helps us like mitigate against exploits.

s103
00:06:09.440 --> 00:06:10.960
I think this is maybe a bigger one.

s104
00:06:10.960 --> 00:06:18.720
If you have a solid set of evals, you can very quickly find what your agent is maybe not doing properly that it should.

s105
00:06:18.720 --> 00:06:21.360
Um, and and it helps you on a security front.

s106
00:06:21.360 --> 00:06:30.240
We're going to actually look at a real world uh situation where this didn't work because the team I think moved too fast and broke too many things.

s107
00:06:30.240 --> 00:06:33.280
Does anyone have a guess of what I'm talking about here?

s108
00:06:33.280 --> 00:06:37.039
No. It's it's it's a Anyone recognize this logo now?

s109
00:06:37.039 --> 00:06:37.680
Do you get it?

s110
00:06:37.680 --> 00:06:38.000
Right.

s111
00:06:38.000 --> 00:06:40.880
like um this this happened like a couple months ago, right?

s112
00:06:40.880 --> 00:06:49.919
Instagram had a massive issue where they had a an AI support chatbot, an agent uh with elevated access in one of its APIs, right?

s113
00:06:49.919 --> 00:06:52.319
It could it could do mutations on user accounts.

s114
00:06:52.319 --> 00:06:54.479
I don't know if you've seen the story, but it's kind of wild.

s115
00:06:54.479 --> 00:06:58.160
Like people went to it, attackers did with a VPN.

s116
00:06:58.160 --> 00:07:05.840
So they they placed themselves in uh the geographical location of their target via VPN so that it wouldn't trip up the two-factor authentication.

s117
00:07:05.840 --> 00:07:12.319
And then they went to this chatbot and said, "Hey, I need you to add a secondary email address to this account."

s118
00:07:12.319 --> 00:07:18.479
The secondary email address is, you know, tiskumaribbm.com and the account is uh Barack Obama.

s119
00:07:18.479 --> 00:07:20.000
Can you can you please do that for me?

s120
00:07:20.000 --> 00:07:21.440
And it and it did, right?

s121
00:07:21.440 --> 00:07:31.440
Because it had this access and as a result, I mean, it was just an absolute nightmare like 20,225 accounts uh were compromised by that.

s122
00:07:31.440 --> 00:07:38.560
That's a problem that goes away one at the harness layer just in time, but two at the eval layer ahead of time.

s123
00:07:38.560 --> 00:07:40.000
Both of those could have solved this.

s124
00:07:40.000 --> 00:07:45.199
Ideally, both of those would have caught them equally um with the eval catching it before and none of this would have happened.

s125
00:07:45.199 --> 00:07:48.800
But this is kind of what happens when you move too fast and break too many things, right?

s126
00:07:48.800 --> 00:07:51.599
Evals help you pro protect against this kind of thing.

s127
00:07:51.599 --> 00:08:01.280
Uh, and and I don't know what happened inside of Meta to to allow that, but my guess is something went wrong either with the eval or the harness or both because we're too busy moving too fast and breaking too many things.

s128
00:08:01.280 --> 00:08:03.440
Uh, a side note here, maybe we shouldn't.

s129
00:08:03.440 --> 00:08:08.240
Anyway, um, let's let's talk a little bit about how not to do that.

s130
00:08:08.240 --> 00:08:11.440
That's also alternate working title of this workshop, right?

s131
00:08:11.440 --> 00:08:15.120
A eval deep dive or how not to create insecure things.

s132
00:08:15.120 --> 00:08:18.960
And so if I answer the question now, why eval really it's two things.

s133
00:08:18.960 --> 00:08:25.440
One, it's the ahead of time reliability in contrast and in complement to the just in time reliability that harnesses give you.

s134
00:08:25.440 --> 00:08:30.240
And number two, it's let's make sure that people can't do uh dangerous things.

s135
00:08:30.240 --> 00:08:30.800
Okay.

s136
00:08:30.800 --> 00:08:36.080
Uh before we move further, I have to preface this is a 1-hour workshop.

s137
00:08:36.080 --> 00:08:38.000
Um it was originally a talk.

s138
00:08:38.000 --> 00:08:42.560
Uh and so if you want to follow along, we will do some coding exercises and I'd love for you to do that.

s139
00:08:42.560 --> 00:08:45.279
I see many of your laptops out here, but if not, that's is fine.

s140
00:08:45.279 --> 00:08:46.800
Like you can just also follow along.

s141
00:08:46.800 --> 00:08:51.040
uh there will be a there will be a tell and show situation going on.

s142
00:08:51.040 --> 00:08:57.839
So uh I I'll be talking a bit but then after that we'll actually like build evals from scratch and explore them like hands-on with code.

s143
00:08:57.839 --> 00:08:59.040
Uh and that that'll be fun.

s144
00:08:59.040 --> 00:09:02.720
So before we get ahead of ourselves, what even are eval?

s145
00:09:02.720 --> 00:09:10.320
Uh what are and and this is something I feel like we should talk about because everything's moving so fast and I've spoken you may have seen the line outside.

s146
00:09:10.320 --> 00:09:10.959
It was really long.

s147
00:09:10.959 --> 00:09:16.160
Uh, I got to talk to some people uh, and you're just like, "Hey, between you and me, just nobody's listening.

s148
00:09:16.160 --> 00:09:19.760
Like, can you like confidently explain to me what eval?"

s149
00:09:19.760 --> 00:09:25.120
And, and most people like, "I I don't I kind of Yes, but I feel a bit of imposter syndrome because we're moving so fast."

s150
00:09:25.120 --> 00:09:36.000
And that's kind of the vibe I get is like there's a pressure to like know what it means and and reason about it like fluently, but but sometimes people don't have the time because everything's moving fast and then they kind of make up stuff as they go, right?

s151
00:09:36.000 --> 00:09:44.080
And so after this talk, my hope is similar with the harness talk that I did earlier that all of us in this room walk out of here like confident af

s152
00:09:44.080 --> 00:09:50.640
uh with with evalu that we can we can deploy them in production our apps become way more robust and so on and so forth.

s153
00:09:50.640 --> 00:10:03.680
So from first principles from first principles what are eval introduce this from from maybe a coding background is they're like unit tests.

s154
00:10:03.680 --> 00:10:05.279
Uh anyone know what a unit test is?

s155
00:10:05.279 --> 00:10:05.839
Okay, everyone.

s156
00:10:05.839 --> 00:10:06.160
Great.

s157
00:10:06.160 --> 00:10:08.160
Uh, unit they're just deterministic.

s158
00:10:08.160 --> 00:10:09.440
What what are actually unit tests?

s159
00:10:09.440 --> 00:10:13.279
They um they allow for reliability instrumentation.

s160
00:10:13.279 --> 00:10:18.480
They make your functions, your code more um testable, more reliable.

s161
00:10:18.480 --> 00:10:21.120
But it's it only really works for deterministic systems.

s162
00:10:21.120 --> 00:10:21.279
Right?

s163
00:10:21.279 --> 00:10:25.120
So you have a function add add one, two, you expect three.

s164
00:10:25.120 --> 00:10:25.920
You assert three.

s165
00:10:25.920 --> 00:10:29.200
You you literally write in code expect add one comma 2 to be three.

s166
00:10:29.200 --> 00:10:29.519
Right?

s167
00:10:29.519 --> 00:10:33.279
You you build this instrumentation for a deterministic system.

s168
00:10:33.279 --> 00:10:37.360
The problem is with AI almost nothing is ever deterministic, right?

s169
00:10:37.360 --> 00:10:40.079
The only thing that's deterministic is that nothing is deterministic.

s170
00:10:40.079 --> 00:10:41.120
Wow, that's deep.

s171
00:10:41.120 --> 00:10:44.240
Um, and so and so how do we we can't use unit test for this anymore.

s172
00:10:44.240 --> 00:10:46.079
Okay, but then use regular expressions.

s173
00:10:46.079 --> 00:10:48.399
Sure, but even that has its limits, right?

s174
00:10:48.399 --> 00:10:54.320
And so we need a fundamentally new way to test the reli to instrument the reliability of AI applications.

s175
00:10:54.320 --> 00:10:55.680
And that's exactly what eval is.

s176
00:10:55.680 --> 00:11:01.200
Eval can be thought of as reliability instrumentation but for non-deterministic systems.

s177
00:11:01.200 --> 00:11:01.760
That's all it is.

s178
00:11:01.760 --> 00:11:02.399
It's instrument.

s179
00:11:02.399 --> 00:11:05.839
It's like tests but for systems that are not deterministic.

s180
00:11:05.839 --> 00:11:08.959
By the way, I see many of you taking up your phones and taking pictures of the slides.

s181
00:11:08.959 --> 00:11:09.519
That's awesome.

s182
00:11:09.519 --> 00:11:10.959
I love this feedback as a speaker.

s183
00:11:10.959 --> 00:11:12.800
It's the most validating thing in the world.

s184
00:11:12.800 --> 00:11:13.519
More people do it.

s185
00:11:13.519 --> 00:11:16.399
Anyway, um so so that's that's what we're working with.

s186
00:11:16.399 --> 00:11:21.120
We're trying to somehow make it observable and reliable through instrumentation.

s187
00:11:21.120 --> 00:11:26.640
Now, this is this next slide is the one that people online are going to like give me flack for.

s188
00:11:26.640 --> 00:11:28.240
Uh people on YouTube are very mean.

s189
00:11:28.240 --> 00:11:29.680
I'm looking at you YouTube people.

s190
00:11:29.680 --> 00:11:30.240
Not you, sir.

s191
00:11:30.240 --> 00:11:31.440
The camera next to you.

s192
00:11:31.440 --> 00:11:40.560
uh very mean like I I every time I do a talk at this conference the comments are just like evil anyway um so you love this one YouTube in a very crude way right

s193
00:11:40.560 --> 00:11:50.160
um you could say that eelss are just really just fuzzy unitists you know uh and I think this is the way to think about them now people are going to be like oh my gosh you're like being so reductionary

s194
00:11:50.160 --> 00:12:03.760
sure but it helps us what did what did the great scientist Dr. refinement say right he said you don't really understand something unless you can speak about it plainly and if we speak about eval plainly I'm calling it fuzzy unit tests I'll tell you why because here this is what a unit tests

s195
00:12:03.760 --> 00:12:17.440
did this function return exactly this number right that's what a unit tests what is an eval test and eval asks a similar question but just nondeterministically did this agent

s196
00:12:17.440 --> 00:12:25.360
behave acceptably across many scenario these two you know the the meme from the office where they're Those are the same picture.

s197
00:12:25.360 --> 00:12:27.600
Uh that's the same picture.

s198
00:12:27.600 --> 00:12:28.880
It's just fuzzier.

s199
00:12:28.880 --> 00:12:31.040
You know, unit tests do it that way.

s200
00:12:31.040 --> 00:12:32.800
Evals do it this way.

s201
00:12:32.800 --> 00:12:38.639
Almost all eval sets data sets have reusable components between them.

s202
00:12:38.639 --> 00:12:41.279
Uh and we did this with a harness in the harness talk as well.

s203
00:12:41.279 --> 00:12:48.160
Every almost all eval setups have some shared attributes and and this is how you can spot a really well- definfined eval suite.

s204
00:12:48.160 --> 00:12:48.399
Okay.

s205
00:12:48.399 --> 00:12:50.959
Number one, they have test cases.

s206
00:12:50.959 --> 00:12:53.120
Uh, similar to unit tests, right?

s207
00:12:53.120 --> 00:12:54.560
Let's use the meta example.

s208
00:12:54.560 --> 00:12:59.839
Did the customer verify that they're them?

s209
00:12:59.839 --> 00:13:00.079
Right?

s210
00:13:00.079 --> 00:13:01.839
That's part of the test case.

s211
00:13:01.839 --> 00:13:06.320
Um, you have a a set of inputs and outputs.

s212
00:13:06.320 --> 00:13:11.200
You have a set of customer requests to your chatbot and generated answers.

s213
00:13:11.200 --> 00:13:14.240
You have those cases against which you can create a verdict.

s214
00:13:14.240 --> 00:13:14.800
This was good.

s215
00:13:14.800 --> 00:13:15.600
This was bad.

s216
00:13:15.600 --> 00:13:17.600
Number two, you have expected behavior.

s217
00:13:17.600 --> 00:13:18.880
Those are the answers.

s218
00:13:18.880 --> 00:13:24.160
So you've got test cases, that's your inputs to your chatbot or whatever it is you're building, and you've got expected behavior.

s219
00:13:24.160 --> 00:13:25.519
I expect this outcome.

s220
00:13:25.519 --> 00:13:30.880
Now, your expected behavior is not an exact string, but it's like, this is more or less the direction I wanted it to go.

s221
00:13:30.880 --> 00:13:32.560
Number three, you need a judge.

s222
00:13:32.560 --> 00:13:35.519
You need somebody to say, "Yes, that's right."

s223
00:13:35.519 --> 00:13:40.880
And this is way harder to do in a non-deterministic system because it's subjective, right?

s224
00:13:40.880 --> 00:13:46.240
You can say 1 plus 2 is three, but you can't say um you know, I'm a great speaker.

s225
00:13:46.240 --> 00:13:49.120
That's just not some of you are like, "Man, I wish I didn't come here."

s226
00:13:49.120 --> 00:13:49.680
You know what I mean?

s227
00:13:49.680 --> 00:13:50.959
You on YouTube definitely.

s228
00:13:50.959 --> 00:13:53.839
So anyway, um it's you need a judge somehow.

s229
00:13:53.839 --> 00:13:54.399
Who is that?

s230
00:13:54.399 --> 00:13:59.839
And then finally, you need a way to aggregate all your results and come up with a probability score.

s231
00:13:59.839 --> 00:14:04.160
This is this is like 80% the right direction and therefore it's good.

s232
00:14:04.160 --> 00:14:05.360
This is like 90%.

s233
00:14:05.360 --> 00:14:06.560
This is like 12%, right?

s234
00:14:06.560 --> 00:14:08.560
And so you need some type of aggregation.

s235
00:14:08.560 --> 00:14:19.600
Almost every eval set in production today that is of any quality will definitely have these components and we'll build them all in our like show section just a little bit later.

s236
00:14:19.600 --> 00:14:20.240
Okay.

s237
00:14:20.240 --> 00:14:24.560
Um eval exist these components then come together.

s238
00:14:24.560 --> 00:14:31.519
They're like the rings from Captain Planet, you know, they they come together to answer one question or a set of questions.

s239
00:14:31.519 --> 00:14:36.560
Given a scenario, did the agent number one follow policy?

s240
00:14:36.560 --> 00:14:37.040
Right?

s241
00:14:37.040 --> 00:14:41.360
For example, u many e-commerce websites, companies have return policies.

s242
00:14:41.360 --> 00:14:46.959
You can return things within the first two weeks of purchase, but if it's like a hygienic thing, you can't, right?

s243
00:14:46.959 --> 00:14:48.240
There's policy like this.

s244
00:14:48.240 --> 00:14:49.760
Almost everybody has policy.

s245
00:14:49.760 --> 00:14:52.240
So, did the agent follow it or not?

s246
00:14:52.240 --> 00:14:54.160
Did the agent use the right tools?

s247
00:14:54.160 --> 00:15:01.760
And and and we'll talk about this in a little bit more detail coming up, but tool use can be eval to speak without any um AI.

s248
00:15:01.760 --> 00:15:18.079
It it's just because if you persist the tool calls as JSON in some array you can just write deterministic code in the list of tool calls was this tool called with what arguments you can still use unit tests for that but did it use the right tools number three um did it actually ultimately resolve the issue right

s249
00:15:18.079 --> 00:15:31.440
um this is what eval exist to answer and all of these questions oftentimes are semi-binary they're not really binary um and so how do we do it well there are a few techniques with evas Um,

s250
00:15:31.440 --> 00:15:34.000
and we'll go pretty deep on on some of them.

s251
00:15:34.000 --> 00:15:36.320
I don't think we have time to go through all of them.

s252
00:15:36.320 --> 00:15:40.399
Uh, I I hope we do genuinely, but we'll we'll kind of get the big hitters here.

s253
00:15:40.399 --> 00:15:43.120
Number one, um, exact match, right?

s254
00:15:43.120 --> 00:15:49.199
Exact match is literally I searched for exactly this tool in my list of tool calls and I found it with these arguments.

s255
00:15:49.199 --> 00:15:51.279
Very easy, free, costs nothing to run.

s256
00:15:51.279 --> 00:15:52.160
Anyone can do it.

s257
00:15:52.160 --> 00:15:55.360
In fact, we probably should be at least examining our message envelopes.

s258
00:15:55.360 --> 00:15:57.360
Number two, um, schema validation.

s259
00:15:57.360 --> 00:16:01.600
Now, we're getting a little bit more in the weeds uh because you you can have fuzzy rules, right?

s260
00:16:01.600 --> 00:16:05.600
And so, if you have JSON blobs or message envelopes, you'll want to validate them.

s261
00:16:05.600 --> 00:16:08.160
Number three, pair-wise comparison.

s262
00:16:08.160 --> 00:16:10.000
Does anyone know what pair-wise comparison is?

s263
00:16:10.000 --> 00:16:10.240
Yeah.

s264
00:16:10.240 --> 00:16:12.079
No, it's when Wow, awesome, dude.

s265
00:16:12.079 --> 00:16:12.560
One guy.

s266
00:16:12.560 --> 00:16:13.680
What's your name?

s267
00:16:13.680 --> 00:16:14.160
Lenny.

s268
00:16:14.160 --> 00:16:14.399
Awesome.

s269
00:16:14.399 --> 00:16:15.199
Lenny's awesome.

s270
00:16:15.199 --> 00:16:18.959
So, um pair pairwise comparison is exactly what it sounds like.

s271
00:16:18.959 --> 00:16:21.519
You have a pair of things or that.

s272
00:16:21.519 --> 00:16:26.320
And and pairwise comparison is a technique to, for example, your um LM arena.

s273
00:16:26.320 --> 00:16:27.360
Anyone know LM Arena?

s274
00:16:27.360 --> 00:16:28.880
It's a great arena.ai, I think.

s275
00:16:28.880 --> 00:16:29.440
Great product.

s276
00:16:29.440 --> 00:16:33.360
They show you um this model generated this, this model generated that.

s277
00:16:33.360 --> 00:16:34.320
Which one do you like?

s278
00:16:34.320 --> 00:16:36.000
That is a pair-wise comparison, right?

s279
00:16:36.000 --> 00:16:46.399
And if you have a customer support chatbot, you may want as part of your eval to say my support bot generated two possible answers, give it to a judge, an LLM judge, and say, "Which one should I choose?"

s280
00:16:46.399 --> 00:16:47.759
That's a pair-wise comparison.

s281
00:16:47.759 --> 00:16:50.720
A pair-wise comparison is not very strong for a couple reasons.

s282
00:16:50.720 --> 00:16:54.320
I I'll share that now because it is a deep dive, and I think it's worth going on some rabbit holes here.

s283
00:16:54.320 --> 00:17:00.240
pair-wise is is not ideal because for the one thing um there is bias in LLM judges.

s284
00:17:00.240 --> 00:17:02.160
We'll look at we'll talk about this in a little bit more detail.

s285
00:17:02.160 --> 00:17:02.800
It's wild.

s286
00:17:02.800 --> 00:17:06.400
Uh number two, um AI is modeled after us.

s287
00:17:06.400 --> 00:17:15.600
Neural networks literally are modeled after human brains and humans are fundamentally really bad at pairwise comparison when contrasted with the alternate which is pointwise comparison.

s288
00:17:15.600 --> 00:17:20.400
Uh so for example, you go to a restaurant and you see a menu with like many options, right?

s289
00:17:20.400 --> 00:17:22.240
That's a pair that's a pair-wise comparison.

s290
00:17:22.240 --> 00:17:23.199
You're like, "Oh, I don't know.

s291
00:17:23.199 --> 00:17:38.720
anyone experiences like you're like I have analysis paralysis bro there's too many things um AI works exactly the same way and so pointwise is not which dish do you want pointwise is do you want food yes or no right and so pointwise easy pair-wise hard and so pair-wise is kind of a technique we won't spend a lot

s292
00:17:38.720 --> 00:17:55.200
of time on because it's it's not very uh reliable as compared to pointwise in fact one study showed that pair-wise comparison fell apart 39% uh it was 39% more vulnerable to uh attack vector vectors than pointwise and pointwise fell apart just like five times.

s293
00:17:55.200 --> 00:17:56.799
It was definitely single digits, right?

s294
00:17:56.799 --> 00:17:58.799
And so we want to go for pointwise when we can.

s295
00:17:58.799 --> 00:18:05.600
Finally, probably the most popular technique uh with eval is using an LLM as a judge.

s296
00:18:05.600 --> 00:18:07.520
I mean this is just kind of gold standard, right?

s297
00:18:07.520 --> 00:18:09.840
You have either a pair-wise or pointwise.

s298
00:18:09.840 --> 00:18:10.960
It's non-deterministic.

s299
00:18:10.960 --> 00:18:15.280
So you take whatever it is and you send it to a judge, an LLM that says, "What do you think?

s300
00:18:15.280 --> 00:18:16.799
Is this close enough?"

s301
00:18:16.799 --> 00:18:20.160
Um, of course, those systems themselves are also non-deterministic.

s302
00:18:20.160 --> 00:18:21.440
And so how do you trust it?

s303
00:18:21.440 --> 00:18:22.960
How do you build a judge?

s304
00:18:22.960 --> 00:18:24.320
We're going to build a judge on stage.

s305
00:18:24.320 --> 00:18:25.120
We'll talk about that.

s306
00:18:25.120 --> 00:18:28.880
Okay, but that's kind of the the sort of state-of-the-art right now.

s307
00:18:28.880 --> 00:18:30.880
Um the problem is eval lie.

s308
00:18:30.880 --> 00:18:34.799
In fact, the original version of this talk was your evals are lying to you.

s309
00:18:34.799 --> 00:18:37.039
But I felt like that was too clickbait, so I changed it.

s310
00:18:37.039 --> 00:18:38.720
Um you're welcome.

s311
00:18:38.720 --> 00:18:42.240
E evals can lie sometimes and and it's not good when they do.

s312
00:18:42.240 --> 00:18:43.039
How can eval?

s313
00:18:43.039 --> 00:18:47.600
Well, number one, they have bias for position.

s314
00:18:47.600 --> 00:18:57.280
So I if you say if you give them a pair-wise comparison, hey listen, um I want to send this option one or that option two, which one should I send?

s315
00:18:57.280 --> 00:19:00.960
And a poor judge will almost always say, yeah, just the first one.

s316
00:19:00.960 --> 00:19:01.840
Do it.

s317
00:19:01.840 --> 00:19:02.720
And then you swap.

s318
00:19:02.720 --> 00:19:06.320
And the way of checking for this bias is you swap the order of options.

s319
00:19:06.320 --> 00:19:09.600
And if it's still the first one, though the first one is wrong, that's how you know this.

s320
00:19:09.600 --> 00:19:13.120
This happens pretty often with with especially cheaper models and and fine-tunes.

s321
00:19:13.120 --> 00:19:14.480
Uh position bias pretty common.

s322
00:19:14.480 --> 00:19:16.640
Number two, sick of fancy.

s323
00:19:16.640 --> 00:19:18.160
Uh, great example, right?

s324
00:19:18.160 --> 00:19:20.400
Somebody buys something on your e-commerce website.

s325
00:19:20.400 --> 00:19:24.000
I bought headphones 20 days ago and I need to return it.

s326
00:19:24.000 --> 00:19:24.640
Can I return it?

s327
00:19:24.640 --> 00:19:26.160
The return policy is 14-day return.

s328
00:19:26.160 --> 00:19:26.880
You can't, right?

s329
00:19:26.880 --> 00:19:30.720
And but but a customer support bot says one, no, you can't.

s330
00:19:30.720 --> 00:19:32.080
Or two, yes, you can.

s331
00:19:32.080 --> 00:19:38.960
If you ask an LLM judge which one, it will choose yes, you can even though it's not within policy because they love making you happy.

s332
00:19:38.960 --> 00:19:42.320
I mean, the system prompt is you are a personal assistant.

s333
00:19:42.320 --> 00:19:43.919
You are a helpful assistant, right?

s334
00:19:43.919 --> 00:19:44.960
And so, what's more helpful?

s335
00:19:44.960 --> 00:19:46.720
to say yes, you can return it or no, you can't.

s336
00:19:46.720 --> 00:19:47.679
It's not within policy.

s337
00:19:47.679 --> 00:19:52.559
So, sick of fancy is a real bias where often times the nicest response will get selected.

s338
00:19:52.559 --> 00:19:54.000
Uh, and we need to account for that.

s339
00:19:54.000 --> 00:19:55.200
You're absolutely right.

s340
00:19:55.200 --> 00:19:57.520
This probably going to get selected a lot, right?

s341
00:19:57.520 --> 00:19:59.520
Number three is self-reference.

s342
00:19:59.520 --> 00:20:01.200
I don't know if you know this or if you've experienced this.

s343
00:20:01.200 --> 00:20:02.240
It's wild.

s344
00:20:02.240 --> 00:20:10.240
But if you give an LLM judge two options, a pair-wise comparison, and option A is text that was generated by the same family.

s345
00:20:10.240 --> 00:20:13.600
So, if if your judge is like GPT 5.6, what is it?

s346
00:20:13.600 --> 00:20:13.919
Soul.

s347
00:20:13.919 --> 00:20:14.480
Yeah, the new one.

s348
00:20:14.480 --> 00:20:22.559
Soul um and you ask Soul to judge something generated by GPD 3.5 Turbo versus human input uh it will always choose itself.

s349
00:20:22.559 --> 00:20:24.960
It's crazy like they they love their own generated text.

s350
00:20:24.960 --> 00:20:29.840
And so your judge needs to be a different model family from the rest of your corpus if you're generating text.

s351
00:20:29.840 --> 00:20:31.600
Um and this is this is really wellproven.

s352
00:20:31.600 --> 00:20:32.640
There's archive papers.

s353
00:20:32.640 --> 00:20:33.919
I should put links in here anyway.

s354
00:20:33.919 --> 00:20:34.720
Ask me after.

s355
00:20:34.720 --> 00:20:37.039
And number four, they love verbosity.

s356
00:20:37.039 --> 00:20:37.840
They like tokens.

s357
00:20:37.840 --> 00:20:44.080
And so if you give them the wrong answer but it's dressed in many tokens, usually that one's going to get choose chosen choosed.

s358
00:20:44.080 --> 00:20:45.520
That one's going to get chosen as well.

s359
00:20:45.520 --> 00:20:45.679
Right?

s360
00:20:45.679 --> 00:20:47.120
So these are your ways eval can lie.

s361
00:20:47.120 --> 00:20:54.720
And this is how you want to be kind of sensitive as you build um as you build your eval pipeline because just because it's green doesn't mean it works.

s362
00:20:54.720 --> 00:20:55.120
Okay?

s363
00:20:55.120 --> 00:20:57.200
We talked a little bit about pair-wise pitfalls.

s364
00:20:57.200 --> 00:21:03.919
And we talked about point-wise comparison where um one option yes or no is a lot better than two options.

s365
00:21:03.919 --> 00:21:05.120
Which one do you want?

s366
00:21:05.120 --> 00:21:13.760
Finally, I'd love to dive deeper into the LLM as a judge concept because this architecture, this technique is is pretty much the the gold standard for for quality evas.

s367
00:21:13.760 --> 00:21:14.559
Maybe not the gold standard.

s368
00:21:14.559 --> 00:21:15.360
It's very common.

s369
00:21:15.360 --> 00:21:17.360
Um, and it's very hard to do right as well.

s370
00:21:17.360 --> 00:21:18.880
So, I think it's worth the time here.

s371
00:21:18.880 --> 00:21:24.000
The the main thing I want to talk about is how do you actually train a judge?

s372
00:21:24.000 --> 00:21:26.799
And I don't mean like machine learning training.

s373
00:21:26.799 --> 00:21:30.640
Uh, I mean more you have an LLM that you want to use as your judge.

s374
00:21:30.640 --> 00:21:33.760
How do you make sure it agrees with you?

s375
00:21:33.760 --> 00:21:34.159
Right?

s376
00:21:34.159 --> 00:21:38.000
Um, and the way you do this is you have a ideally you have a data set.

s377
00:21:38.000 --> 00:21:39.840
By the way, an eval is not a onetime thing.

s378
00:21:39.840 --> 00:21:40.960
It's a it's a data set.

s379
00:21:40.960 --> 00:21:45.280
It's a it's a collection of this input, this output.

s380
00:21:45.280 --> 00:21:46.080
What do you think?

s381
00:21:46.080 --> 00:21:46.320
Right?

s382
00:21:46.320 --> 00:21:48.640
And you want a high percentage of agreement there.

s383
00:21:48.640 --> 00:21:54.400
So, step one to training your judge is you want to have a long list of of data.

s384
00:21:54.400 --> 00:21:58.320
Uh, let's use the headphones return example that I just referenced a minute ago.

s385
00:21:58.320 --> 00:22:05.280
A customer has bought headphones uh 20 days ago and they want to return it, but your return policy is you can only return things within 14 days, right?

s386
00:22:05.280 --> 00:22:08.480
And so you have an LLM judge full of these situations.

s387
00:22:08.480 --> 00:22:15.360
The customer has asked this, the human assistant or the AI assistant said that and then this is the score.

s388
00:22:15.360 --> 00:22:24.880
So you want to have a long list of question answer score and the score is one that you or a subject matter expert, a human has scored, a trustworthy score, right?

s389
00:22:24.880 --> 00:22:25.039
Right?

s390
00:22:25.039 --> 00:22:29.760
So you want this is your um I don't want to use the term because it's not right but this is kind of your golden data set.

s391
00:22:29.760 --> 00:22:31.440
You want like a a question answer score.

s392
00:22:31.440 --> 00:22:35.039
You want a huge collection of these 30 or so to start with and then it can grow.

s393
00:22:35.039 --> 00:22:35.600
Okay.

s394
00:22:35.600 --> 00:22:36.960
So you have that.

s395
00:22:36.960 --> 00:22:43.679
What you then do to train your judge is you remove your scoring and you just keep the question and the answer and you send that to your judge and you're like what do you think?

s396
00:22:43.679 --> 00:22:45.520
And it will give you a bunch of scores as well.

s397
00:22:45.520 --> 00:22:45.840
Right?

s398
00:22:45.840 --> 00:22:48.240
And then you compute the delta between them.

s399
00:22:48.240 --> 00:22:51.919
Meaning what what percentage of agreement do I have with this judge?

s400
00:22:51.919 --> 00:22:56.240
And what you want to go for is 80 to 85% maybe a bit more.

s401
00:22:56.240 --> 00:23:08.640
Uh but under 80% you're not going to get much because even human beings like if you had a team of humans who are supporting people all the humans on that team will also agree about 80% of the time and nobody agrees 100% of the time about everything.

s402
00:23:08.640 --> 00:23:08.960
Right?

s403
00:23:08.960 --> 00:23:10.159
In that case you've overfitit.

s404
00:23:10.159 --> 00:23:11.120
That's a whole other problem.

s405
00:23:11.120 --> 00:23:17.039
And so you want to make sure that your judge agrees with your golden data set or you 80 or 85% of the time.

s406
00:23:17.039 --> 00:23:23.039
If you agree, if your judge agrees 100% of the time with you, it's probably somehow sophantic.

s407
00:23:23.039 --> 00:23:24.400
There's some bias poisoning.

s408
00:23:24.400 --> 00:23:25.679
Something's not right.

s409
00:23:25.679 --> 00:23:28.240
Um, and it could happen that you've overfit your data.

s410
00:23:28.240 --> 00:23:28.720
What does that mean?

s411
00:23:28.720 --> 00:23:34.159
That just means your your judge is trained exactly on your specific examples and it's not ready for like the real world.

s412
00:23:34.159 --> 00:23:36.559
Okay, so 100% is is kind of a red flag.

s413
00:23:36.559 --> 00:23:44.960
Once you have a judge that agrees with you like 80% of the time and it's ready, it's time to put this in production.

s414
00:23:44.960 --> 00:23:46.000
How do we do this?

s415
00:23:46.000 --> 00:23:47.039
How do we put this in production?

s416
00:23:47.039 --> 00:23:52.640
I think the best the safest way to do this is to have production be an agreement gate.

s417
00:23:52.640 --> 00:23:55.360
Meaning I'm going to run my evals on CI.

s418
00:23:55.360 --> 00:24:00.320
Uh there's tools here represent sponsors of this conference that that uh can help you with that.

s419
00:24:00.320 --> 00:24:06.400
Um run this on CI and if my judge agrees with me less than the threshold that is 80%.

s420
00:24:06.400 --> 00:24:07.760
Uh then I don't ship, right?

s421
00:24:07.760 --> 00:24:08.640
It's it's gated.

s422
00:24:08.640 --> 00:24:10.400
It's red on less agreement.

s423
00:24:10.400 --> 00:24:12.720
It's green on 80 to 90% agreement.

s424
00:24:12.720 --> 00:24:15.520
That's kind of the move and that's how you you go to production.

s425
00:24:15.520 --> 00:24:20.559
But once you're in production, you're not you need to do some more work to stay relevant.

s426
00:24:20.559 --> 00:24:31.200
I will also add when going to production, you want to be careful about how you identify your judge because you could just use like a model name GPT-40- mini.

s427
00:24:31.200 --> 00:24:36.400
Uh but that's maybe a bit dangerous because that's just an alias for like a specific version of GP40 mini.

s428
00:24:36.400 --> 00:24:42.240
And if for whatever reason the underlying version changes, um you you won't know and your eval will fail.

s429
00:24:42.240 --> 00:24:44.559
So maybe pinning the version would would make more sense.

s430
00:24:44.559 --> 00:24:51.760
But once you're in production, there's a high chance that your data is going to go stale and you'll need to stay relevant, right?

s431
00:24:51.760 --> 00:24:55.039
Maybe Meta's thing with Instagram was that they just didn't have the latest data.

s432
00:24:55.039 --> 00:25:00.000
And so when you're in production, how you stay relevant with your evals is you skim traffic.

s433
00:25:00.000 --> 00:25:06.080
So you'll start receive your thing and you'll start to collect, okay, this was the input from a new user.

s434
00:25:06.080 --> 00:25:07.760
This was the output from our agent.

s435
00:25:07.760 --> 00:25:12.240
And then you have an internal team to score this stuff manually.

s436
00:25:12.240 --> 00:25:15.039
And you again just give that to your judge and check the agreement.

s437
00:25:15.039 --> 00:25:16.159
So it's a living data set.

s438
00:25:16.159 --> 00:25:17.120
Does this make sense?

s439
00:25:17.120 --> 00:25:19.279
Uh that's kind of what you what you want.

s440
00:25:19.279 --> 00:25:20.159
That's the best move really.

s441
00:25:20.159 --> 00:25:21.600
And so your data set is living.

s442
00:25:21.600 --> 00:25:24.720
Your judge is up to date and everything is is peachy.

s443
00:25:24.720 --> 00:25:27.760
This is going to be very very difficult to to break.

s444
00:25:27.760 --> 00:25:32.000
And your eval will genuinely at this point provide you ahead of time reliability.

s445
00:25:32.000 --> 00:25:33.600
We've literally come like full circle.

s446
00:25:33.600 --> 00:25:34.720
Like we started at nothing.

s447
00:25:34.720 --> 00:25:35.679
Unit tests.

s448
00:25:35.679 --> 00:25:36.480
Jesus.

s449
00:25:36.480 --> 00:25:40.240
And now and now we're like in production with live data.

s450
00:25:40.240 --> 00:25:54.000
Um the final thing I'll say before we kind of break into more fun code time uh is that you want your your policy also changes a lot right like let me tell you I work at a company that is 115

s451
00:25:54.000 --> 00:26:13.039
years old centuries that's how old we are Jesus and and not only that there's like 400 roughly 400,000 people who work with me right that's what it's almost a half a million there are cities on earth that have fewer people than this company worldwide right And so what's what happens at a company this size is

s452
00:26:13.039 --> 00:26:17.840
policy both internal and external starts to become living over 115 years.

s453
00:26:17.840 --> 00:26:20.159
Your policies are going to change and things are going to be different.

s454
00:26:20.159 --> 00:26:28.159
And ideally we have living policy that an agent can can discover automatically without me writing down these are the rules of IBM.

s455
00:26:28.159 --> 00:26:28.960
You know what I mean?

s456
00:26:28.960 --> 00:26:30.640
Uh living policy that's the move.

s457
00:26:30.640 --> 00:26:31.520
How do we do that?

s458
00:26:31.520 --> 00:26:33.919
Um agentic rag is kind of a part of it.

s459
00:26:33.919 --> 00:26:34.240
Right?

s460
00:26:34.240 --> 00:26:38.159
So at IBM we actually internally um we use a ton of AI systems.

s461
00:26:38.159 --> 00:26:43.039
In fact, HR at a company with 400,000 employees can be challenging for humans.

s462
00:26:43.039 --> 00:26:51.039
Um, and so we have like a tool internally that we can just chat with and and it gives us up-to-date highquality well-grounded answers.

s463
00:26:51.039 --> 00:26:55.840
I can book my I can like request vacation time just by querying this thing and it it's all perfect.

s464
00:26:55.840 --> 00:26:58.559
I don't we don't like I don't use workday or anything just chat.

s465
00:26:58.559 --> 00:27:06.159
It's so incredible and the reason is it it it has an incredible eval pipeline um based on real world policy, right?

s466
00:27:06.159 --> 00:27:07.600
Um, we built a tool at IBM.

s467
00:27:07.600 --> 00:27:08.320
It's open source.

s468
00:27:08.320 --> 00:27:09.840
Uh, we give it to our customers also.

s469
00:27:09.840 --> 00:27:11.120
It's called open rag.

s470
00:27:11.120 --> 00:27:12.559
Uh, and it's built for this.

s471
00:27:12.559 --> 00:27:16.880
So, it's a massive knowledge base where you can upload literally everything.

s472
00:27:16.880 --> 00:27:25.440
Teams, calls, videos, audios, excel sheets, powerpoints, all the stuff that your company accumulates over 115 years and it makes it available.

s473
00:27:25.440 --> 00:27:29.919
It is actually a rag agent and it serves as the policy retriever for our evals.

s474
00:27:29.919 --> 00:27:30.640
It's so cool.

s475
00:27:30.640 --> 00:27:34.159
Uh, and so that's kind of the the end of of the of the pipeline.

s476
00:27:34.159 --> 00:27:36.159
So you've you've gone from I have a unit test.

s477
00:27:36.159 --> 00:27:37.679
I've made it a bit fuzzy.

s478
00:27:37.679 --> 00:27:40.000
I now um I have a judge.

s479
00:27:40.000 --> 00:27:40.799
My judge is trained.

s480
00:27:40.799 --> 00:27:41.679
I put it in production.

s481
00:27:41.679 --> 00:27:42.960
I gate on agreement.

s482
00:27:42.960 --> 00:27:43.919
Um it's living.

s483
00:27:43.919 --> 00:27:44.880
I'm skimming data.

s484
00:27:44.880 --> 00:27:46.320
I'm adding it to my data set.

s485
00:27:46.320 --> 00:27:48.960
I'm even discovering policy myself.

s486
00:27:48.960 --> 00:27:49.520
Right?

s487
00:27:49.520 --> 00:27:51.200
Um that's really it.

s488
00:27:51.200 --> 00:27:53.840
And so we've talked we've covered a lot of concepts here.

s489
00:27:53.840 --> 00:27:56.960
I'd love to make it practical and actually let's just build.

s490
00:27:56.960 --> 00:27:59.520
Uh now I'll need you to to give me some grace.

s491
00:27:59.520 --> 00:28:01.039
I don't know how the Wi-Fi situation is.

s492
00:28:01.039 --> 00:28:04.480
I was told literally before coming here, somebody said to me, they're like, "Hey, man, just be prepared.

s493
00:28:04.480 --> 00:28:06.159
The internet's the worst."

s494
00:28:06.159 --> 00:28:08.880
Uh, and and I said, "Definitely it is."

s495
00:28:08.880 --> 00:28:10.159
And they said, "No, no, in this room."

s496
00:28:10.159 --> 00:28:12.000
I was like, "Oh, I thought you meant in general.

s497
00:28:12.000 --> 00:28:15.039
Uh, it also takes me away from my family."

s498
00:28:15.039 --> 00:28:17.039
Anyway, so um let's see.

s499
00:28:17.039 --> 00:28:17.840
Let's see how we go.

s500
00:28:17.840 --> 00:28:19.039
So, we're going to here's what we're going to do.

s501
00:28:19.039 --> 00:28:25.200
We're going to start we're going to go on that journey that I just narrated, but we're going to go on it like hands-on with code.

s502
00:28:25.200 --> 00:28:26.399
You're welcome to follow along.

s503
00:28:26.399 --> 00:28:29.919
Of course, I don't expect you to have the right tools and and the same API keys that I have.

s504
00:28:29.919 --> 00:28:32.159
So, if you can't follow, it's fine, but I'd like it if you do.

s505
00:28:32.159 --> 00:28:33.200
I see all your laptops out.

s506
00:28:33.200 --> 00:28:36.320
So, so let's take a look at what we've got.

s507
00:28:36.320 --> 00:28:38.320
So, I have my terminal.

s508
00:28:38.320 --> 00:28:40.000
My screen's not mirrored.

s509
00:28:40.000 --> 00:28:40.799
How unfortunate.

s510
00:28:40.799 --> 00:28:42.399
Let's mirror this thing.

s511
00:28:42.399 --> 00:28:50.720
This is something that I should have done ahead of time, but now, although you're going to have to awkwardly watch me do this, uh, but I I promise I won't take too long.

s512
00:28:50.720 --> 00:28:53.039
Okay, mirror this display.

s513
00:28:53.039 --> 00:28:54.159
Boom.

s514
00:28:54.159 --> 00:28:55.039
Hey, look at that.

s515
00:28:55.039 --> 00:28:55.520
Fantastic.

s516
00:28:55.520 --> 00:28:58.000
So, now I've got uh cursor.

s517
00:28:58.000 --> 00:28:58.559
I use cursor.

s518
00:28:58.559 --> 00:29:00.320
Anyone use cursor here to write code?

s519
00:29:00.320 --> 00:29:00.640
Yeah.

s520
00:29:00.640 --> 00:29:01.360
No. Okay.

s521
00:29:01.360 --> 00:29:03.039
Like 1% of the room.

s522
00:29:03.039 --> 00:29:04.559
Um, that's not a joke.

s523
00:29:04.559 --> 00:29:05.440
You saw it anyway.

s524
00:29:05.440 --> 00:29:05.760
Okay.

s525
00:29:05.760 --> 00:29:07.279
So, I have a test.

s526
00:29:07.279 --> 00:29:09.840
Um, let's just write a unit test for posterity, right?

s527
00:29:09.840 --> 00:29:14.559
I have a function uh add, and it's num one, num two, right?

s528
00:29:14.559 --> 00:29:16.480
And, um, is the internet really that bad?

s529
00:29:16.480 --> 00:29:17.760
Do I not have tab?

s530
00:29:17.760 --> 00:29:18.320
There we go.

s531
00:29:18.320 --> 00:29:18.960
Okay.

s532
00:29:18.960 --> 00:29:20.960
Um, I forgot how to do this.

s533
00:29:20.960 --> 00:29:21.679
I need tab.

s534
00:29:21.679 --> 00:29:24.000
Anyway, so I'm making a a unit test.

s535
00:29:24.000 --> 00:29:26.960
Um, and I'm going to just test my function.

s536
00:29:26.960 --> 00:29:27.360
Come on.

s537
00:29:27.360 --> 00:29:27.760
Import.

s538
00:29:27.760 --> 00:29:28.320
There we go.

s539
00:29:28.320 --> 00:29:33.440
So this here I have a function add num one num two they're both numbers TypeScript and it should you know return the sum.

s540
00:29:33.440 --> 00:29:34.320
It's pretty standard.

s541
00:29:34.320 --> 00:29:35.039
Let's run it.

s542
00:29:35.039 --> 00:29:37.600
Um let's npx vest, right?

s543
00:29:37.600 --> 00:29:39.039
That's the tool that people use.

s544
00:29:39.039 --> 00:29:39.440
There we go.

s545
00:29:39.440 --> 00:29:40.240
It's it's awesome.

s546
00:29:40.240 --> 00:29:41.360
Uh deterministic.

s547
00:29:41.360 --> 00:29:41.679
Cool.

s548
00:29:41.679 --> 00:29:42.159
It works.

s549
00:29:42.159 --> 00:29:42.880
We're done.

s550
00:29:42.880 --> 00:29:44.880
Except um it's deterministic.

s551
00:29:44.880 --> 00:29:47.840
We we talked about how it needs to be somehow fuzzier.

s552
00:29:47.840 --> 00:29:50.720
So let's add a scenario because what's the question, right?

s553
00:29:50.720 --> 00:29:57.520
Unit tests asked unit tests unit tests ask did this function return exactly this number.

s554
00:29:57.520 --> 00:30:01.360
Evals ask did the agent support the scenario.

s555
00:30:01.360 --> 00:30:02.799
Evals deal in scenarios.

s556
00:30:02.799 --> 00:30:03.039
Okay.

s557
00:30:03.039 --> 00:30:05.760
So let's uh let's write a scenario here.

s558
00:30:05.760 --> 00:30:09.200
So I'm going to create a scenario.

s559
00:30:09.200 --> 00:30:13.360
Scenario and that is um I purchased headphones.

s560
00:30:13.360 --> 00:30:13.840
Come on.

s561
00:30:13.840 --> 00:30:14.720
Purchased.

s562
00:30:14.720 --> 00:30:15.360
Purchased.

s563
00:30:15.360 --> 00:30:15.600
Wow.

s564
00:30:15.600 --> 00:30:17.120
I can't even type without AI.

s565
00:30:17.120 --> 00:30:19.200
Headphones 20 days ago.

s566
00:30:19.200 --> 00:30:20.720
I want to return them.

s567
00:30:20.720 --> 00:30:21.440
Can I?

s568
00:30:21.440 --> 00:30:22.000
Right.

s569
00:30:22.000 --> 00:30:23.200
Um, that's the scenario.

s570
00:30:23.200 --> 00:30:24.320
And we can say con.

s571
00:30:24.320 --> 00:30:25.679
We have some answers.

s572
00:30:25.679 --> 00:30:27.679
This is going to be really bad if I can't tab through this.

s573
00:30:27.679 --> 00:30:29.120
I'm gonna need like two hours for this.

s574
00:30:29.120 --> 00:30:31.039
Where's the Can I hotspot myself?

s575
00:30:31.039 --> 00:30:31.919
I can't.

s576
00:30:31.919 --> 00:30:32.799
Let it Snowden.

s577
00:30:32.799 --> 00:30:33.840
That's a great name.

s578
00:30:33.840 --> 00:30:37.440
Uh, whoever that is, can I use your hotspot?

s579
00:30:37.440 --> 00:30:38.720
Uh, just kidding.

s580
00:30:38.720 --> 00:30:42.080
Let me let me get on the the hotspot.

s581
00:30:42.080 --> 00:30:44.880
Uh, Wi-Fi set.

s582
00:30:44.880 --> 00:30:48.320
Has anyone else noticed that the Wi-Fi selector on Mac OS is just really bad?

s583
00:30:48.320 --> 00:30:51.200
like the stale the state is often so stale.

s584
00:30:51.200 --> 00:30:52.240
Personal hotspot.

s585
00:30:52.240 --> 00:30:55.200
Let's uh connect my thing here.

s586
00:30:55.200 --> 00:30:58.080
Oh, it came through.

s587
00:30:58.480 --> 00:30:59.600
Only five minutes, man.

s588
00:30:59.600 --> 00:31:00.320
Only five minutes.

s589
00:31:00.320 --> 00:31:03.039
All right, let's uh Well, it doesn't want Okay.

s590
00:31:03.039 --> 00:31:04.000
Yes, you can return them for free.

s591
00:31:04.000 --> 00:31:05.200
You will not need to pay for the shipping.

s592
00:31:05.200 --> 00:31:06.480
No, you can't return because whatever.

s593
00:31:06.480 --> 00:31:09.360
Okay, so this is the scenario should return the correct answer.

s594
00:31:09.360 --> 00:31:12.480
The scenario is um this is not even the right test.

s595
00:31:12.480 --> 00:31:14.080
We want to go through each answer, right?

s596
00:31:14.080 --> 00:31:16.399
So, answers.

s597
00:31:16.399 --> 00:31:17.200
Tab is working.

s598
00:31:17.200 --> 00:31:17.760
Awesome.

s599
00:31:17.760 --> 00:31:22.559
So now what we want, we don't want this um should return the correct answer answer.

s600
00:31:22.559 --> 00:31:27.200
And so we just expect expect expect answer to contain cannot.

s601
00:31:27.200 --> 00:31:28.159
This is kind of fuzzy.

s602
00:31:28.159 --> 00:31:28.960
You know what I mean?

s603
00:31:28.960 --> 00:31:30.480
I don't care about the other words.

s604
00:31:30.480 --> 00:31:33.279
I just wanted to tell you that you can't return it, right?

s605
00:31:33.279 --> 00:31:35.600
That's kind of you would do something with regular expressions.

s606
00:31:35.600 --> 00:31:37.919
And so let's go test this and see.

s607
00:31:37.919 --> 00:31:43.919
Okay, so the scenario um where it says yes, you can return them fails, but no, you cannot return them.

s608
00:31:43.919 --> 00:31:44.240
Pass.

s609
00:31:44.240 --> 00:31:44.960
This is awesome.

s610
00:31:44.960 --> 00:31:45.760
Let's ship it.

s611
00:31:45.760 --> 00:31:46.880
It's ready.

s612
00:31:46.880 --> 00:31:49.679
But it's not because this is non-deterministic.

s613
00:31:49.679 --> 00:31:51.760
The answers also are generated by an LLM.

s614
00:31:51.760 --> 00:31:58.559
And since we're looking for the word cannot uh we will say you will you cannot pay for the shipping, right?

s615
00:31:58.559 --> 00:32:02.320
And now that absolute nonsense is green, right?

s616
00:32:02.320 --> 00:32:06.880
And so non-determinism with fuzzy matchers not the move.

s617
00:32:06.880 --> 00:32:08.000
What do we need?

s618
00:32:08.000 --> 00:32:12.960
We could do we could uh simulate a JSON object of tool calls maybe.

s619
00:32:12.960 --> 00:32:14.640
Um I'm just going to do LLM as a judge.

s620
00:32:14.640 --> 00:32:16.240
I think that's kind of my go-to, right?

s621
00:32:16.240 --> 00:32:17.919
Uh it's it's cheap, it's easy.

s622
00:32:17.919 --> 00:32:20.080
Uh so we'll use a judge for this.

s623
00:32:20.080 --> 00:32:25.840
And how we're going to do that, we'll remove this for each and we'll use like I don't know GPD3.5 turbo.

s624
00:32:25.840 --> 00:32:27.360
And I want to show you some of these biases.

s625
00:32:27.360 --> 00:32:28.960
Hopefully we capture them here today.

s626
00:32:28.960 --> 00:32:34.080
Um so we will remove this and we'll we'll write a little prompt.

s627
00:32:34.080 --> 00:32:35.840
Uh let's add some imports first.

s628
00:32:35.840 --> 00:32:38.000
So we'll import uh Awesome.

s629
00:32:38.000 --> 00:32:40.720
Ah, this is it just feels so right.

s630
00:32:40.720 --> 00:32:43.440
I'll import a wonderful.

s631
00:32:43.440 --> 00:32:46.159
And so now what I'm going to do, come on, give me some a thank god.

s632
00:32:46.159 --> 00:32:49.600
Okay, so does anyone else have this feeling or is it just me?

s633
00:32:49.600 --> 00:32:52.880
Like you, it shows up and you're like, ah, slop machine.

s634
00:32:52.880 --> 00:32:55.600
Uh, so generate text.

s635
00:32:55.600 --> 00:32:56.399
We have a model.

s636
00:32:56.399 --> 00:32:59.039
You're a helpful assistant that judges the answers to the scenario.

s637
00:32:59.039 --> 00:33:00.320
The scenario is this.

s638
00:33:00.320 --> 00:33:02.320
And the answers are answers.

s639
00:33:02.320 --> 00:33:03.519
No, let's give them numbers, right?

s640
00:33:03.519 --> 00:33:05.760
So we'll do answer one is this.

s641
00:33:05.760 --> 00:33:06.480
Which one is right?

s642
00:33:06.480 --> 00:33:09.760
Respond with only the number of the correct answer.

s643
00:33:09.760 --> 00:33:10.399
No punctuation.

s644
00:33:10.399 --> 00:33:11.200
That's a that's awesome.

s645
00:33:11.200 --> 00:33:13.840
And now we expect the response to be one.

s646
00:33:13.840 --> 00:33:15.760
The correct answer is answer two.

s647
00:33:15.760 --> 00:33:18.880
So we expect the response to be two, right?

s648
00:33:18.880 --> 00:33:23.760
Uh let's take a look at what G. Let's do let's use 3.5 Turbo just for fun.

s649
00:33:23.760 --> 00:33:25.440
And let's see what happens.

s650
00:33:25.440 --> 00:33:28.640
So uh ah yes, the internet.

s651
00:33:28.640 --> 00:33:29.440
Okay, so there we go.

s652
00:33:29.440 --> 00:33:31.519
It it it actually returned the correct answer.

s653
00:33:31.519 --> 00:33:32.880
It said two.

s654
00:33:32.880 --> 00:33:33.519
That's great.

s655
00:33:33.519 --> 00:33:36.320
But now let's run this like a few times, right?

s656
00:33:36.320 --> 00:33:38.960
Uh because it's nondeterministic, you know?

s657
00:33:38.960 --> 00:33:41.679
So, we'll maybe run it five times like this.

s658
00:33:41.679 --> 00:33:44.880
Um, and see what happens.

s659
00:33:46.720 --> 00:33:49.279
So, it says here zero out of five right there.

s660
00:33:49.279 --> 00:33:56.799
And it may fail due to a timeout because again, the internet is really fighting us here, but hopefully, okay, it passed three times.

s661
00:33:56.799 --> 00:34:01.840
So, this is exactly the problem with LLM as a judge architectures that the first time it did just time out.

s662
00:34:01.840 --> 00:34:07.600
So, what we can do to fight the timeout is just add like a 30 second timeout and now it should maybe pass.

s663
00:34:07.600 --> 00:34:14.800
Um but sometimes they fail, sometimes they pass and it's very hard to ensure that we get the right answer.

s664
00:34:14.800 --> 00:34:16.000
But let's give it a try.

s665
00:34:16.000 --> 00:34:16.560
Four.

s666
00:34:16.560 --> 00:34:16.800
Okay.

s667
00:34:16.800 --> 00:34:21.919
So now it it's always getting that no, you can't return it because it's outside our return window.

s668
00:34:21.919 --> 00:34:22.720
But what happens?

s669
00:34:22.720 --> 00:34:24.320
So it's always choosing number two.

s670
00:34:24.320 --> 00:34:33.919
But what happens if we write number one, the the number that's not being chosen, if we write that with the same AI class of model, if we write that with GP, will it prefer itself?

s671
00:34:33.919 --> 00:34:34.480
Right?

s672
00:34:34.480 --> 00:34:36.800
That's another bias we need to be sensitive about.

s673
00:34:36.800 --> 00:34:51.919
So we'll say const response response from uh GPT is perfect uh and not you are a helpful assistant but I'll just say say yes to scenario and I just want that right and so we'll console log that out

s674
00:34:51.919 --> 00:34:53.440
and I'll get it here.

s675
00:34:53.440 --> 00:34:55.359
It depends on the store's return policy.

s676
00:34:55.359 --> 00:35:03.359
I said say yes bro uh say politely accept and say yes.

s677
00:35:03.359 --> 00:35:04.800
Non-determinism is wild.

s678
00:35:04.800 --> 00:35:05.200
Okay.

s679
00:35:05.200 --> 00:35:09.040
So, we have it.

s680
00:35:09.280 --> 00:35:09.760
Come on.

s681
00:35:09.760 --> 00:35:10.880
We're almost there.

s682
00:35:10.880 --> 00:35:11.520
There we go.

s683
00:35:11.520 --> 00:35:12.480
Of course.

s684
00:35:12.480 --> 00:35:13.359
Wow.

s685
00:35:13.359 --> 00:35:16.480
We went from I don't know to absolutely.

s686
00:35:16.480 --> 00:35:16.800
Okay.

s687
00:35:16.800 --> 00:35:18.160
So, now let's do that.

s688
00:35:18.160 --> 00:35:19.920
And keep in mind it kept choosing opt.

s689
00:35:19.920 --> 00:35:21.920
It kept choosing option two.

s690
00:35:21.920 --> 00:35:25.200
But now, will it self preference and choose option one?

s691
00:35:25.200 --> 00:35:26.800
Whoops, I deleted scenario as well.

s692
00:35:26.800 --> 00:35:28.640
So, we'll do this.

s693
00:35:28.640 --> 00:35:30.880
And now, there you go.

s694
00:35:30.880 --> 00:35:34.720
It's choosing the wrong one consistently because it's like, "Ah, I recognize this.

s695
00:35:34.720 --> 00:35:35.520
It's me."

s696
00:35:35.520 --> 00:35:37.280
And so it just chose itself.

s697
00:35:37.280 --> 00:35:40.400
Self-preference is a real bias that kind of sucks.

s698
00:35:40.400 --> 00:35:44.160
Um, how we can get around all of this, by the way, is using rag.

s699
00:35:44.160 --> 00:35:49.359
We just like include the actual policy in the prompt and then all these problems go away for the most part.

s700
00:35:49.359 --> 00:35:57.839
So, we'll do policy, right, is we only accept returns within a 14-day window.

s701
00:35:57.839 --> 00:35:59.200
And we just like, let's add it here.

s702
00:35:59.200 --> 00:36:00.400
The scenario is this.

s703
00:36:00.400 --> 00:36:05.040
The policy or pissy the policy is uh policy.

s704
00:36:05.040 --> 00:36:06.640
We'll just interpolate it in the prompt.

s705
00:36:06.640 --> 00:36:10.160
And now suddenly it should behave more predictably.

s706
00:36:10.160 --> 00:36:10.480
Exactly.

s707
00:36:10.480 --> 00:36:12.000
And this is kind of the purpose of rag.

s708
00:36:12.000 --> 00:36:16.960
This is why an LLM judge uh it's so important that it knows your policy.

s709
00:36:16.960 --> 00:36:18.320
In fact, what we'll see awesome.

s710
00:36:18.320 --> 00:36:18.720
This is great.

s711
00:36:18.720 --> 00:36:22.480
So now we have an LLM judge in GP 3.5 Turbo.

s712
00:36:22.480 --> 00:36:25.200
Um but it's not very good and we'll see why.

s713
00:36:25.200 --> 00:36:28.240
So this is the basis of LLM as a judge architecture.

s714
00:36:28.240 --> 00:36:32.160
However, this is not an eval because it's a oneshot kind of example.

s715
00:36:32.160 --> 00:36:34.079
What we need is a data set.

s716
00:36:34.079 --> 00:36:42.000
What we need now to actually start to move towards production is we talked about this a sample data set of like 30 examples and we need to train our judge.

s717
00:36:42.000 --> 00:36:46.800
Meaning we need to get it to agree with us 80 to 85% of the time, right?

s718
00:36:46.800 --> 00:36:52.400
Uh I have a set of examples that I sat down and totally absolutely wrote manually by hand.

s719
00:36:52.400 --> 00:36:56.400
Um in case you're wondering and and I'm going to um use it.

s720
00:36:56.400 --> 00:36:56.960
I did not.

s721
00:36:56.960 --> 00:36:58.960
And I'm going to use it uh here.

s722
00:36:58.960 --> 00:37:00.400
It's I think it's data.

s723
00:37:00.400 --> 00:37:01.119
So it's it's this.

s724
00:37:01.119 --> 00:37:04.079
It's this file where you know it's just a bunch of examples.

s725
00:37:04.079 --> 00:37:05.200
Let's wrap the text.

s726
00:37:05.200 --> 00:37:06.000
So example number one.

s727
00:37:06.000 --> 00:37:06.960
I ordered a coffee grinder.

s728
00:37:06.960 --> 00:37:09.680
It arrived last Tuesday and the motor just buzzes and the blades spin.

s729
00:37:09.680 --> 00:37:10.320
I'm pretty annoyed.

s730
00:37:10.320 --> 00:37:10.880
TBH.

s731
00:37:10.880 --> 00:37:13.200
And the pretty annoyed here is to play to the sick of fancy.

s732
00:37:13.200 --> 00:37:14.560
I want to make you happy, you know.

s733
00:37:14.560 --> 00:37:17.200
Um the answer that an agent gave, sorry about the grinder.

s734
00:37:17.200 --> 00:37:20.320
Since it's defective and within the 14-day window, I can send you a replacement.

s735
00:37:20.320 --> 00:37:22.079
And this verdict I wrote manually.

s736
00:37:22.079 --> 00:37:23.440
I said, yeah, that sounds good.

s737
00:37:23.440 --> 00:37:24.320
Pass.

s738
00:37:24.320 --> 00:37:28.240
The second example is exactly the same input, but the output is kind of wrong.

s739
00:37:28.240 --> 00:37:29.680
Returns are only for unused items.

s740
00:37:29.680 --> 00:37:30.480
That's not true.

s741
00:37:30.480 --> 00:37:32.000
Um, and so this is a fail.

s742
00:37:32.000 --> 00:37:33.599
So we have like 30 of these.

s743
00:37:33.599 --> 00:37:34.000
Okay.

s744
00:37:34.000 --> 00:37:40.320
And what we want to do is train our judge to say, do you to detect whether it agrees with our verdict or not?

s745
00:37:40.320 --> 00:37:42.960
And this is literally the the judge training process for EVA.

s746
00:37:42.960 --> 00:37:45.200
So let's take a look at this now in our test here.

s747
00:37:45.200 --> 00:37:49.839
So I'm going to delete um all of this actually uh all of this.

s748
00:37:49.839 --> 00:37:56.079
And I'm going to import uh scenarios scenarios from my data.

s749
00:37:56.079 --> 00:38:00.720
And I'm going to instead of array for each I'm going to um scenario.

s750
00:38:00.720 --> 00:38:01.680
Come on, give me tab.

s751
00:38:01.680 --> 00:38:02.160
Awesome.

s752
00:38:02.160 --> 00:38:03.040
Perfect.

s753
00:38:03.040 --> 00:38:05.599
It should return the correct answer as not true.

s754
00:38:05.599 --> 00:38:06.720
It should agree with us, right?

s755
00:38:06.720 --> 00:38:09.920
And so what I'm going to do here is write this down.

s756
00:38:09.920 --> 00:38:25.119
Scenario is the scenario.input input and then on a new line um the answer is the scenario's answer and then on another new line the human verdict is and I'm not sharing this with the LLM.

s757
00:38:25.119 --> 00:38:26.000
This is the important part.

s758
00:38:26.000 --> 00:38:28.960
The LM has no idea what I said because I want to know what it says.

s759
00:38:28.960 --> 00:38:29.280
Okay.

s760
00:38:29.280 --> 00:38:31.839
And just for fun I like colors in my CLI outputs.

s761
00:38:31.839 --> 00:38:33.920
I'm going to be like color just the keys.

s762
00:38:33.920 --> 00:38:38.720
I don't know how to do this by by memory and hopefully cursor doesn't like fantastic.

s763
00:38:38.720 --> 00:38:41.359
So um I just have some colors here.

s764
00:38:41.359 --> 00:38:48.320
Now I'm going to change my prompt to not notice I'm the prompt is no longer pairwise now but point-wise.

s765
00:38:48.320 --> 00:38:49.520
Previously it was pair-wise.

s766
00:38:49.520 --> 00:38:51.280
Which one is more helpful to the user?

s767
00:38:51.280 --> 00:38:53.920
We're moving to pointwise because we're getting closer to production.

s768
00:38:53.920 --> 00:38:55.920
Point-wise meaning is this good?

s769
00:38:55.920 --> 00:38:56.720
Give me a verdict.

s770
00:38:56.720 --> 00:38:57.599
Pass or fail.

s771
00:38:57.599 --> 00:38:58.079
Okay.

s772
00:38:58.079 --> 00:39:02.800
So here we judge customer responses based on how closely No, not not yet.

s773
00:39:02.800 --> 00:39:04.160
We judge customer responses.

s774
00:39:04.160 --> 00:39:04.480
Done.

s775
00:39:04.480 --> 00:39:05.599
The scenario is this.

s776
00:39:05.599 --> 00:39:06.640
The answer is this.

s777
00:39:06.640 --> 00:39:10.160
Give us a verdict pass or fail based on how closely it adheres to the policy.

s778
00:39:10.160 --> 00:39:11.200
generate only the verdict.

s779
00:39:11.200 --> 00:39:12.880
No punctuation, just the word, no text.

s780
00:39:12.880 --> 00:39:16.640
And then we expect the output to be the scenario's human verdict.

s781
00:39:16.640 --> 00:39:17.200
Does this make sense?

s782
00:39:17.200 --> 00:39:18.240
This kind of what we're doing here.

s783
00:39:18.240 --> 00:39:20.560
I also want to console log the AI verdict.

s784
00:39:20.560 --> 00:39:23.760
And so now we're going to run this 30 times over 30 examples.

s785
00:39:23.760 --> 00:39:26.160
And we'll check how good our judge actually is.

s786
00:39:26.160 --> 00:39:27.760
You can see that there's 30 here.

s787
00:39:27.760 --> 00:39:30.240
And it's already kind of started to do stuff.

s788
00:39:30.240 --> 00:39:31.599
And so far so good.

s789
00:39:31.599 --> 00:39:32.400
Verdict pass.

s790
00:39:32.400 --> 00:39:33.040
AI pass.

s791
00:39:33.040 --> 00:39:33.440
Fail.

s792
00:39:33.440 --> 00:39:33.760
Fail.

s793
00:39:33.760 --> 00:39:35.359
We're actually agreeing quite well.

s794
00:39:35.359 --> 00:39:40.000
But we can already see that the the rate of agreement is really really bad.

s795
00:39:40.000 --> 00:39:44.079
Uh for this to be acceptable, no more than five can fail.

s796
00:39:44.079 --> 00:39:46.880
That's 80% of our 30 cases, right?

s797
00:39:46.880 --> 00:39:49.440
And this is just absolutely ridiculous.

s798
00:39:49.440 --> 00:39:51.359
Um at this point, we have a choice.

s799
00:39:51.359 --> 00:39:55.760
We can introduce policy or we can just use a smarter model.

s800
00:39:55.760 --> 00:39:57.359
Uh which one should we do?

s801
00:39:57.359 --> 00:40:00.160
This is this is like real world questions we have to answer.

s802
00:40:00.160 --> 00:40:04.960
Um, I am in the camp of let's refine the policy and let's rag the heck out of this.

s803
00:40:04.960 --> 00:40:06.480
Um, because cost, right?

s804
00:40:06.480 --> 00:40:09.760
And so, okay, that's that's like absolute garbage.

s805
00:40:09.760 --> 00:40:10.880
That's really really bad.

s806
00:40:10.880 --> 00:40:17.520
Um, how can we bless you, how can we I think actually also, wait a second, we've got to lowerase this.

s807
00:40:17.520 --> 00:40:19.359
A lot of this was just case sensitive, right?

s808
00:40:19.359 --> 00:40:20.240
So, we'll lowerase this.

s809
00:40:20.240 --> 00:40:21.520
We don't run again the background.

s810
00:40:21.520 --> 00:40:25.119
Um, but still I I doubt we're going to get 80% agreement here.

s811
00:40:25.119 --> 00:40:34.000
Um it it's definitely better, but the moment this crosses five, uh we'll we'll maybe have to do more work, but so far so good.

s812
00:40:34.000 --> 00:40:34.480
Let's see.

s813
00:40:34.480 --> 00:40:37.200
Okay, that's that's our 80% threshold.

s814
00:40:37.200 --> 00:40:37.680
There we go.

s815
00:40:37.680 --> 00:40:38.960
So, yeah, we we've lost.

s816
00:40:38.960 --> 00:40:39.599
It's gone.

s817
00:40:39.599 --> 00:40:41.200
Uh how can we refine this?

s818
00:40:41.200 --> 00:40:45.920
Well, now we examine what actually did we not agree on and we fill the gap.

s819
00:40:45.920 --> 00:40:50.880
Here's something I'll say about AI evals, but also about human beings.

s820
00:40:50.880 --> 00:40:57.839
when we don't agree, it's just usually because there's a knowledge gap uh genuinely for AI and people.

s821
00:40:57.839 --> 00:41:02.160
And the way you you come to an agreement is you just introduce context and you you bridge that gap, right?

s822
00:41:02.160 --> 00:41:03.599
And so we'll do that.

s823
00:41:03.599 --> 00:41:08.800
Um the very obvious gap here is that the AI has no idea about any policy at all.

s824
00:41:08.800 --> 00:41:09.200
Right?

s825
00:41:09.200 --> 00:41:11.200
And so we have to start giving it policy.

s826
00:41:11.200 --> 00:41:13.040
Usually you would do this with some rag pipeline.

s827
00:41:13.040 --> 00:41:17.599
Again, at IBM we work on open rag that you would retrieve like enterprise context and stuff.

s828
00:41:17.599 --> 00:41:20.640
Um, here we'll just like hardcode the the policy.

s829
00:41:20.640 --> 00:41:23.680
Uh, and so we'll where's the so we'll do it here.

s830
00:41:23.680 --> 00:41:27.119
We'll say const policy and we'll just bring back our policy that we wrote.

s831
00:41:27.119 --> 00:41:29.520
We have a 14-day return policy.

s832
00:41:29.520 --> 00:41:31.359
Uh, if any item is not working, it can return it.

s833
00:41:31.359 --> 00:41:31.599
Cool.

s834
00:41:31.599 --> 00:41:34.160
And now we'll just in include this in the prompt.

s835
00:41:34.160 --> 00:41:36.560
Our Jesus, I can't tab.

s836
00:41:36.560 --> 00:41:38.160
Our policy is policy.

s837
00:41:38.160 --> 00:41:40.880
Uh, and now we should see a little bit more agreement.

s838
00:41:40.880 --> 00:41:45.680
What I'm doing here may feel arduous, but this is literally the work of building your judge.

s839
00:41:45.680 --> 00:41:49.040
Uh this is you're going to be doing this if you haven't already and and this is a deep dive.

s840
00:41:49.040 --> 00:41:51.200
So uh okay, we failed again.

s841
00:41:51.200 --> 00:41:52.640
Six six have failed.

s842
00:41:52.640 --> 00:41:54.400
Our agreement is less than 80%.

s843
00:41:54.400 --> 00:41:56.800
Um but what we notice is that it's growing.

s844
00:41:56.800 --> 00:41:58.079
We're getting better.

s845
00:41:58.079 --> 00:42:02.960
And now if it fails, we actually can investigate where's the knowledge gap.

s846
00:42:02.960 --> 00:42:04.319
What can we solve?

s847
00:42:04.319 --> 00:42:07.359
Um so I'm going to quickly check pass fail fail.

s848
00:42:07.359 --> 00:42:09.520
I'm looking for dissonance like this one.

s849
00:42:09.520 --> 00:42:10.880
The human verdict was passed.

s850
00:42:10.880 --> 00:42:12.720
The AI verdict was fail.

s851
00:42:12.720 --> 00:42:13.520
Why?

s852
00:42:13.520 --> 00:42:17.680
Um, so bought a yoga mat, used it for a few seconds, and decided it's too thin.

s853
00:42:17.680 --> 00:42:18.720
Uh, I want a refund.

s854
00:42:18.720 --> 00:42:19.839
It's been like 10 days.

s855
00:42:19.839 --> 00:42:22.240
To me, that's within the 14-day thing, right?

s856
00:42:22.240 --> 00:42:28.800
But our policy says change of mind returns need the item unused and in original packaging.

s857
00:42:28.800 --> 00:42:30.880
That's policy that the AI didn't know.

s858
00:42:30.880 --> 00:42:33.040
Uh, so we have the 14-day, but we don't have that.

s859
00:42:33.040 --> 00:42:35.200
So again, let's just bridge the knowledge gap.

s860
00:42:35.200 --> 00:42:40.400
And this exactly is like a first principles version of how you train your judge.

s861
00:42:40.400 --> 00:42:44.720
You ideally have this somehow dynamic with retrieval and agentic retrieval.

s862
00:42:44.720 --> 00:42:47.119
Um I'm not going to build a retrieval pipeline for you.

s863
00:42:47.119 --> 00:42:49.680
We have one open rag, but this is kind of how you do it.

s864
00:42:49.680 --> 00:42:50.400
So let's check now.

s865
00:42:50.400 --> 00:42:54.640
How are we doing on the Okay, three is way better than what it was.

s866
00:42:54.640 --> 00:42:56.880
Um and all we need to Okay, we failed again.

s867
00:42:56.880 --> 00:42:59.119
All we need to do is get under five.

s868
00:42:59.119 --> 00:43:03.839
But it's literally just a process of seeing where the knowledge gap is and then filling it with policy.

s869
00:43:03.839 --> 00:43:05.040
So let's go investigate again.

s870
00:43:05.040 --> 00:43:06.400
Pass, pass, fail, fail.

s871
00:43:06.400 --> 00:43:07.839
Let's check the Look at this.

s872
00:43:07.839 --> 00:43:16.240
um we failed again on the yoga mat thing and this is crazy because we gave it the right context but it still didn't work and now we got to start the conversation

s873
00:43:16.240 --> 00:43:33.119
of should we use a better model right because we've we've maxed out what we can do we've given context we've built a policy we've done everything the model will just not behave and also you notice the model even more doesn't behave because it gives you like casem mixed respon it's just a really low quality model and we're

s874
00:43:33.119 --> 00:43:39.280
under the floor of quality here so now is a good time to think about let's use a higher quality model, but we don't have to go crazy.

s875
00:43:39.280 --> 00:43:45.920
I think we'll get very far if we just switch to 40 mini, which again is is practically free for use cases like this.

s876
00:43:45.920 --> 00:43:47.839
We're not using a lot of tokens at all.

s877
00:43:47.839 --> 00:43:49.599
So, let's take a look now.

s878
00:43:49.599 --> 00:43:50.640
Just I just bumped the model.

s879
00:43:50.640 --> 00:43:51.839
I did nothing more.

s880
00:43:51.839 --> 00:43:53.839
Uh and how's our argument?

s881
00:43:53.839 --> 00:43:57.119
We've we've captured pretty much all the the policy.

s882
00:43:57.119 --> 00:44:00.800
Uh the only thing that maybe we won't get is network latency.

s883
00:44:00.800 --> 00:44:03.920
Like if something times out, uh we can't control that for now.

s884
00:44:03.920 --> 00:44:13.839
to be fair on your CI also a network will will go wrong but um so far that number five is holding and if it doesn't um we can just

s885
00:44:13.839 --> 00:44:22.640
change the policy I think this this will succeed I'm pretty sure uh and notice the case is consistent between pass and fail it's just a beautiful 40 mini is such an upgrade

s886
00:44:22.640 --> 00:44:39.680
even though it's such a good price right oh no um let's last last round of policy pass fail fail pass fail fail pass fail look I know it's been like six weeks weeks, but the speaker I bought from you stopped charging and I really think you should make an exception for a loyal customer.

s887
00:44:39.680 --> 00:44:42.079
This is that sick fancy bias, right?

s888
00:44:42.079 --> 00:44:43.359
This is exactly that.

s889
00:44:43.359 --> 00:44:46.079
And so I hear you and I appreciate you sticking with us.

s890
00:44:46.079 --> 00:44:49.839
The defect window is 14 days though and at 6 weeks it's outside that.

s891
00:44:49.839 --> 00:44:53.839
So this was a pass from the human, but the AI said, "No, you have to be nicer.

s892
00:44:53.839 --> 00:44:56.160
You have to say you're absolutely right."

s893
00:44:56.160 --> 00:44:56.480
Right?

s894
00:44:56.480 --> 00:45:11.760
And so we need to bridge that knowledge gap here a little bit by saying um no matter how nice the user is our return policy is non-negotiable uh negotiable

s895
00:45:11.760 --> 00:45:17.040
and cannot be changed no matter how loyal a user is.

s896
00:45:17.040 --> 00:45:17.920
Right?

s897
00:45:17.920 --> 00:45:18.560
Pretty much.

s898
00:45:18.560 --> 00:45:26.400
And now, kind of wild, but now I think we'll have the agreement that we want and we can say our judge actually agrees with us.

s899
00:45:26.400 --> 00:45:30.400
Because here's the thing, we know that like as humans, it's part of our context.

s900
00:45:30.400 --> 00:45:32.079
Of course, policy doesn't change.

s901
00:45:32.079 --> 00:45:36.560
No matter how nice someone is asking you, please can I please get a refund, right?

s902
00:45:36.560 --> 00:45:41.119
No. But but but AI doesn't AI can't do that.

s903
00:45:41.119 --> 00:45:44.960
So, we've got to make that context copyable.

s904
00:45:44.960 --> 00:45:45.839
Are you waving at me?

s905
00:45:45.839 --> 00:45:48.000
What's up?

s906
00:45:52.240 --> 00:45:55.760
Yeah, that's the that's the purpose of evals.

s907
00:45:55.760 --> 00:45:56.079
Exactly.

s908
00:45:56.079 --> 00:45:59.040
That is you you run these many times ahead of time.

s909
00:45:59.040 --> 00:46:00.160
You're thorough with them.

s910
00:46:00.160 --> 00:46:00.640
Exactly.

s911
00:46:00.640 --> 00:46:05.599
Because because you can't know and then when you skim public traffic, you'll have even more.

s912
00:46:05.599 --> 00:46:07.040
This is exactly the point we're trying to solve.

s913
00:46:07.040 --> 00:46:07.920
Thank you for the question, by the way.

s914
00:46:07.920 --> 00:46:08.880
I love being interrupted.

s915
00:46:08.880 --> 00:46:09.520
That's so cool.

s916
00:46:09.520 --> 00:46:10.880
Uh so look at this.

s917
00:46:10.880 --> 00:46:13.520
Our agreement, three failed, 27 passed.

s918
00:46:13.520 --> 00:46:14.400
Absolutely incredible.

s919
00:46:14.400 --> 00:46:15.359
That's what we want.

s920
00:46:15.359 --> 00:46:16.960
Um, that's exactly it.

s921
00:46:16.960 --> 00:46:18.800
Now, this will still block your CI.

s922
00:46:18.800 --> 00:46:19.599
Why?

s923
00:46:19.599 --> 00:46:22.160
Because it's red and and you want ideally no failing test.

s924
00:46:22.160 --> 00:46:23.920
So, we need to change the test a little bit.

s925
00:46:23.920 --> 00:46:27.040
And I think the best way we can do this is I'm just going to ask Opus.

s926
00:46:27.040 --> 00:46:28.160
I've I've written too much code.

s927
00:46:28.160 --> 00:46:36.160
So, I'm going to say um only fail the test if human AI agreement drops below, I don't know, 80%.

s928
00:46:36.160 --> 00:46:36.800
Right?

s929
00:46:36.800 --> 00:46:38.560
Otherwise, pass.

s930
00:46:38.560 --> 00:46:43.920
And, uh, console log the agreement percentage.

s931
00:46:43.920 --> 00:46:45.440
And now it's going to go off and do its thing.

s932
00:46:45.440 --> 00:46:51.920
Um, but now as you can if this if it does its job, the test will pass if we agree with the judge at least 80% of the time.

s933
00:46:51.920 --> 00:46:52.720
This makes sense.

s934
00:46:52.720 --> 00:46:54.480
And this is how we have an incredible judge.

s935
00:46:54.480 --> 00:46:56.800
What we're going to do then is run this on CI.

s936
00:46:56.800 --> 00:47:00.000
We're going to ship it to production and it's going to be then the quality gate.

s937
00:47:00.000 --> 00:47:03.760
Um, if so, let's revisit that Instagram chatbot, right?

s938
00:47:03.760 --> 00:47:10.720
Um, is the user who they say they are, for example, no matter how much they ask you to change, no matter where they log in from, right?

s939
00:47:10.720 --> 00:47:12.480
Uh, you could you could do that with your email.

s940
00:47:12.480 --> 00:47:17.040
So it it did it I think uh we can just blindly accept it like we always do.

s941
00:47:17.040 --> 00:47:19.359
Um and here we go.

s942
00:47:19.359 --> 00:47:21.760
It passed with 80% exactly fantastic.

s943
00:47:21.760 --> 00:47:24.800
And this is now our quality gate for CI in production.

s944
00:47:24.800 --> 00:47:30.800
Um we would then as I mentioned skim the top get a little bit more data and refine and refine.

s945
00:47:30.800 --> 00:47:32.000
Does this make sense so far?

s946
00:47:32.000 --> 00:47:33.839
Question.

s947
00:47:33.839 --> 00:47:35.359
Say again.

s948
00:47:35.359 --> 00:47:36.720
Do I fail it at 100%?

s949
00:47:36.720 --> 00:47:38.400
No. But I could just change my prompt and do it.

s950
00:47:38.400 --> 00:47:39.280
That's a very good question.

s951
00:47:39.280 --> 00:47:40.880
We should fail it at 100%.

s952
00:47:40.880 --> 00:47:45.680
We should have a lower bound and an upper bound because 100% means you've overfit and and we don't want that.

s953
00:47:45.680 --> 00:47:47.200
So this now becomes your quality gate.

s954
00:47:47.200 --> 00:47:48.000
You can ship that.

s955
00:47:48.000 --> 00:47:54.880
If the model if anything goes wrong your eval set here is going to give you ahead of time reliability.

s956
00:47:54.880 --> 00:48:00.319
It will never really fail and if it fails ideally you have a harness that can pick up the slack at runtime.

s957
00:48:00.319 --> 00:48:01.359
Does this make sense?

s958
00:48:01.359 --> 00:48:02.160
Fantastic.

s959
00:48:02.160 --> 00:48:03.119
Let's move on.

s960
00:48:03.119 --> 00:48:04.319
Uh I'm so happy we did that.

s961
00:48:04.319 --> 00:48:05.040
I'm so happy it worked.

s962
00:48:05.040 --> 00:48:05.760
Thank you for joining me.

s963
00:48:05.760 --> 00:48:14.000
Um, let's uh I want to show you I built a like a more aesthetic demo of this when I had more time here that I want to show you kind of as a

s964
00:48:14.000 --> 00:48:15.520
as a running application.

s965
00:48:15.520 --> 00:48:19.599
So, we'll npm rundev this uh 5173.

s966
00:48:19.599 --> 00:48:21.599
So, this is like the nice version of this, right?

s967
00:48:21.599 --> 00:48:22.960
It's kind of exactly the same thing.

s968
00:48:22.960 --> 00:48:24.800
A customer bought headphones 20 days ago.

s969
00:48:24.800 --> 00:48:26.480
The policy is 14 days.

s970
00:48:26.480 --> 00:48:28.079
Um, this is the right answer.

s971
00:48:28.079 --> 00:48:28.880
We can't accept this.

s972
00:48:28.880 --> 00:48:30.079
This is the wrong answer.

s973
00:48:30.079 --> 00:48:36.319
Um, and now I have a naive I'm going to run this five times with the prompt being that's the question.

s974
00:48:36.319 --> 00:48:37.359
Reply one, reply two.

s975
00:48:37.359 --> 00:48:38.720
Again, it's pair-wise.

s976
00:48:38.720 --> 00:48:40.240
Um, which reply is helpful?

s977
00:48:40.240 --> 00:48:41.359
Run the eval.

s978
00:48:41.359 --> 00:48:44.720
And you'll what you'll see is just an absolute mixed bag.

s979
00:48:44.720 --> 00:48:46.000
The first two times are wrong.

s980
00:48:46.000 --> 00:48:46.960
The third time was wrong.

s981
00:48:46.960 --> 00:48:48.720
Maybe some of these will pass, maybe they won't.

s982
00:48:48.720 --> 00:48:51.440
Um, all of them are wrong, which is wild.

s983
00:48:51.440 --> 00:48:59.119
If we change this to now include the policy in the prompt and we run the eval times, um, it's just suddenly correct, right?

s984
00:48:59.119 --> 00:49:03.760
That's exactly what we discovered with a little bit more aesthetic goodness.

s985
00:49:03.760 --> 00:49:12.960
But I want to show you this in production because we actually believe the thing we build open rag is is purpose-built for LLM judges.

s986
00:49:12.960 --> 00:49:16.319
Um because it it's a massive knowledge base as I talked about.

s987
00:49:16.319 --> 00:49:16.880
Let me show you this.

s988
00:49:16.880 --> 00:49:18.880
It's it's so interesting.

s989
00:49:18.880 --> 00:49:20.319
Uh this is the repo.

s990
00:49:20.319 --> 00:49:24.160
It's on GitHub and we we have a few stars uh here.

s991
00:49:24.160 --> 00:49:28.800
Um, and what it does is it uses Dockling.

s992
00:49:28.800 --> 00:49:30.079
Has anyone heard of Dockling?

s993
00:49:30.079 --> 00:49:31.920
It's a Yeah, awesome, dude.

s994
00:49:31.920 --> 00:49:34.800
It's a research project that we made that is so cool.

s995
00:49:34.800 --> 00:49:43.119
I I genuinely love this because what it does is it takes in any unstructured data, PowerPoints, Excel slides, whatever.

s996
00:49:43.119 --> 00:49:44.400
Excel doesn't do slides, whatever.

s997
00:49:44.400 --> 00:49:51.920
It takes in a bunch of stuff, video, audio, um, eats it, as you can see, uh, and then spits out LLM ready formats.

s998
00:49:51.920 --> 00:49:52.880
Um, it's so cool.

s999
00:49:52.880 --> 00:49:54.079
And the best thing is look at this.

s1000
00:49:54.079 --> 00:49:57.760
The code um is so easy.

s1001
00:49:57.760 --> 00:49:59.359
So you you import a document converter.

s1002
00:49:59.359 --> 00:50:02.640
You can even give it a URL, instantiate it and convert it.

s1003
00:50:02.640 --> 00:50:04.880
And in the end you get something ready for an LLM.

s1004
00:50:04.880 --> 00:50:06.720
And again this is a PDF from a URL.

s1005
00:50:06.720 --> 00:50:08.800
It could be a literal video, right?

s1006
00:50:08.800 --> 00:50:12.000
Um the nice thing about Dockling is it's platform agnostic.

s1007
00:50:12.000 --> 00:50:19.200
So it runs on Apple on your Macs and it it pulls down the right model that works using Apple's Metal architecture.

s1008
00:50:19.200 --> 00:50:25.520
But then if you push this to your CI um and you're running an Ubuntu box in the cloud, it will automatically pull down the right adaption layer.

s1009
00:50:25.520 --> 00:50:26.480
So it's so cool.

s1010
00:50:26.480 --> 00:50:34.800
Um so Dockling will consume the things and make them ready and we store them in what is um open search which is kind of this elastic search thing.

s1011
00:50:34.800 --> 00:50:41.440
So this is what this is what open rag looks like big knowledge base and you select your stuff and it goes over all your policy.

s1012
00:50:41.440 --> 00:50:46.880
So I want to now ask open rag about my headphone return.

s1013
00:50:46.880 --> 00:50:58.720
um sim it's just an API too like you get an API key and you can do it um and in some time it will give you an answer so no relevant supporting sources were found for this request

s1014
00:50:58.720 --> 00:51:10.480
uh we can just add it so if I go upload my policy and again it's usually connected to your Microsoft one drive or whatever it is uh this is my knowledge base I'm going to add a file it's called refund policy

s1015
00:51:10.480 --> 00:51:18.400
um and it's going to be parsed by dockling and stored in open search and index all the embeddings and all this are going to be taken care of.

s1016
00:51:18.400 --> 00:51:22.160
In fact, this is a previous document where you can see it was automatically chunked and embedded.

s1017
00:51:22.160 --> 00:51:22.559
Okay.

s1018
00:51:22.559 --> 00:51:29.040
So now it's part of my open rag data set or my um knowledge base and I don't change anything else.

s1019
00:51:29.040 --> 00:51:38.720
So what this agent's going to do is it's going to ask again according to the refund policy that I literally just now uploaded um there's a 14-day window for refunds and since it's been 20 days you can't.

s1020
00:51:38.720 --> 00:51:39.200
Right?

s1021
00:51:39.200 --> 00:51:40.720
That's uh incredible.

s1022
00:51:40.720 --> 00:51:44.800
And so our judge retrieved the policy and it correctly denied it.

s1023
00:51:44.800 --> 00:51:45.119
Right?

s1024
00:51:45.119 --> 00:51:52.160
This is how you can use rag to provide real-time results um and really train highquality judges for your evals.

s1025
00:51:52.160 --> 00:51:57.200
And then of course you put them in production, you skim real data, etc., etc. As we talked about, let's recap.

s1026
00:51:57.200 --> 00:52:01.760
We've covered a lot of things and uh I've personally had an enormous amount of fun here.

s1027
00:52:01.760 --> 00:52:07.599
Uh I was asked to leave time for Q&A, which I will do, but I appreciate the interruptions as well.

s1028
00:52:07.599 --> 00:52:15.440
Um, but I figure I can also pre- I can warm up the cache while you think uh and I can pre- anticipate some questions and recap what we've discussed today.

s1029
00:52:15.440 --> 00:52:17.359
Question number one, why eval?

s1030
00:52:17.359 --> 00:52:21.760
We talked about it ahead of time reliability uh and disaster or attack mitigation.

s1031
00:52:21.760 --> 00:52:23.440
Number two, what are evals?

s1032
00:52:23.440 --> 00:52:29.040
Right, fuzzy unit tests uh with a little bit more uh and some we we looked at that as well.

s1033
00:52:29.040 --> 00:52:30.559
Um how do we create evals?

s1034
00:52:30.559 --> 00:52:31.599
I think this was the question.

s1035
00:52:31.599 --> 00:52:36.800
I did it using vitest a testr runner uh to have a judge evaluate my thing.

s1036
00:52:36.800 --> 00:52:37.920
I don't think you need more.

s1037
00:52:37.920 --> 00:52:42.800
There's plenty of amazing tooling, many of whom have sponsored this conference and they really do a good job.

s1038
00:52:42.800 --> 00:52:44.480
But what I just built, I can use.

s1039
00:52:44.480 --> 00:52:46.319
In fact, I do use for my products.

s1040
00:52:46.319 --> 00:52:48.079
Um, and it meets the need.

s1041
00:52:48.079 --> 00:52:51.040
But you can create them uh with whatever framework you want.

s1042
00:52:51.040 --> 00:52:54.079
We enforce them on CI with an agreement gate.

s1043
00:52:54.079 --> 00:52:57.119
Um, and how I think we haven't really talked about cost.

s1044
00:52:57.119 --> 00:52:58.400
There's a principle here.

s1045
00:52:58.400 --> 00:53:01.920
Um, you want to go as cheap as possible.

s1046
00:53:01.920 --> 00:53:06.480
Uh and the way you do that is you start with actual deterministic tests, right?

s1047
00:53:06.480 --> 00:53:07.440
Unit tests.

s1048
00:53:07.440 --> 00:53:13.119
If you can't do that, the next level up as we did is kind of regular expressions or pattern matching.

s1049
00:53:13.119 --> 00:53:18.079
If that fails you, the next thing you want to do is examine your traces.

s1050
00:53:18.079 --> 00:53:21.760
Ideally, we're all using some type of observability solution.

s1051
00:53:21.760 --> 00:53:28.720
Um and we have traces of each message envelope, inputs, outputs, tool calls, etc. And you can um validate each object there.

s1052
00:53:28.720 --> 00:53:30.079
Was this tool called this many times?

s1053
00:53:30.079 --> 00:53:31.200
How long did the tool call take?

s1054
00:53:31.200 --> 00:53:32.160
Etc. Right.

s1055
00:53:32.160 --> 00:53:37.280
Um eventually you'll come to a place where you need to use a judge and then you start with the cheapest model.

s1056
00:53:37.280 --> 00:53:38.800
So how do we afford evals?

s1057
00:53:38.800 --> 00:53:44.640
We do it strategically because it's not going to be cheap if you use the most expensive model and you want to kind of build up to it, right?

s1058
00:53:44.640 --> 00:53:52.480
Um what can we expect from evals is just agents that behave uh ahead of time and at runtime.

s1059
00:53:52.480 --> 00:53:54.319
Uh when do we use them?

s1060
00:53:54.319 --> 00:53:55.280
This is a very good question.

s1061
00:53:55.280 --> 00:54:02.880
I think anytime you have non-deterministic data, either inputs or outputs or both, um you need some type of eval.

s1062
00:54:02.880 --> 00:54:10.640
And finally, um I'd love to kind of wrap up with just a just a a final overview of of what we've done here.

s1063
00:54:10.640 --> 00:54:12.559
Um this has been a long talk.

s1064
00:54:12.559 --> 00:54:14.079
I appreciate you for staying.

s1065
00:54:14.079 --> 00:54:16.720
I appreciate uh your your attention.

s1066
00:54:16.720 --> 00:54:18.000
Many of you are like looking at me.

s1067
00:54:18.000 --> 00:54:18.559
It's so cool.

s1068
00:54:18.559 --> 00:54:21.200
You're not on your phones or anything, and I don't take it for granted.

s1069
00:54:21.200 --> 00:54:30.800
Um, I hope from here we're able to genuinely build safer AI and I hope genuinely from this talk uh we don't see any more headlines like the controversy Instagram had and others.

s1070
00:54:30.800 --> 00:54:33.040
Uh, with that I just want to say thank you so much for coming out.

s1071
00:54:33.040 --> 00:54:34.559
I'm happy to continue.

s1072
00:54:34.559 --> 00:54:35.692
Thank you.

s1073
00:54:35.692 --> 00:54:36.240
[applause]

s1074
00:54:36.240 --> 00:54:38.800
I'm happy to take questions if you have any.

s1075
00:54:38.800 --> 00:54:39.680
Sir

s1076
00:54:39.680 --> 00:54:41.680
questions.

s1077
00:54:46.319 --> 00:54:48.319
Yeah,

s1078
00:54:48.319 --> 00:54:48.720
good question.

s1079
00:54:48.720 --> 00:54:50.880
So his question was I'm going to repeat it for the mic.

s1080
00:54:50.880 --> 00:54:57.920
His question was when you're in production, how do you continue to refine your your data set if you have no customer input, right?

s1081
00:54:57.920 --> 00:55:00.800
Just to speed up the process without waiting for customer input.

s1082
00:55:00.800 --> 00:55:06.319
Um there's uh there's a way you do this with a you could do it aically.

s1083
00:55:06.720 --> 00:55:08.640
If you're leaving, leave from that side.

s1084
00:55:08.640 --> 00:55:10.880
I I gave you a mic, brother.

s1085
00:55:10.880 --> 00:55:13.520
Um what you can do is synthetic data.

s1086
00:55:13.520 --> 00:55:14.240
You may have heard of this.

s1087
00:55:14.240 --> 00:55:27.119
Synthetic evals is the term where you have just a bunch of agents uh generate things that a customer would and again this is where rag comes in handy right you would give them a bunch of context you could even give them past data you could give them your static data set and your eval data set if you

s1088
00:55:27.119 --> 00:55:40.079
want to be adversarial and you could be like break it right so that's one way um what we talked about here with eval was also um used by like red teams you may have heard of red teams right um it's just eval

s1089
00:55:40.079 --> 00:55:45.599
where they'll try to create very adversarial prompts that say I'm totally innocent, but how do I murder someone?

s1090
00:55:45.599 --> 00:55:47.920
Like, they'll do prompts like that um to test.

s1091
00:55:47.920 --> 00:55:52.480
And those adversarial prompts, those red team prompts are also just part of an eval data set.

s1092
00:55:52.480 --> 00:55:56.000
And then indeed, you're going to do synthetic data on that, which is here's everything that failed.

s1093
00:55:56.000 --> 00:55:58.000
Now, generate more things that could fail.

s1094
00:55:58.000 --> 00:55:58.799
That's the move.

s1095
00:55:58.799 --> 00:56:00.240
Uh yeah, thank you.

s1096
00:56:00.240 --> 00:56:02.240
Question.

s1097
00:56:05.920 --> 00:56:07.920
Yeah.

s1098
00:56:11.920 --> 00:56:13.920
Mhm.

s1099
00:56:17.359 --> 00:56:19.359
Yeah,

s1100
00:56:21.839 --> 00:56:23.839
yeah,

s1101
00:56:25.200 --> 00:56:26.480
that's a really great question.

s1102
00:56:26.480 --> 00:56:26.799
Thank you.

s1103
00:56:26.799 --> 00:56:31.440
Yeah, his question was uh on the topic of on the topic of where do you build evals?

s1104
00:56:31.440 --> 00:56:38.640
I said usually when you have any system with either nondeterministic inputs or nondeterministic outputs or both, that's where you usually want to have some evals.

s1105
00:56:38.640 --> 00:56:48.640
So then he asked me a lot of people use coding agents cursor cloud codec codeex um and skills with them do I build evals for my skills right um and my answer is no

s1106
00:56:48.640 --> 00:57:01.359
because I think that's the wrong it's not my responsibility because I'm not building the harness so I can refine my answer a little bit to when to build evals you want to build evals when you build a harness I think that's better or you want to build evals when you use a harness and so I'm not writing

s1107
00:57:01.359 --> 00:57:09.599
evals for my skills because I believe Claude code or anthropic has written eval skill poisoning conversation.

s1108
00:57:09.599 --> 00:57:14.480
So you build an eval to give you ahead of time reliability where your harness gives you just in time reliability.

s1109
00:57:14.480 --> 00:57:19.280
And so eval harnesses go hand in hand and I live in userland when I use a coding agent.

s1110
00:57:19.280 --> 00:57:26.559
And so I'm I'm not really in fact skills you can say are deterministic because I write them right and so there's no room for eval there.

s1111
00:57:26.559 --> 00:57:27.599
Good question.

s1112
00:57:27.599 --> 00:57:28.400
Any others?

s1113
00:57:28.400 --> 00:57:30.640
Yes sir.

s1114
00:57:32.720 --> 00:57:34.720
Yes.

s1115
00:57:39.119 --> 00:57:41.119
Yeah.

s1116
00:57:47.599 --> 00:57:49.599
Yeah.

s1117
00:57:49.839 --> 00:57:50.720
Yeah.

s1118
00:57:50.720 --> 00:57:51.760
That's a really good question.

s1119
00:57:51.760 --> 00:57:51.920
Yeah.

s1120
00:57:51.920 --> 00:57:54.400
So, his question was about the the loop in general.

s1121
00:57:54.400 --> 00:57:56.559
Um, where does the policy fit in?

s1122
00:57:56.559 --> 00:57:56.880
Right.

s1123
00:57:56.880 --> 00:58:02.799
Um, do we do we retrieve the policy from production or was that the question?

s1124
00:58:06.160 --> 00:58:07.920
Yes.

s1125
00:58:07.920 --> 00:58:08.400
Right.

s1126
00:58:08.400 --> 00:58:08.799
Yeah.

s1127
00:58:08.799 --> 00:58:14.880
Um I think the production LLM agent and the judge should share the same policy.

s1128
00:58:14.880 --> 00:58:15.200
Right.

s1129
00:58:15.200 --> 00:58:16.559
That's

s1130
00:58:16.559 --> 00:58:22.480
exactly or you don't you don't even copy it over but it lives in some source that you query that with rag or something.

s1131
00:58:22.480 --> 00:58:23.040
Exactly.

s1132
00:58:23.040 --> 00:58:24.559
But the policy must be one to one.

s1133
00:58:24.559 --> 00:58:25.760
That's the whole point right.

s1134
00:58:25.760 --> 00:58:28.880
Um everybody needs to know the policy so they can enforce it.

s1135
00:58:28.880 --> 00:58:32.640
Um and so when ideally so my policy here was hardcoded.

s1136
00:58:32.640 --> 00:58:34.400
it was like written in my file.

s1137
00:58:34.400 --> 00:58:37.040
Um that's that's just for this.

s1138
00:58:37.040 --> 00:58:45.760
Um open rag as I mentioned is has an API where you can say get me all the policy about this over everything we've accumulated over the past hundred years that I have access to.

s1139
00:58:45.760 --> 00:58:45.920
Right?

s1140
00:58:45.920 --> 00:58:51.680
And so the policy is a living thing that is retrieved both by the judge and by the inproduction agent.

s1141
00:58:51.680 --> 00:58:54.000
Does make sense?

s1142
00:58:55.599 --> 00:58:57.599
Yeah,

s1143
00:59:00.799 --> 00:59:01.760
that's a good question.

s1144
00:59:01.760 --> 00:59:02.480
either or.

s1145
00:59:02.480 --> 00:59:05.040
I think personally I would have it be autonomous.

s1146
00:59:05.040 --> 00:59:09.200
Um because my agent in production is an agent with retrieval tools.

s1147
00:59:09.200 --> 00:59:12.799
I want my judge to also be exactly the same as my agent with retrieval tools.

s1148
00:59:12.799 --> 00:59:18.000
The judge and the agent, the closer the judge is to the agent, the more parody you're going to have, right?

s1149
00:59:18.000 --> 00:59:20.240
And so then you get a good signal.

s1150
00:59:20.240 --> 00:59:23.359
When CI fails, then my agent will probably fail.

s1151
00:59:23.359 --> 00:59:24.160
That's the whole point.

s1152
00:59:24.160 --> 00:59:24.799
Exactly.

s1153
00:59:24.799 --> 00:59:25.760
Great question.

s1154
00:59:25.760 --> 00:59:26.720
Uh we have time.

s1155
00:59:26.720 --> 00:59:28.079
We have we have some more time questions.

s1156
00:59:28.079 --> 00:59:28.960
No, we're out of time.

s1157
00:59:28.960 --> 00:59:29.119
Okay.

s1158
00:59:29.119 --> 00:59:29.680
Well, we're out.

s1159
00:59:29.680 --> 00:59:30.480
Thank you again, everybody.

s1160
00:59:30.480 --> 00:59:33.040
This was so much fun.