WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=qZj7sqmidqA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/qZj7sqmidqA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.960 --> 00:00:15.519
Hello everyone.

s2
00:00:15.519 --> 00:00:21.119
Uh I have a packed room for a two-hour workshop which is amazing.

s3
00:00:21.119 --> 00:00:22.400
Welcome to AI Engineer.

s4
00:00:22.400 --> 00:00:26.960
I know no one's officially welcomed you but uh welcome to AI Engineer.

s5
00:00:26.960 --> 00:00:28.560
Uh this venue is amazing.

s6
00:00:28.560 --> 00:00:35.280
This crowd is really quite surprising and I have uh a ton of content for you today.

s7
00:00:35.280 --> 00:00:36.880
Uh my name is Lori Voss.

s8
00:00:36.880 --> 00:00:40.000
I'm head of developer relations at AriseAI.

s9
00:00:40.000 --> 00:00:47.120
Uh some of you may remember me from when I used to co-found npm Inc. Uh so you may remember me from like JavaScript days.

s10
00:00:47.120 --> 00:00:53.360
Uh but these days I talk about AI uh and how to test it uh and how to make it work.

s11
00:00:53.360 --> 00:00:55.920
So what are we going to be covering today?

s12
00:00:55.920 --> 00:00:58.800
Uh we've got a really good stretch of time together.

s13
00:00:58.800 --> 00:01:02.480
Uh so we're going to cover a lot of ground and I'm going to start with fundamentals.

s14
00:01:02.480 --> 00:01:05.119
That's why this is called a 101 session.

s15
00:01:05.119 --> 00:01:14.000
Uh we're going to start with what eval are, why you need them, uh and why agents make evaluation harder than simple LLM applications.

s16
00:01:14.000 --> 00:01:21.439
Uh then we're going to set up tracing in Arise AX, uh which is how you capture the raw data you need to run evals in the first place.

s17
00:01:21.439 --> 00:01:26.240
Uh, we're going to build a simple AI agent uh with the claw agent SDK.

s18
00:01:26.240 --> 00:01:30.240
We're going to run it and we're going to look at uh the traces it produces.

s19
00:01:30.240 --> 00:01:35.200
Can I get a show of hands if you've already built an agent?

s20
00:01:35.200 --> 00:01:36.720
All right, great.

s21
00:01:36.720 --> 00:01:44.640
I'm going to spend no time at all on how the agent works because I assume everybody knows how an agent works and you're here to learn how to evaluate them.

s22
00:01:44.640 --> 00:01:50.960
Uh once we've got some data, uh we're going to do something that a lot of tutorials skip, which is we're going to actually look at the data.

s23
00:01:50.960 --> 00:01:52.320
Uh we're going to read our traces.

s24
00:01:52.320 --> 00:01:54.560
We're going to categorize what went wrong.

s25
00:01:54.560 --> 00:01:58.399
Uh and we're going to figure out what to measure before we write a single eval.

s26
00:01:58.399 --> 00:02:00.880
Then we're going to write three kinds of evals.

s27
00:02:00.880 --> 00:02:07.040
There are code evals, which are simple deterministic checks, uh very similar to unit tests.

s28
00:02:07.040 --> 00:02:16.800
Then we're going to use built-in LLM evals uh like faithfulness uh where a second LLM judges the uh output of your first one.

s29
00:02:16.800 --> 00:02:21.920
Uh and we're going to do a custom eval uh LLM as a judge from scratch.

s30
00:02:21.920 --> 00:02:28.480
Uh and we're also going to test whether our judges are judging correctly, a process called meta evaluation.

s31
00:02:28.480 --> 00:02:35.360
Then we're going to finish with data sets and experiments, which is how you iterate on your agent uh and actually measure whether it's getting better.

s32
00:02:35.360 --> 00:02:43.680
uh including we're going to use cloud code to automatically improve our app in response to feedback from our evals.

s33
00:02:43.680 --> 00:02:48.160
So, first things first, uh get your notebook.

s34
00:02:48.160 --> 00:02:53.040
Uh I messed up earlier and didn't have permissions set correctly.

s35
00:02:53.040 --> 00:02:56.160
So, if you scanned this earlier and it didn't work, it works now.

s36
00:02:56.160 --> 00:03:00.080
Uh grab this notebook and make a copy for yourself.

s37
00:03:00.080 --> 00:03:03.519
That is where we're going to be living all day.

s38
00:03:04.800 --> 00:03:11.519
And I'm going to wait for that to happen because otherwise I'm going to have to keep swapping back to this slide.

s39
00:03:11.519 --> 00:03:15.360
All right, most of those are down.

s40
00:03:15.360 --> 00:03:19.120
No, no, I'm still seeing people taking pictures.

s41
00:03:19.840 --> 00:03:22.400
All right, cool.

s42
00:03:22.720 --> 00:03:26.080
We're also going to need a certain amount of AI infrastructure.

s43
00:03:26.080 --> 00:03:30.000
I'm going to be using anthropic and cloud code today.

s44
00:03:30.000 --> 00:03:33.040
Uh, Arise AX works with OpenAI.

s45
00:03:33.040 --> 00:03:37.680
It works with every other uh, lab provider, but I had to pick one in order to do a workshop about it.

s46
00:03:37.680 --> 00:03:38.720
So, I've used Claude.

s47
00:03:38.720 --> 00:03:45.920
So, this is going, this notebook is going to expect you to have uh, an anthropic API key, which you can get from console.anthropic.com.

s48
00:03:45.920 --> 00:03:50.959
If not, you can, if you are very clever, do a fast swap to OpenAI.

s49
00:03:50.959 --> 00:03:53.519
Um, you're also going to need an Arise account.

s50
00:03:53.519 --> 00:03:56.480
Go to arise.com and start a free trial.

s51
00:03:56.480 --> 00:04:05.280
Um, from there you will need an Arise Space ID and an Arise API key, both of which you can get from the settings page.

s52
00:04:05.280 --> 00:04:08.959
Uh, my lovely assistant Dat is here.

s53
00:04:08.959 --> 00:04:10.799
Uh, that's him in the corner.

s54
00:04:10.799 --> 00:04:21.680
If you run into any problems getting your API keys or anything like that, uh, just flag him down and he will run over to you and help you figure it out, which is,

s55
00:04:21.680 --> 00:04:23.040
uh, incredibly nice of him.

s56
00:04:23.040 --> 00:04:24.800
Thank you, Dad.

s57
00:04:24.800 --> 00:04:30.240
Um, yes.

s58
00:04:30.240 --> 00:04:34.560
So, uh, who who had all that stuff already?

s59
00:04:34.560 --> 00:04:38.400
Anybody already got an Aris X account?

s60
00:04:38.880 --> 00:04:41.280
All right, I'm going to let you hang out for this.

s61
00:04:41.280 --> 00:04:44.880
I'm going to let you set that one up for a little while.

s62
00:04:48.639 --> 00:05:02.560
The way you do it is you go to arise.com and you hit Oops, it's sent me to the docs because I always live in the docs.

s63
00:05:02.560 --> 00:05:04.160
You go to arise.com.

s64
00:05:04.160 --> 00:05:05.360
Thank you.

s65
00:05:05.360 --> 00:05:15.759
Uh and you hit the big pink get started button and then you sign up from here.

s66
00:05:16.080 --> 00:05:18.160
Do not accidentally sign up for Phoenix.

s67
00:05:18.160 --> 00:05:23.919
Phoenix is our lovely open- source project which also has a hosted service underneath Phoenix.

s68
00:05:23.919 --> 00:05:27.039
Uh sometimes when you Google Arise, you get sent to Phoenix instead.

s69
00:05:27.039 --> 00:05:28.240
Don't sign up for Phoenix.

s70
00:05:28.240 --> 00:05:31.280
You're doing AX today.

s71
00:05:34.720 --> 00:05:35.360
All right.

s72
00:05:35.360 --> 00:05:39.840
While you are signing up, uh I'm going to say a little bit more bit more about what AX is.

s73
00:05:39.840 --> 00:05:42.800
It is an observability and evaluations platform.

s74
00:05:42.800 --> 00:05:45.840
It captures traces from your AI application.

s75
00:05:45.840 --> 00:05:51.280
It stores and runs your evaluations and it watches your app in production and alerts you when something goes wrong.

s76
00:05:51.280 --> 00:05:53.520
Uh by default, we host it for you.

s77
00:05:53.520 --> 00:05:58.720
That is why we're using it today because you don't have to like run some software to make it run.

s78
00:05:58.720 --> 00:06:01.120
Uh so there's no infrastructure to manage.

s79
00:06:01.120 --> 00:06:09.600
Uh though if you are from an enterprise that is super precious about your data and you're like no, we need to run it ourselves, you can absolutely do that if you want to.

s80
00:06:09.600 --> 00:06:17.280
Um, and like I said, uh, if you've used Arise Phoenix, which is our open source tool, a lot of this will feel familiar.

s81
00:06:17.280 --> 00:06:19.199
Uh, they have the same core ideas.

s82
00:06:19.199 --> 00:06:29.759
One is, uh, built for open source accessibility, and the other one is built for, uh, enterprise with production and teams and things like that.

s83
00:06:30.560 --> 00:06:35.039
So, how are we doing on signing up for keys right now?

s84
00:06:35.600 --> 00:06:36.880
Wi-Fi sucks.

s85
00:06:36.880 --> 00:06:38.400
The Wi-Fi sucks.

s86
00:06:38.400 --> 00:06:44.000
That is predictable really, isn't it?

s87
00:06:49.280 --> 00:06:50.720
All right.

s88
00:06:50.720 --> 00:06:56.319
Is anybody like desperately in need of assistance right now?

s89
00:07:05.039 --> 00:07:07.120
All right.

s90
00:07:12.240 --> 00:07:12.560
Okay.

s91
00:07:12.560 --> 00:07:14.880
Well, the advice is to switch to your phones as a hot spot.

s92
00:07:14.880 --> 00:07:17.840
I apologize for the Wi-Fi.

s93
00:07:18.720 --> 00:07:29.120
I'm really worried about my 2PM workshop where I have a Git repo that I know is 200 megabytes large and I'm like, no one is going to get the repo because it's too big.

s94
00:07:29.440 --> 00:07:32.000
Yeah, start cloning that repo now.

s95
00:07:32.000 --> 00:07:35.680
Anyway, uh let's talk about these these fundamentals.

s96
00:07:35.680 --> 00:07:39.840
There's a lot of theory in this workshop which gives you time to uh catch up.

s97
00:07:39.840 --> 00:07:42.080
Um here's the mental model.

s98
00:07:42.080 --> 00:07:46.960
Evals are testing for AI and traces are logs for EI AI.

s99
00:07:46.960 --> 00:07:49.520
Why we had to come up with two completely new names for those things?

s100
00:07:49.520 --> 00:07:50.880
I don't really know.

s101
00:07:50.880 --> 00:07:53.680
Uh you write tests for your code.

s102
00:07:53.680 --> 00:07:55.840
Given this input, I expect this output.

s103
00:07:55.840 --> 00:08:02.560
Uh eval do the same thing, but for AI outputs where the outputs are never exactly the same twice.

s104
00:08:02.560 --> 00:08:08.240
Just as logs record what your server did at runtime, uh, traces record what your AI did.

s105
00:08:08.240 --> 00:08:15.360
So, every agent call, every tool call, every LLM invocation with the inputs and outputs at each step.

s106
00:08:15.360 --> 00:08:19.120
The building blocks of a trace are called spans.

s107
00:08:19.120 --> 00:08:22.800
Uh, in this little screenshot, you can see a trace in Arise AX.

s108
00:08:22.800 --> 00:08:24.160
It's made up of a bunch of spans.

s109
00:08:24.160 --> 00:08:26.240
It's got this tree structure.

s110
00:08:26.240 --> 00:08:29.199
Uh, and each span represents one step in the execution.

s111
00:08:29.199 --> 00:08:30.560
An LLM call is a span.

s112
00:08:30.560 --> 00:08:31.520
A tool call is a span.

s113
00:08:31.520 --> 00:08:35.360
And a full agent turn is a span that contains other spans inside of it.

s114
00:08:35.360 --> 00:08:40.959
Each span records its input, its output, uh timing, token counts, and a whole bunch of metadata.

s115
00:08:40.959 --> 00:08:44.000
Uh everything that you need to understand what happened at that step.

s116
00:08:44.000 --> 00:08:45.519
Traces tell you what happened.

s117
00:08:45.519 --> 00:08:47.760
Evals tell you whether it was any good.

s118
00:08:47.760 --> 00:08:50.560
That is the model for mental model for today.

s119
00:08:50.560 --> 00:08:53.760
This is not rocket science.

s120
00:08:54.240 --> 00:08:57.200
But let me talk about why we even need evals.

s121
00:08:57.200 --> 00:08:59.680
Uh and that is the vibes problem.

s122
00:08:59.680 --> 00:09:07.519
Uh AI features usually get shipped by building an AI feature, running a few queries, saying does this look right?

s123
00:09:07.519 --> 00:09:08.399
Yeah, it looks good.

s124
00:09:08.399 --> 00:09:09.680
And then you ship it.

s125
00:09:09.680 --> 00:09:14.080
That is by far the most popular method of shipping AI applications today.

s126
00:09:14.080 --> 00:09:18.720
Uh and the problem is that then it fails on inputs that you didn't test.

s127
00:09:18.720 --> 00:09:23.519
Uh and the thing is it probably looks good on your test queries, but the problem is that you ran it three times.

s128
00:09:23.519 --> 00:09:25.839
Three times is not a test suite.

s129
00:09:25.839 --> 00:09:33.680
Uh the usual fix for uh flaky software is unit tests and they don't work here because the output is non-deterministic.

s130
00:09:33.680 --> 00:09:36.480
Uh the same prompt produces different text every run.

s131
00:09:36.480 --> 00:09:38.880
So two different responses might both be correct.

s132
00:09:38.880 --> 00:09:42.000
So there's no expected string that you can assert against.

s133
00:09:42.000 --> 00:09:43.920
So teams fall back on human review.

s134
00:09:43.920 --> 00:09:45.040
They watch it run.

s135
00:09:45.040 --> 00:09:47.040
It looks fine and they ship it.

s136
00:09:47.040 --> 00:09:48.640
Uh that doesn't scale.

s137
00:09:48.640 --> 00:09:50.000
It doesn't catch regressions.

s138
00:09:50.000 --> 00:09:53.600
And most importantly, it doesn't run in CI.

s139
00:09:54.240 --> 00:09:56.399
Here's some things you can't do without evals.

s140
00:09:56.399 --> 00:10:00.320
Uh if you change your system prompt to fix a tone issue, then the tone would get better.

s141
00:10:00.320 --> 00:10:03.440
Uh but now your uh bot is hallucinating product features.

s142
00:10:03.440 --> 00:10:08.080
Without eval catch that uh until a user reports it.

s143
00:10:08.080 --> 00:10:12.160
With a faithfulness eval, you would have seen the spike before it shipped.

s144
00:10:12.160 --> 00:10:17.360
Uh and every prompt potentially affects every kind of input that users send.

s145
00:10:17.360 --> 00:10:28.079
So you can be changing the prompt for one thing and have a completely unexpected effect at some totally different part of your application because the prompt is just a huge block of text that you're sending to your AI.

s146
00:10:28.079 --> 00:10:30.959
Uh you can't manually test every combination.

s147
00:10:30.959 --> 00:10:32.880
So without eval you're playing whack-a-ole.

s148
00:10:32.880 --> 00:10:36.000
You fix one problem and you break something somewhere else.

s149
00:10:36.000 --> 00:10:41.839
Evals instead give you a number that you can track and compare and act on.

s150
00:10:41.839 --> 00:10:44.079
You also can't switch models without eval.

s151
00:10:44.079 --> 00:10:47.200
This matters more and more because this field moves so fast.

s152
00:10:47.200 --> 00:10:49.760
Uh new models drop every few months.

s153
00:10:49.760 --> 00:10:53.839
Uh without eval switching models means weeks of manual testing.

s154
00:10:53.839 --> 00:10:54.320
Is it better?

s155
00:10:54.320 --> 00:10:54.880
Is it worse?

s156
00:10:54.880 --> 00:10:56.640
Did something get did something break?

s157
00:10:56.640 --> 00:10:57.680
You don't know.

s158
00:10:57.680 --> 00:10:59.440
With evals, you run the suite.

s159
00:10:59.440 --> 00:11:02.240
You compare the scores uh and you know within hours.

s160
00:11:02.240 --> 00:11:08.079
And that is the difference between flying by the seat of your pants and flying by instruments.

s161
00:11:08.240 --> 00:11:10.160
And this is not theoretical.

s162
00:11:10.160 --> 00:11:26.320
uh real teams, Dcript, Bolt, Anthropic's own clawed code all follow the same arc of shipping on vibes and then discovering that vibes don't scale uh and then uh building evals and that is the same arc that we're going to follow in today's workshop.

s163
00:11:27.120 --> 00:11:30.399
As I mentioned earlier, there are two broad types of evals.

s164
00:11:30.399 --> 00:11:32.560
Code evals are deterministic functions.

s165
00:11:32.560 --> 00:11:39.200
They run in milliseconds and they cost nothing and they give you an immediate unambiguous answer like did the output parses JSON?

s166
00:11:39.200 --> 00:11:41.440
Is the output less than 500 tokens?

s167
00:11:41.440 --> 00:11:43.839
Did it mention the thing that I asked about?

s168
00:11:43.839 --> 00:11:48.720
The big advantage of Kodi valves is that they are fast and cheap and totally reproducible.

s169
00:11:48.720 --> 00:11:52.480
They get the same they get the same input and they give the same answer every time.

s170
00:11:52.480 --> 00:11:55.360
That makes them very similar to old style unit tests.

s171
00:11:55.360 --> 00:11:59.120
The downside is that like old style unit tests, they can be brittle.

s172
00:11:59.120 --> 00:12:04.000
Uh we have some strategies that can help avoid that and I'm going to be covering those later.

s173
00:12:04.000 --> 00:12:14.160
um because two perfectly correct answers might be worded completely differently and you have to adjust your expectations of unit tests to be able to handle that.

s174
00:12:14.160 --> 00:12:17.120
The second type of eval is an LLM as a judge eval.

s175
00:12:17.120 --> 00:12:21.040
These use a second LLM to grade your outputs against a rubric.

s176
00:12:21.040 --> 00:12:25.279
Rubric being yet another word that we made up uh that just means rules.

s177
00:12:25.279 --> 00:12:29.760
It is your set of rules that you put in your prompt about what counts as good.

s178
00:12:29.760 --> 00:12:32.800
Uh LLM evals are much more flexible.

s179
00:12:32.800 --> 00:12:36.240
uh they can handle the kinds of questions that Kodi valves can't answer.

s180
00:12:36.240 --> 00:12:39.279
Uh is this response uh factually accurate?

s181
00:12:39.279 --> 00:12:41.120
Did it stay faithful to the source material?

s182
00:12:41.120 --> 00:12:42.399
Is the tone right?

s183
00:12:42.399 --> 00:12:51.680
Uh for a customer service context, the strength of LLM judges uh is that they understand meaning uh not just the strings that are in your output.

s184
00:12:51.680 --> 00:12:53.839
But LLM judges have trade-offs.

s185
00:12:53.839 --> 00:12:58.880
The weakness is that they cost money to run and they are nondeterministic themselves.

s186
00:12:58.880 --> 00:13:05.519
Uh so they can be wrong which means that you need to calibrate them against human judgment and we're going to do that later today.

s187
00:13:05.519 --> 00:13:10.320
Um there's also a third type of eval which is human evaluation.

s188
00:13:10.320 --> 00:13:13.279
Uh a domain expert can read your output and grade it.

s189
00:13:13.279 --> 00:13:16.000
Uh and that is the gold standard for quality.

s190
00:13:16.000 --> 00:13:21.519
Uh but it is slow and expensive and humans get tired which is why we can't use it all the time.

s191
00:13:21.519 --> 00:13:23.920
Uh fun fact that's not very fun.

s192
00:13:23.920 --> 00:13:27.680
Human annotators miss up to 50% of defects due to fatigue.

s193
00:13:27.680 --> 00:13:29.279
So they're not perfect either.

s194
00:13:29.279 --> 00:13:34.480
So when you build an LLM as a judge and you're like, well, it doesn't always catch my errors.

s195
00:13:34.480 --> 00:13:39.360
Remember that if you were judging with a human, the human would also not always catch your errors.

s196
00:13:39.360 --> 00:13:41.519
Uh but these are complimentary approaches.

s197
00:13:41.519 --> 00:13:49.360
Most real applications are going to use a a collection of Kodi valves, LLM as a judge, and human evaluation.

s198
00:13:49.360 --> 00:13:51.440
So when do you use which?

s199
00:13:51.440 --> 00:13:54.480
Use Kodi valves when the answer is deterministic.

s200
00:13:54.480 --> 00:14:11.199
So, format validation, length limits, forbidden phrases, uh required fields, use LLM judges when you need semantic understanding, a correctness, eval accurately, a faithfulness eval asks, did it stick to the source documents without hallucinating?

s201
00:14:11.199 --> 00:14:16.079
Uh, and you should keep humans in the loop for failure modes that you haven't seen before.

s202
00:14:16.079 --> 00:14:19.519
Uh, and to verify that your LLM judges are actually judging correctly.

s203
00:14:19.519 --> 00:14:21.760
LLM judges can be wrong.

s204
00:14:21.760 --> 00:14:27.120
uh judging them against human judgment is how you know whether or not you can trust them.

s205
00:14:27.839 --> 00:14:31.199
All of that is true of any application that uses LLMs.

s206
00:14:31.199 --> 00:14:35.760
Uh but agents make things even harder in some ways that are uh worth calling out.

s207
00:14:35.760 --> 00:14:38.320
You can think of it as a ladder of complexity.

s208
00:14:38.320 --> 00:14:40.720
A single LM call is relatively simple.

s209
00:14:40.720 --> 00:14:42.480
You get an input, you get an output.

s210
00:14:42.480 --> 00:14:51.600
When you're talking about an agent, an agent is making a series of tool calls and a series of decisions, which means that at every single step in the uh chain,

s211
00:14:51.600 --> 00:15:00.000
it is the chances that it has made a mistake are getting higher and so the chances that it will go off the rails uh are going to get uh greater.

s212
00:15:00.000 --> 00:15:03.199
So you have to judge the intermediate steps as well.

s213
00:15:03.199 --> 00:15:06.880
You have to uh check whether your agent is using the right tool.

s214
00:15:06.880 --> 00:15:08.320
Did it pass the right arguments?

s215
00:15:08.320 --> 00:15:09.839
Did it know when to stop?

s216
00:15:09.839 --> 00:15:13.760
getting stuck in loops is a real failure case for agents.

s217
00:15:13.760 --> 00:15:18.639
Uh and then multi- aent systems if you are building those add yet another level of complexity.

s218
00:15:18.639 --> 00:15:19.600
They add handoffs.

s219
00:15:19.600 --> 00:15:22.240
One agent decides to pass work to another.

s220
00:15:22.240 --> 00:15:24.240
Did the triage agent route correctly?

s221
00:15:24.240 --> 00:15:27.680
Did the specialist agent handle the the handoff gracefully.

s222
00:15:27.680 --> 00:15:30.959
Each layer adds new ways that things go wrong.

s223
00:15:30.959 --> 00:15:37.920
Uh and non-determinism means that you need to have more evals because errors can cascade.

s224
00:15:38.079 --> 00:15:41.920
Uh here's what an example of an cascading failure looks like.

s225
00:15:41.920 --> 00:15:58.320
If you're you ask a query about Tesla, meaning the car company, your agent searches the web, finds a whole bunch of information about an 18th century inventor, passes that back to your agent, which writes a beautiful, well- formatted report about an 18th century inventor, and passes that to your boss.

s226
00:15:58.320 --> 00:16:00.079
Nothing that it did there was wrong.

s227
00:16:00.079 --> 00:16:02.160
It you told it to search the web for Tesla.

s228
00:16:02.160 --> 00:16:03.279
It searched the web for Tesla.

s229
00:16:03.279 --> 00:16:06.000
It got information about Tesla, and it wrote a report about Tesla.

s230
00:16:06.000 --> 00:16:08.079
where where did it go wrong?

s231
00:16:08.079 --> 00:16:14.720
Uh this is worse than an obvious failure because you have to be reading your inputs and outputs for that to work.

s232
00:16:14.720 --> 00:16:18.959
You can't there was no unit test that would have caught that.

s233
00:16:19.920 --> 00:16:21.839
Uh but agents can also do the opposite.

s234
00:16:21.839 --> 00:16:26.720
They can get things right in a way that your tests weren't expecting and get graded wrong as a result.

s235
00:16:26.720 --> 00:16:34.720
This happened to Anthropic when they ran a benchmark called Tao Squared Bench uh which simulates multi-turn customer to service tasks.

s236
00:16:34.720 --> 00:16:44.880
uh they asked the agent to book flights uh and asked it to do something that they thought was impossible which was reschedule an economycl class flight.

s237
00:16:44.880 --> 00:16:52.240
Uh but it turned out that the policies that they gave the agent allowed the agent uh to first upgrade the flight to first class.

s238
00:16:52.240 --> 00:16:57.120
First class flights can be rescheduled which uh economy flights cannot.

s239
00:16:57.120 --> 00:17:06.799
So the agent found a way to reschedule an economy flight uh and failed the test because it was supposed to be not able to do that and it was able to do that.

s240
00:17:06.799 --> 00:17:15.760
Uh so this is an example of how agents being flexible agents finding things that you didn't think of uh needs to be accounted for in your tests.

s241
00:17:15.760 --> 00:17:25.520
Uh there is a difference between creatively correct and wrong and we're going to come back to that when we talk about grading your agents later or rather grading your evals.

s242
00:17:25.520 --> 00:17:34.559
Um there's also a second way of categorizing your eval into two categories uh which is uh capability evals and regression evals.

s243
00:17:34.559 --> 00:17:39.440
A capability eval asks uh can my agent do this thing at all?

s244
00:17:39.440 --> 00:17:42.880
Um capability evals are expected to mostly fail.

s245
00:17:42.880 --> 00:17:45.760
Your agent is expected to get a very low score on them.

s246
00:17:45.760 --> 00:17:47.520
Uh because they give you a hill to climb.

s247
00:17:47.520 --> 00:17:50.640
It gives you something for your agent to try and get better at.

s248
00:17:50.640 --> 00:17:54.720
Uh once you've climbed that hill, your capability eval turns into a regression eval.

s249
00:17:54.720 --> 00:18:02.320
A regression eval is an eval that you expect your agent to pass completely or nearly completely every single time.

s250
00:18:02.320 --> 00:18:16.640
So as your agent matures, you'll be giving it capability eval after capability eval and slowly turning them into a suite of regression evals that you run all the time to make sure that uh previous behavior has not regressed.

s251
00:18:17.600 --> 00:18:20.000
This is what an eval result looks like.

s252
00:18:20.000 --> 00:18:22.640
Uh every eval produces a score and a label.

s253
00:18:22.640 --> 00:18:26.160
In this case, it's a numeric score just often just zero or one.

s254
00:18:26.160 --> 00:18:29.760
Binary scores are very easy for uh LLMs to do.

s255
00:18:29.760 --> 00:18:34.880
Uh and it comes with a human readable label like correct or incorrect or valid or invalid.

s256
00:18:34.880 --> 00:18:38.880
Um LLM judges add a third thing which is the explanation.

s257
00:18:38.880 --> 00:18:42.640
Um code eval produce explanations because they are just code.

s258
00:18:42.640 --> 00:18:51.039
Um but LLM, an LLM as a judge will say not just that something is incorrect, but it will say why it decided that something is correct.

s259
00:18:51.039 --> 00:19:05.520
And this is incredibly valuable uh because if you are living in a world of coding agents, you can take a whole bunch of explanations from a bunch of evals and pass them back to your coding agent and say here is all the reasons that you messed up in the last test.

s260
00:19:05.520 --> 00:19:07.679
What could you do that would make you better at this?

s261
00:19:07.679 --> 00:19:10.559
And your coding agent will just go cool thank you for the feedback.

s262
00:19:10.559 --> 00:19:14.640
Improve your application uh and it will get better at it next time.

s263
00:19:14.640 --> 00:19:17.679
This is something I'm going to show you uh in today's workshop.

s264
00:19:17.679 --> 00:19:22.480
It is how you turn a failing eval into a prompt improvement.

s265
00:19:22.559 --> 00:19:24.799
This is a real judge explanation.

s266
00:19:24.799 --> 00:19:28.960
Um, in this example, it it's a travel planning agent.

s267
00:19:28.960 --> 00:19:31.520
Um, and you're running a correctness eval.

s268
00:19:31.520 --> 00:19:32.880
The judge doesn't just say incorrect.

s269
00:19:32.880 --> 00:19:34.320
It tells you exactly what's missing.

s270
00:19:34.320 --> 00:19:35.440
You asked for a budget flight.

s271
00:19:35.440 --> 00:19:37.039
It didn't give a budget breakdown.

s272
00:19:37.039 --> 00:19:41.120
It says the agent gave destin destination info and recommendations.

s273
00:19:41.120 --> 00:19:45.440
Uh, but the user asked about budget travel and there were no cost estimates.

s274
00:19:45.440 --> 00:19:50.160
That explanation makes the eval actionable uh because you now have a concrete failure.

s275
00:19:50.160 --> 00:19:51.600
You know what to fix in the prompt.

s276
00:19:51.600 --> 00:19:57.520
Uh and the explanation is what makes evals into a useful debugging tool and not just a scoreboard.

s277
00:19:57.520 --> 00:20:07.360
Uh and if you're seeing the same explanation across 50 different traces, uh you know you have a systematic problem and not a one-off edge case, which makes it the first thing you should fix.

s278
00:20:07.360 --> 00:20:14.720
So you have to take your eval explanations uh and categorize them into types of failures uh and count up the categories.

s279
00:20:14.720 --> 00:20:15.919
This is called coding.

s280
00:20:15.919 --> 00:20:21.120
Again, we can make up words from the L from the ML word uh from the ML world.

s281
00:20:21.120 --> 00:20:24.720
Um but categorizing things is known as coding.

s282
00:20:24.720 --> 00:20:28.960
Um and of course each explanation is natural language.

s283
00:20:28.960 --> 00:20:34.960
Uh so it is hard to do this coding uh unless you use yet another LLM to do the categorization for you.

s284
00:20:34.960 --> 00:20:39.039
We'll be seeing how that works uh towards the end of today.

s285
00:20:39.039 --> 00:20:46.559
Um, and like I said, you can hand a whole pile of these explanations to a coding agent and let the app fix it for you.

s286
00:20:46.559 --> 00:20:49.200
Uh, this is the full loop that we're going to build today.

s287
00:20:49.200 --> 00:20:53.039
We're going to, uh, instrument, trace, eval, and iterate.

s288
00:20:53.039 --> 00:20:54.320
Each step feeds the next.

s289
00:20:54.320 --> 00:20:56.640
You define what you want.

s290
00:20:56.640 --> 00:20:59.280
Uh, you build it, you measure how well it works.

s291
00:20:59.280 --> 00:21:03.520
You ship it, you monitor it in production, and you iterate based on what you see.

s292
00:21:03.520 --> 00:21:06.640
Every traditional software product goes through a loop like this.

s293
00:21:06.640 --> 00:21:12.400
Uh but for AI the measurement step is where most teams fall down and evals are how you measure.

s294
00:21:12.400 --> 00:21:16.960
Evals are the connective tissue uh across this whole life cycle.

s295
00:21:16.960 --> 00:21:19.919
Uh so let's get started building it.

s296
00:21:19.919 --> 00:21:23.840
Step one is setting up tracing with Arise EX AX before we can run evals.

s297
00:21:23.840 --> 00:21:27.600
We need something uh some data to run our evals on.

s298
00:21:27.600 --> 00:21:30.000
Uh and you can't evaluate what you can't observe.

s299
00:21:30.000 --> 00:21:33.039
So observation comes first.

s300
00:21:33.679 --> 00:21:46.240
So hopefully you've conquered the Wi-Fi uh and you have our notebook at this point and that is running around to make sure uh that you do.

s301
00:21:46.240 --> 00:21:52.720
Uh let's go to our very first code cell which is where I install our dependencies.

s302
00:21:52.720 --> 00:21:56.240
Uh claude agent SDK is self-explanatory.

s303
00:21:56.240 --> 00:21:58.320
That is the claud agent SDK.

s304
00:21:58.320 --> 00:22:05.440
uh open inference open inference instrumentation claude agent SDK is the auto instrumentation package for the claude agent SDK.

s305
00:22:05.440 --> 00:22:12.000
This is how it knows to capture traces from cloud agents uh without you changing your application code.

s306
00:22:12.000 --> 00:22:17.600
Uh as I mentioned earlier, you can use open AI instead if you want to uh or some framework.

s307
00:22:17.600 --> 00:22:19.919
Um we have packages for all of those as well.

s308
00:22:19.919 --> 00:22:21.760
You just need to swap them in.

s309
00:22:21.760 --> 00:22:28.080
Um, Arise and Arise Otel are the packages that send uh your traces to AX and let you read them back.

s310
00:22:28.080 --> 00:22:32.640
Uh, and there's a Phoenix package in there that we use as utility package.

s311
00:22:32.640 --> 00:22:34.480
Don't worry about it.

s312
00:22:34.480 --> 00:22:42.000
Um, the anthropic package lets you use Claude both to power our agent and to judge its output later.

s313
00:22:42.000 --> 00:22:45.600
Uh, so uh, go ahead and run this cell if you haven't already.

s314
00:22:45.600 --> 00:22:50.159
And while it's installing, I'm going to talk about what it is that we're going to actually build.

s315
00:22:50.159 --> 00:22:55.840
Um the claude agent SDK if you haven't already used it is Anthropic's framework for building agents.

s316
00:22:55.840 --> 00:22:59.200
So Anthropic's answer to lang chain and things like that.

s317
00:22:59.200 --> 00:23:04.080
Uh it can use tools, it can search the web, it can maintain conversation context across turns.

s318
00:23:04.080 --> 00:23:13.840
Uh open AI has their own uh agent SDK and of course there are whole agent frameworks like crewa align chain and mastra and uh AX is compatible with all of those.

s319
00:23:13.840 --> 00:23:23.840
So no matter what framework you've built uh your agent in uh it is already instrumented and you can just turn on logging and all of your traces will light up.

s320
00:23:23.840 --> 00:23:32.159
Um hopefully your install is done by now which is uh optimistic timing on my part.

s321
00:23:32.159 --> 00:23:35.600
Um next set your keys.

s322
00:23:35.600 --> 00:23:40.400
Uh in this case I have stolen my keys from uh collab.

s323
00:23:40.400 --> 00:23:44.799
If you are feeling naughty, you can just paste them in directly to your cell and put them there.

s324
00:23:44.799 --> 00:23:47.679
Um, three things go in here.

s325
00:23:47.679 --> 00:23:54.960
Like I said earlier, you need an anthropic API key, uh, which is going to power our agents LLM calls, and it's also going to power the judge later.

s326
00:23:54.960 --> 00:24:06.000
You need an Arise API key, which you get from your settings, which authenticates your traffic to the AX service, and you need an Arise space ID, which tells AX which uh, workspace to put all of your data in.

s327
00:24:06.000 --> 00:24:10.400
Uh if your keys aren't working, the usual culprit is a copype error.

s328
00:24:10.400 --> 00:24:16.480
Uh and of course that is still running around to help you if uh your keys aren't working.

s329
00:24:16.480 --> 00:24:20.000
That is has been described as a walking security hole.

s330
00:24:20.000 --> 00:24:25.440
Uh so he will definitely give you a key if you can't figure out how to get your keys to work.

s331
00:24:25.440 --> 00:24:29.120
So now we scroll to the register section.

s332
00:24:29.120 --> 00:24:32.720
Uh I'm going to bump up my fonts here.

s333
00:24:38.080 --> 00:24:39.200
One more time.

s334
00:24:39.200 --> 00:24:41.919
The QR up one more time.

s335
00:24:41.919 --> 00:24:42.720
Sure.

s336
00:24:42.720 --> 00:24:44.720
Probably.

s337
00:24:56.400 --> 00:24:58.400
Sorry.

s338
00:25:01.440 --> 00:25:02.080
Uh, yeah.

s339
00:25:02.080 --> 00:25:04.400
I didn't put my mine lives in Collab.

s340
00:25:04.400 --> 00:25:08.320
So, Collab's secrets feature pulls it out.

s341
00:25:09.360 --> 00:25:13.200
All right, I'm going to go back to where I was.

s342
00:25:13.200 --> 00:25:16.320
Hopefully, I can do that.

s343
00:25:24.880 --> 00:25:25.919
All right.

s344
00:25:25.919 --> 00:25:33.600
Uh so we've installed dependencies, we've talked about the SDK, we've talked about the secrets.

s345
00:25:33.600 --> 00:25:36.640
Uh and now we're at the register.

s346
00:25:36.880 --> 00:25:38.080
So this is the magic.

s347
00:25:38.080 --> 00:25:44.240
Uh anybody who works for Arise will tell you that you only have to add two lines of code to your application in order to instrument your call.

s348
00:25:44.240 --> 00:25:46.799
And this is these are the two lines of code.

s349
00:25:46.799 --> 00:25:56.159
Uh what we're doing here is we're passing in our arise space ID, our API key, uh and a name for it to trace everything to.

s350
00:25:56.159 --> 00:25:59.840
Um the register function is setting up open telemetry.

s351
00:25:59.840 --> 00:26:03.279
Open telemetry is the industry standard for application observability.

s352
00:26:03.279 --> 00:26:08.080
Uh and it has a layer on top called open inference which adds LLM specific attributes.

s353
00:26:08.080 --> 00:26:14.400
Things like prompt text, completion text, token counts, which model was called, which tools were invoked.

s354
00:26:14.400 --> 00:26:21.200
um we point it at ax uh and we give it a project name and that is how your traces get grouped in the UI.

s355
00:26:21.200 --> 00:26:33.840
Uh in this case we're also passing batch equals false because uh when you run in a notebook uh you want to send your telemetry as soon as it happens and not batch stuff up which it does for efficiency in production.

s356
00:26:33.840 --> 00:26:43.760
Um that one line at the bottom, the claude agent SDK instrument tells the cloud SDK to send a span to ax every time it makes an LLM call or invokes a tool.

s357
00:26:43.760 --> 00:26:48.320
Uh and that is now instrumented.

s358
00:26:48.320 --> 00:26:53.600
The reason this works with so little code is because open telemetry is an open standard that nearly everybody uses.

s359
00:26:53.600 --> 00:27:06.159
So the cloud the cloud SDK authors, the open AAI SDK authors, the lang chain authors, all of them have already written the code inside of their SDKs that calls open that calls open inference and sends data back.

s360
00:27:06.159 --> 00:27:09.520
So all you have to do is say, "Hey, I'm an open inference collector.

s361
00:27:09.520 --> 00:27:10.480
I live here.

s362
00:27:10.480 --> 00:27:13.039
Send it to me."

s363
00:27:13.520 --> 00:27:17.840
One more bit of setup is that the register call sends data to AX.

s364
00:27:17.840 --> 00:27:21.360
Uh and the Arise client reads data back out of AX.

s365
00:27:21.360 --> 00:27:24.880
Uh so we'll use it to pull our spans back into the notebook.

s366
00:27:24.880 --> 00:27:26.640
Uh that is what this line is about.

s367
00:27:26.640 --> 00:27:32.559
So you need to make sure that you've put in the same keys there or uh copied them across.

s368
00:27:40.880 --> 00:27:47.840
Oh, you need to make a you need to make a copy of the notebook from the file menu to be able to edit it.

s369
00:27:55.520 --> 00:27:56.960
Cool.

s370
00:27:56.960 --> 00:27:59.360
Um, all right.

s371
00:27:59.360 --> 00:28:03.039
So, if you've run that run those cells, then we are ready to build.

s372
00:28:03.039 --> 00:28:10.240
Uh, before we build, one thing worth knowing about, uh, everything we're going to do by hand today, you can also get your coding agent to do for you.

s373
00:28:10.240 --> 00:28:16.640
The reason we are doing it by hand is so that you have a fundamental understanding of what it is that you're doing and you're not just vibing your way to success.

s374
00:28:16.640 --> 00:28:24.480
Um but we publish a set of skills called the Arise skills uh which know how to do all of this for you at the skill level.

s375
00:28:24.480 --> 00:28:33.840
So you can install your Arise skills uh and um run evaluations, create data sets, uh instrument your app in the first place.

s376
00:28:33.840 --> 00:28:39.279
Um they all use uh AX's recommended patterns and you install them once with a single npx command.

s377
00:28:39.279 --> 00:28:44.320
They work in cloud code, they work in cursor, they work in codeex and dozens of other coding agents.

s378
00:28:44.320 --> 00:28:53.840
Um, so we're going to do stuff manually today, but in production in a real life environment, we are using our own skills every day to do this stuff.

s379
00:28:54.399 --> 00:28:57.200
Uh, so now let's build our agents.

s380
00:28:57.200 --> 00:29:01.039
Uh, when I asked if everybody had built an agent before, everybody put their hand up.

s381
00:29:01.039 --> 00:29:03.679
So I'm not going to spend a lot of time explaining how this agent works.

s382
00:29:03.679 --> 00:29:09.120
I'm going to assume that you already have an agent somewhere and you're just trying to instrument it and make it work for you.

s383
00:29:09.120 --> 00:29:16.399
Um, the fake agent that I'm using today is a financial analysis chatbot uh using the cloud agent SDK.

s384
00:29:16.399 --> 00:29:22.240
You give it a stock ticker and a focus area and it searches the web for the latest information about that stock ticker.

s385
00:29:22.240 --> 00:29:26.240
Uh, researches real current financial data and writes a report for you.

s386
00:29:26.240 --> 00:29:29.600
Uh, this is a a real use case that whole startups are built around.

s387
00:29:29.600 --> 00:29:32.960
Although uh my agent version is extremely simple.

s388
00:29:32.960 --> 00:29:36.960
Um, I'm assuming uh that you built an agent already.

s389
00:29:36.960 --> 00:29:39.200
Our agent works in two turns.

s390
00:29:39.200 --> 00:29:43.760
Um there's first a research turn that uses tools to search the web and gather data.

s391
00:29:43.760 --> 00:29:48.799
Then there is a writing turn that compiles the research into a readable report.

s392
00:29:48.799 --> 00:29:51.840
Uh so let me walk you through that.

s393
00:29:51.840 --> 00:29:54.240
Uh here is my research prompt and my write prompt.

s394
00:29:54.240 --> 00:29:55.840
These are deliberately extremely simple.

s395
00:29:55.840 --> 00:30:00.880
They are going to cause errors later and those are the things that we are going to use our evals to debug.

s396
00:30:00.880 --> 00:30:03.840
Uh and then I'm setting up my claude agent options.

s397
00:30:03.840 --> 00:30:13.679
I'm using uh Claude Haiku uh as the agent underneath because Haiku is you know capable but it will make mistakes and we want some mistakes so that we can fix them.

s398
00:30:13.679 --> 00:30:19.919
Uh and it's also fast and cheap so I don't burn a lot of money uh of your money when as we do this.

s399
00:30:19.919 --> 00:30:27.840
Um the permission mode controls uh what the agent is allowed to do autonomously.

s400
00:30:27.840 --> 00:30:33.600
There is a catch hiding in the in the specific choices that I have made and the specific allowed tools.

s401
00:30:33.600 --> 00:30:38.080
uh that I am giving this agent right now as we're going to find out later.

s402
00:30:38.080 --> 00:30:40.799
Uh and now let's look at our two turns.

s403
00:30:40.799 --> 00:30:44.799
Um turn number one is research.

s404
00:30:44.799 --> 00:30:51.600
There is a bunch of boiler plate in here that I'm not going to explain which is just about outputting the output so that we can see what it's doing as it's going.

s405
00:30:51.600 --> 00:30:56.480
Uh turn two is writing the report with again a pile of boiler plate.

s406
00:30:56.480 --> 00:31:00.240
Um the prompts are critical.

s407
00:31:00.240 --> 00:31:04.000
So this is where I want you to start doing your own changes in your own notebook.

s408
00:31:04.000 --> 00:31:06.720
I have given it extremely basic prompts at the top.

s409
00:31:06.720 --> 00:31:15.039
If you can see if you can think of a better prompt just off the top of your head uh that is going to do a better job than the prompts that I put in

s410
00:31:15.039 --> 00:31:16.799
uh at solving this problem.

s411
00:31:16.799 --> 00:31:21.440
What would be better than what I put in to uh do the research in the first place?

s412
00:31:21.440 --> 00:31:26.159
What would be better than what I put in uh to do the report writing?

s413
00:31:26.159 --> 00:31:32.640
Um, and if you oneshot it, then you're going to have uh less stuff to do later on.

s414
00:31:32.640 --> 00:31:40.240
Um, one of the things we do here is we wrap the whole thing in an open telemetry span.

s415
00:31:40.240 --> 00:31:51.120
Uh, this is because if I don't do that, then claude inter Claude sends this as two the two turns as two separate spans and we wanted them to be grouped together in the UI.

s416
00:31:51.120 --> 00:31:55.840
Uh, so I've wrapped them in a span together so that they come out as a single agent turn.

s417
00:31:55.840 --> 00:31:58.399
Um, this is just saving us some time later.

s418
00:31:58.399 --> 00:32:02.960
I wanted to be clear about why I'm doing this uh bit of code.

s419
00:32:02.960 --> 00:32:06.240
Um, and now we can run our agents.

s420
00:32:06.240 --> 00:32:10.640
Uh, I have run my agent in advance because I wanted to know exactly what it was going to do.

s421
00:32:10.640 --> 00:32:15.840
Um, but now is the time to kick off your own agent.

s422
00:32:15.840 --> 00:32:21.279
Um, I've asked it to analyze Tesla, specifically its financial performance and growth outlook.

s423
00:32:21.279 --> 00:32:24.000
Um, you should go ahead and run this cell.

s424
00:32:24.000 --> 00:32:27.039
It'll take a minute or two because the agent will actually search the web.

s425
00:32:27.039 --> 00:32:30.000
Uh, and the LLM is doing multiple rounds of reasoning.

s426
00:32:30.000 --> 00:32:33.200
Um, the agent receives our research prompt.

s427
00:32:33.200 --> 00:32:35.679
It decides it needs to search the web, which is a tool call.

s428
00:32:35.679 --> 00:32:36.880
It reads the search results.

s429
00:32:36.880 --> 00:32:40.559
It decides if it has enough information and maybe searches again.

s430
00:32:40.559 --> 00:32:42.080
That is the agentic part.

s431
00:32:42.080 --> 00:32:47.519
You didn't say do a web search, take the web information from the web search and write a report about it.

s432
00:32:47.519 --> 00:32:50.559
The agent is deciding is this web search enough information?

s433
00:32:50.559 --> 00:32:53.120
And if not, I will write I will run more web searches.

s434
00:32:53.120 --> 00:32:59.919
It will run an in indeterminate number of web searches to get enough information until it has decided what enough means.

s435
00:32:59.919 --> 00:33:02.880
Uh and then it will write the report.

s436
00:33:02.880 --> 00:33:07.600
This execution path is non-deterministic by design and that is the key here to understand.

s437
00:33:07.600 --> 00:33:11.600
Uh if you run this again with the same input, your agent might search for different things.

s438
00:33:11.600 --> 00:33:12.880
It might find different results.

s439
00:33:12.880 --> 00:33:14.799
It might write a different report.

s440
00:33:14.799 --> 00:33:17.039
Uh and both reports might be good.

s441
00:33:17.039 --> 00:33:18.880
Um or one might be good and one might be bad.

s442
00:33:18.880 --> 00:33:21.039
And that is exactly why we need evals.

s443
00:33:21.039 --> 00:33:23.600
We can't predict the output from the input alone.

s444
00:33:23.600 --> 00:33:28.480
Uh, and every single one of those decisions is being captured as a trace.

s445
00:33:28.480 --> 00:33:38.159
Uh, so I'm just going to pop open my last 12 hours or my last 24 hours, I guess.

s446
00:33:38.159 --> 00:33:39.120
Cool.

s447
00:33:39.120 --> 00:33:40.399
This is a pile of traces.

s448
00:33:40.399 --> 00:33:45.200
You're going to understand what all of this is later, but here you can see the Tesla one that I ran earlier.

s449
00:33:45.200 --> 00:33:47.360
This is what it looks like when I go through.

s450
00:33:47.360 --> 00:33:49.760
You can see every single turn.

s451
00:33:49.760 --> 00:33:51.600
Uh it's calling a skill.

s452
00:33:51.600 --> 00:33:52.880
It's doing a tool search.

s453
00:33:52.880 --> 00:33:54.799
It's doing a web search.

s454
00:33:54.799 --> 00:33:57.679
Uh you can see every single thing that it does.

s455
00:33:57.679 --> 00:34:01.840
You can see that it ran four web searches before finally deciding that was enough information.

s456
00:34:01.840 --> 00:34:06.080
Then it did the second uh call in the step and it wrote the report.

s457
00:34:06.080 --> 00:34:15.839
Uh one of the things that it does uh that we're going to see later is uh I told you there was a catch in the tools that I gave it.

s458
00:34:15.839 --> 00:34:20.879
Uh here's this step that was unexpected when we were putting this demo together.

s459
00:34:20.879 --> 00:34:22.399
It tried to write the report.

s460
00:34:22.399 --> 00:34:23.599
I told it to write a report.

s461
00:34:23.599 --> 00:34:26.240
It tried to write the report as a markdown file to disk.

s462
00:34:26.240 --> 00:34:27.760
It is living in a collab notebook.

s463
00:34:27.760 --> 00:34:31.440
So there is no uh there's no file system for it to write to.

s464
00:34:31.440 --> 00:34:33.760
Uh so the write step would fail.

s465
00:34:33.760 --> 00:34:36.000
Uh and this is something that you need evals for.

s466
00:34:36.000 --> 00:34:41.200
You need evals to detect when your agent is doing something helpful but incorrect.

s467
00:34:41.200 --> 00:34:43.520
Uh because this fails silently.

s468
00:34:43.520 --> 00:34:52.159
It gives you output anyway, but it's doing this unnecessary extra step of trying to write to disk uh without you knowing that it was there.

s469
00:34:52.159 --> 00:34:59.280
Um, so back to the collab, this is a pretty good report.

s470
00:34:59.280 --> 00:35:06.880
Um, it's a financial performance table with real numbers, growth drivers, risk factors, analyst price targets, and a conclusion.

s471
00:35:06.880 --> 00:35:11.040
Um, it's not bad for a first pass, but it not bad as a vibe.

s472
00:35:11.040 --> 00:35:14.160
And we are here to replace vibes with actual numbers.

s473
00:35:14.160 --> 00:35:25.280
Um, so like I said, uh, you can see all of that stuff in the uh, traces.

s474
00:35:25.280 --> 00:35:27.839
We didn't have to do anything to get all of that information.

s475
00:35:27.839 --> 00:35:32.320
We didn't have to like instrument every single line to get the LLM call and get the tool call and all of that stuff.

s476
00:35:32.320 --> 00:35:34.240
All of that is built into cloud agent SDK.

s477
00:35:34.240 --> 00:35:37.680
just it is like it's built into open AIS SDK.

s478
00:35:37.680 --> 00:35:42.160
Uh and each one of these rows in this table is a span.

s479
00:35:42.160 --> 00:35:46.079
Um traces reveal every decision the agent made.

s480
00:35:46.079 --> 00:35:49.200
Without them, all you see is I gave it a prompt and I got a report.

s481
00:35:49.200 --> 00:35:51.599
With traces, you see every step.

s482
00:35:51.599 --> 00:35:57.920
Uh if you click into any span, like I said, you'll see exactly what the model received as input and exactly what it returned.

s483
00:35:57.920 --> 00:36:04.240
Um, and that is what observability means in practice.

s484
00:36:04.240 --> 00:36:09.119
Not just did it work, uh, but how did it work and where exactly did it go wrong?

s485
00:36:09.119 --> 00:36:16.320
Uh, when we run our eval in a few minutes, uh, we're going to pull the input and attribute output attributes out of these spans and feed them to our evaluators.

s486
00:36:16.320 --> 00:36:18.400
The input becomes the evaluator's input.

s487
00:36:18.400 --> 00:36:23.040
The output uh, becomes what gets graded.

s488
00:36:26.400 --> 00:36:30.320
Uh now we need to generate some test data to run meaningful evals.

s489
00:36:30.320 --> 00:36:33.119
Uh we need more than the one trace that you've generated so far.

s490
00:36:33.119 --> 00:36:34.480
We need a body of data.

s491
00:36:34.480 --> 00:36:39.760
Uh in the notebook there are 12 test queries covering different tickers and analysis types.

s492
00:36:39.760 --> 00:36:41.359
Uh you should kick those off now.

s493
00:36:41.359 --> 00:36:46.640
They take about 5 to 10 minutes to run assuming that you've got Wi-Fi and everything's working for you.

s494
00:36:46.640 --> 00:36:50.640
Um I'm going to show you what they look like.

s495
00:36:50.640 --> 00:36:58.960
Uh and oops this is them here.

s496
00:37:01.040 --> 00:37:04.800
So the key thing here is uh diversity.

s497
00:37:04.800 --> 00:37:08.160
I've given it a bunch of obvious queries that should work.

s498
00:37:08.160 --> 00:37:09.839
I've also given it some edge cases.

s499
00:37:09.839 --> 00:37:16.079
For instance, I've given it one where I gave it two tickers, Apple and Microsoft, and told it to run a comparison.

s500
00:37:16.079 --> 00:37:19.359
Uh which is something it could theoretically do.

s501
00:37:19.359 --> 00:37:30.720
uh I've also given it the same uh tickers multiple times uh in places so that we can see uh the non-deterministic output uh when we say uh Amazon profitability

s502
00:37:30.720 --> 00:37:51.119
trends and outlook versus Amazon AWS performance and profitability um different tickers and different questions different levels of complexity are important uh to generate test data that's going to cover the range of things that users will actually ask although spoiler alert in reality your users are going to ask really weird things that you're not going to be able

s503
00:37:51.119 --> 00:37:55.520
to predict in advance and that is another one of the reasons that eval are important.

s504
00:37:55.520 --> 00:38:10.640
Um the uh Rivian query rivn uh asks about a company with much less public data than Apple or Microsoft and the Coca-Cola dividend yield query is a very different kind of analysis from a growth growth stock query.

s505
00:38:10.640 --> 00:38:14.160
Um again this is diversity that is important.

s506
00:38:14.160 --> 00:38:21.760
Um, diversity in your test set is how you catch uh things that your agent is going to be not good at.

s507
00:38:22.160 --> 00:38:29.119
Uh, so while yours are running, mine have already loaded so I can show you all of my data.

s508
00:38:29.119 --> 00:38:33.599
Uh, these are all of my spans from all of those runs.

s509
00:38:33.599 --> 00:38:41.920
Um, you can see uh at the bottom there are 13 uh the which is the initial Tesla run and 12 test queries.

s510
00:38:41.920 --> 00:38:45.680
Each one is a complete execution of the cloud uh financial analyst.

s511
00:38:45.680 --> 00:38:53.040
So now we have the data but before we write any evals uh we need to actually uh look at the data.

s512
00:38:53.040 --> 00:38:56.640
Um this is error analysis.

s513
00:38:56.640 --> 00:38:58.720
Uh and the whole step is one instruction.

s514
00:38:58.720 --> 00:39:01.200
Read your traces before you write eval.

s515
00:39:01.200 --> 00:39:04.800
Before you write a single evaluator start with your data.

s516
00:39:04.800 --> 00:39:07.680
Uh read or a dozen or more of your traces end to end.

s517
00:39:07.680 --> 00:39:11.119
This is the most important practice in this entire workshop.

s518
00:39:11.119 --> 00:39:13.680
uh focus on the ones where something went wrong.

s519
00:39:13.680 --> 00:39:14.560
What was the input?

s520
00:39:14.560 --> 00:39:15.599
What was the output?

s521
00:39:15.599 --> 00:39:17.839
What specifically is broken?

s522
00:39:17.839 --> 00:39:19.680
This sounds extremely old-fashioned.

s523
00:39:19.680 --> 00:39:21.119
Just read the input.

s524
00:39:21.119 --> 00:39:25.280
Uh but it is one of the highest value activities in agent development.

s525
00:39:25.280 --> 00:39:34.320
Uh Anthropic, for instance, invested in tooling specifically for viewing eval transcripts and their team spends a whole lot of time just looking at eval transcripts every day.

s526
00:39:34.320 --> 00:39:40.880
Um, a trace tells you whether the agent made a genuine mistake or whether your graders rejected a valid solution.

s527
00:39:40.880 --> 00:39:50.720
Uh, if you automate before you understand your failures, then you're going to create an eval that measures what's easy to measure instead of what actually matters.

s528
00:39:54.000 --> 00:39:57.119
You need requirements first before you can categorize failures.

s529
00:39:57.119 --> 00:39:58.720
You need to know what success looks like.

s530
00:39:58.720 --> 00:40:03.359
You can't say it doesn't work if you haven't defined what it works means.

s531
00:40:03.359 --> 00:40:07.359
Uh for our financial analyst, what does a good report look like?

s532
00:40:07.359 --> 00:40:09.440
Uh it should reference the correct ticker.

s533
00:40:09.440 --> 00:40:15.599
It should include some re real recent actionable financial data uh and include actionable recommendations.

s534
00:40:15.599 --> 00:40:18.880
It should distinguish between forward-looking analysis and historical summary.

s535
00:40:18.880 --> 00:40:26.000
Those are our success criteria and they seem obvious uh when you write them down, but lots of teams never do.

s536
00:40:26.000 --> 00:40:31.119
Um they ship an agent and then react to complaints instead of defining the bar up front.

s537
00:40:31.119 --> 00:40:40.480
Uh, writing requirements down turns vague disappointment into specific testable criteria and those criteria are what turn into your evals.

s538
00:40:40.480 --> 00:40:48.720
Here's the thing about defining success that it's important to note though, which is that it is not something that your engineers can do or not something your engineers can do alone.

s539
00:40:48.720 --> 00:40:57.520
The definition of good usually lives in the domain knowledge and the domain knowledge lives in the people that your company will often laughably refer to as non-technical.

s540
00:40:57.520 --> 00:41:03.920
So your product managers, your QA, your support team, those are the people who know what the definition of good really is.

s541
00:41:03.920 --> 00:41:06.319
They know what the failures are going to look like.

s542
00:41:06.319 --> 00:41:09.440
Those are the people who should be helping you write your evals.

s543
00:41:09.440 --> 00:41:11.920
Uh OpenAI put it really nicely.

s544
00:41:11.920 --> 00:41:14.640
They said that people management skills are AI skills.

s545
00:41:14.640 --> 00:41:23.839
So clear goals, uh direct feedback, knowing what your value proposition is, uh those skills matter more than ever when the system is probabilistic.

s546
00:41:23.839 --> 00:41:26.160
Uh and this is one of the things that AX is built around.

s547
00:41:26.160 --> 00:41:31.359
Anyone on your team can read traces or add annotations and contribute to your criteria.

s548
00:41:31.359 --> 00:41:33.520
This is not just an engineering surface.

s549
00:41:33.520 --> 00:41:38.000
You're expected to have other members of your team in here.

s550
00:41:38.240 --> 00:41:40.720
A quick note on where to get this test data.

s551
00:41:40.720 --> 00:41:44.640
Uh we have these traces because we already ran the agent ourselves.

s552
00:41:44.640 --> 00:41:48.000
But what if you're building something new and you don't have real traffic yet?

s553
00:41:48.000 --> 00:41:51.119
Uh you can do what we just did, which is you can use synthetic data.

s554
00:41:51.119 --> 00:41:55.280
You can have an LLM generate diverse queries across your expected categories.

s555
00:41:55.280 --> 00:41:59.839
So, uh, research Tesla financial rep performance is one phrasing.

s556
00:41:59.839 --> 00:42:03.520
What's going on with Tesla stock is another query that's asking the same thing.

s557
00:42:03.520 --> 00:42:08.240
Yo, is Tesla a buy right now is a third way of expressing the same query.

s558
00:42:08.240 --> 00:42:10.880
Uh, they're all the same intent, but they look very different.

s559
00:42:10.880 --> 00:42:14.960
So, vary the phrasing, the complexity, the level of specificity.

s560
00:42:14.960 --> 00:42:18.400
Uh, a domain expert should review your set of queries.

s561
00:42:18.400 --> 00:42:25.920
Uh, because LLM generated queries tend to cluster around obvious phrasings and miss the weird stuff that real users write.

s562
00:42:25.920 --> 00:42:28.400
You also need to include edge cases in your test data.

s563
00:42:28.400 --> 00:42:32.640
So things like non-existent tickers, multi-part questions, jailbreak attempts.

s564
00:42:32.640 --> 00:42:39.520
Uh those might be 1% of your traffic, but they are the 1% of your traffic that ends up, you know, being a PR disaster in the press.

s565
00:42:39.520 --> 00:42:44.319
Your test data should look like production data, not what you wish production looked like.

s566
00:42:44.319 --> 00:42:51.760
Uh so synthetic data can get you started, but as soon as you have production data, production data should be what you're running on.

s567
00:42:52.000 --> 00:42:55.839
So let's actually examine our traces.

s568
00:42:55.920 --> 00:43:01.680
Uh like I said, the first 13 traces are the first 13 traces that I wrote.

s569
00:43:01.680 --> 00:43:06.160
Um most of them produced long structured reports in line.

s570
00:43:06.160 --> 00:43:09.119
So Tesla, Apple, Nvidia, stuff like that.

s571
00:43:09.119 --> 00:43:18.640
Uh 3 to 7,000 characters uh of report right there in the trace output with executive summaries, valuation tables, explicit buy or hold recommendations.

s572
00:43:18.640 --> 00:43:29.599
On the surface, most of these look fine, but like I said, three of the 13 were doing something weird, which was uh they were uh the rep output in them is much shorter

s573
00:43:29.599 --> 00:43:32.720
because it just says, "Oh, I wrote this to disk for you."

s574
00:43:32.720 --> 00:43:36.560
Uh and the output went to disk.

s575
00:43:36.560 --> 00:43:45.440
Uh so instead of having a full report, uh it has a summary of the report and the the actual body of the report went to the non-existent disk.

s576
00:43:45.440 --> 00:43:48.079
This is what the eval is designed to find.

s577
00:43:48.079 --> 00:43:52.400
Um the agent decided to write a file with no permission.

s578
00:43:52.400 --> 00:43:55.280
The write silently failed and it told us the report was saved.

s579
00:43:55.280 --> 00:44:00.240
Uh and that is the kind of pattern that you only see if you actually read the trace.

s580
00:44:00.240 --> 00:44:05.359
Um so this is I believe one of the three sources.

s581
00:44:05.359 --> 00:44:08.400
Nope, that one had worked.

s582
00:44:09.520 --> 00:44:10.800
Oh no, the Wi-Fi.

s583
00:44:10.800 --> 00:44:13.119
Here we go.

s584
00:44:24.880 --> 00:44:28.240
So, this one's showing another kind of failure where it just got stuck in a loop forever.

s585
00:44:28.240 --> 00:44:31.839
It was trying to find information about who was it trying to find information about?

s586
00:44:31.839 --> 00:44:32.720
Microsoft.

s587
00:44:32.720 --> 00:44:36.560
Uh, and it just did an endless series of web searches and took forever.

s588
00:44:36.560 --> 00:44:41.839
Um, this is what it tried to write to disk.

s589
00:44:41.839 --> 00:44:45.119
Uh, and this is what it actually wrote to output.

s590
00:44:45.119 --> 00:44:46.640
Uh, right here we go.

s591
00:44:46.640 --> 00:44:50.720
This is the report is saved as Microsoft cloud segment financial report.mmd.

s592
00:44:50.720 --> 00:44:54.400
This report is useless because it's referring to a report that doesn't exist.

s593
00:44:54.400 --> 00:45:00.720
Um, there's also uh an example of a confidently wrong.

s594
00:45:00.720 --> 00:45:08.319
If we go to the Rivian report uh which is RIBN, where did it go?

s595
00:45:08.319 --> 00:45:09.839
There we go.

s596
00:45:09.839 --> 00:45:24.319
Uh, Rivian has the same problem that the Microsoft one does, but even inside of the full report that it tried to write to disk, there's a whole bunch of really confident information about a private company that you can't possibly verify.

s597
00:45:24.319 --> 00:45:33.680
Uh, these might be hallucinations, they might not be hallucinations, but we haven't done anything to check whether or not uh this report is really grounded in reality.

s598
00:45:33.680 --> 00:45:37.599
Uh, so we're going to do an eval about that later.

s599
00:45:37.680 --> 00:45:43.280
There is a structured way to do this reading uh which as I mentioned earlier is called coding.

s600
00:45:43.280 --> 00:45:48.240
There's open coding and aial coding uh and it comes from qualitative research.

s601
00:45:48.240 --> 00:45:52.000
Open coding means you read each trace and you write down what you see.

s602
00:45:52.000 --> 00:45:56.960
So stuff like vague recommendation made up a number talked about the wrong ticker.

s603
00:45:56.960 --> 00:45:58.960
Uh you're not trying to be neat.

s604
00:45:58.960 --> 00:46:07.520
As a programmer, it is very uh tempting to try and come up with categories in advance and say, "Oh, this is a you know, retrieval failure or whatever."

s605
00:46:07.520 --> 00:46:10.160
Uh don't do that when you're doing open coding.

s606
00:46:10.160 --> 00:46:14.079
Just write down what the problem was in as natural language as possible.

s607
00:46:14.079 --> 00:46:23.440
Uh and uh then when you find the failure, sorry, and then uh you want to do aial coding afterwards.

s608
00:46:23.440 --> 00:46:29.599
AIAL coding is when you look at uh all of the open codes that you've put together and you then you do the categorization.

s609
00:46:29.599 --> 00:46:32.000
You say okay now there are five things that look similar.

s610
00:46:32.000 --> 00:46:37.359
I'm going to give them all a single category and uh turn them into categories.

s611
00:46:37.359 --> 00:46:44.319
Um the second pass the the uh aial coding pass is structural.

s612
00:46:44.319 --> 00:46:51.760
Uh and like I said the tend the tendency the temptation is to try and go straight to a coding and you should resist that.

s613
00:46:51.760 --> 00:46:54.800
When you find a failure, you should ask why it failed.

s614
00:46:54.800 --> 00:46:56.880
So the response was wrong is a symptom.

s615
00:46:56.880 --> 00:46:58.800
Did it get bad search results?

s616
00:46:58.800 --> 00:47:00.240
That is a retrieval failure.

s617
00:47:00.240 --> 00:47:03.200
Did it get the right data uh but the wrong conclusion?

s618
00:47:03.200 --> 00:47:04.800
Then that is a reasoning error.

s619
00:47:04.800 --> 00:47:06.079
Did it make up a stock price?

s620
00:47:06.079 --> 00:47:07.520
That's obviously hallucination.

s621
00:47:07.520 --> 00:47:09.440
Did it answer outside of its domain?

s622
00:47:09.440 --> 00:47:10.720
That is a scope violation.

s623
00:47:10.720 --> 00:47:14.000
Did it do something that you didn't tell it that it should be able to do.

s624
00:47:14.000 --> 00:47:18.720
Each root cause, each axial code uh points to a different fix.

s625
00:47:18.720 --> 00:47:23.599
If you don't know the right call, if you don't know the cause, you can't pick the right remedy.

s626
00:47:23.599 --> 00:47:37.680
Once you've categorized uh which I did here, uh you can sum them up into uh a table.

s627
00:47:37.760 --> 00:47:44.480
So, uh confession time, I didn't actually read all of those traces and code them.

s628
00:47:44.480 --> 00:47:46.640
I just gave them random axial codes.

s629
00:47:46.640 --> 00:47:56.400
Uh but what you end up with is this table of root cause frequency things like looks good possible hallucination reasoning gap unverifiable data missing recommendation.

s630
00:47:56.400 --> 00:48:02.960
It's tempting to look at the top one the possible hoodation and decide that that is the most important one to fix.

s631
00:48:02.960 --> 00:48:12.560
But in reality uh you're going to want to uh balance between uh frequency and severity.

s632
00:48:12.560 --> 00:48:23.760
So if one time in a hundred instead of doing a financial report, it gives the user instructions on how to make a bomb, you fix that one first because that is the most severe possible failure.

s633
00:48:23.760 --> 00:48:30.559
Uh so you have to uh multiply the severity by the frequency to get to which ones you decide to fix first.

s634
00:48:30.559 --> 00:48:35.760
Uh and that is why you look at the data before you write your evals.

s635
00:48:36.000 --> 00:48:41.200
Uh one more concept before we get started on that is the Swiss cheese model.

s636
00:48:41.200 --> 00:48:43.760
um which is a concept from safety engineering.

s637
00:48:43.760 --> 00:48:47.599
Imagine each layer of defense as a slice of Swiss cheese.

s638
00:48:47.599 --> 00:48:51.359
Each slice has holes uh gaps where problems can slip through.

s639
00:48:51.359 --> 00:48:54.319
Uh but if you stack enough slices, the holes don't line up.

s640
00:48:54.319 --> 00:48:56.960
So what gets through one layer gets caught by the next.

s641
00:48:56.960 --> 00:49:00.240
Uh your code eval catches format issues but misses semantic problems.

s642
00:49:00.240 --> 00:49:04.319
Your LLM judge catches reasoning gaps but misses subtle hallucinations.

s643
00:49:04.319 --> 00:49:08.559
And your human review catches the subtle stuff but it can't scale to every trace.

s644
00:49:08.559 --> 00:49:14.960
No single eval in that set uh catches everything and the combination is what gives you coverage.

s645
00:49:14.960 --> 00:49:18.400
Uh so stack the cheese.

s646
00:49:18.400 --> 00:49:23.520
Um now that we know what's wrong, let's automate the checking.

s647
00:49:23.520 --> 00:49:26.720
Um the next step is code evals.

s648
00:49:26.720 --> 00:49:30.880
This is no model configuration, no API calls, uh just Python.

s649
00:49:30.880 --> 00:49:34.800
We're going to write the simplest useful eval that I can think of.

s650
00:49:34.800 --> 00:49:38.240
Uh, our agent is supposed to analyze a specific stock ticker.

s651
00:49:38.240 --> 00:49:43.839
What I'm going to do is write a Kodi valve that checks whether that stock ticker actually showed up in the report.

s652
00:49:43.839 --> 00:49:48.800
Uh, which is a test that my initial demos absolutely failed a number of times.

s653
00:49:48.800 --> 00:49:51.040
Uh, an LLM judge would be overkill for this.

s654
00:49:51.040 --> 00:49:52.480
You're looking for a specific string.

s655
00:49:52.480 --> 00:49:53.680
It's somewhere in the report.

s656
00:49:53.680 --> 00:49:58.240
You don't need an LLM to judge whether or not you mentioned the ticker.

s657
00:49:58.800 --> 00:50:01.760
So, there's two ways to do this.

s658
00:50:01.760 --> 00:50:03.520
There's two ways to do everything in AX.

s659
00:50:03.520 --> 00:50:07.200
one is programmatically uh and one is via the UI.

s660
00:50:07.200 --> 00:50:12.000
Uh in the uh notebook you can see the programmatic way.

s661
00:50:12.000 --> 00:50:16.880
What I'm going to show you on screen is how you do it in the UI.

s662
00:50:19.200 --> 00:50:22.319
So this big button in the top right is what you want.

s663
00:50:22.319 --> 00:50:23.839
You want to add an evaluator.

s664
00:50:23.839 --> 00:50:33.280
In this case, I've already added my mentions evaluator, but you will want to add an evaluator which will give you this screen.

s665
00:50:33.280 --> 00:50:38.319
Uh here you're going to see basically some Python.

s666
00:50:38.319 --> 00:50:47.839
Um in the notebook I've given you the Python that you can use uh to either do your uh programmatic version or your version in the UI.

s667
00:50:47.839 --> 00:51:00.079
Uh this is a very simple Python function which returns uh an evaluation result which is looks exactly like those evaluation results that I showed you earlier.

s668
00:51:00.079 --> 00:51:02.800
Uh it has a label, it has a score.

s669
00:51:02.800 --> 00:51:18.000
Um so uh once you've written what your code evaluator does, you're going to have uh parameters to your code evaluator.

s670
00:51:18.000 --> 00:51:20.160
In this case, I've called them query and report.

s671
00:51:20.160 --> 00:51:26.880
And you're going to have to tell AX uh which variables in your trace are those variables.

s672
00:51:26.880 --> 00:51:45.599
So there's this extremely fun UI, the single trace UI, uh where you look at a single trace, and you can literally just click uh the specific thing in your output that you want to be that variable, and you tell it, okay, that one should be the query, and that one should be the report, and then it's mapped.

s673
00:51:47.280 --> 00:51:57.760
Um, you should also uh be changing your evaluator to uh work on uh a trace as opposed to a span.

s674
00:51:57.760 --> 00:52:00.160
That is what gives you that UI.

s675
00:52:00.160 --> 00:52:06.800
Um, once you've done that, you need to save your code evaluator to our evaluator hub.

s676
00:52:06.800 --> 00:52:11.520
Uh, and then you hit save to save the evaluation.

s677
00:52:17.280 --> 00:52:20.319
So assuming you've done that.

s678
00:52:20.319 --> 00:52:22.319
Yeah.

s679
00:52:28.480 --> 00:52:30.079
Sure.

s680
00:52:30.079 --> 00:52:33.920
Uh so he asked for more detail on how code evaluators work.

s681
00:52:33.920 --> 00:52:37.760
Uh so code evaluators are they're completely deterministic functions.

s682
00:52:37.760 --> 00:52:43.280
You're just taking input and output and saying was this output good according to some definition of good.

s683
00:52:43.280 --> 00:52:47.520
So in this code uh I'm doing very very simple Python.

s684
00:52:47.520 --> 00:52:52.720
I'm just saying uh look for all of the words that are in capital letters.

s685
00:52:52.720 --> 00:53:02.800
Uh strip out the ones that are obvious acronyms that I already know and of the remaining words are any of them the uh the acronym which is the stock ticker that I'm looking for?

s686
00:53:02.800 --> 00:53:04.319
That's all it's doing.

s687
00:53:04.319 --> 00:53:07.359
Uh lots of code evaluators are even shorter than that.

s688
00:53:07.359 --> 00:53:11.200
They're something like is this output 500 characters long or less?

s689
00:53:11.200 --> 00:53:14.160
Uh is this output parsible JSON?

s690
00:53:14.160 --> 00:53:23.440
You're doing something very simple, very deterministic uh for the purposes of having that first line of defense in your Swiss cheese.

s691
00:53:23.760 --> 00:53:31.200
Um so I talked about how to add stuff already.

s692
00:53:31.200 --> 00:53:37.839
Uh in the notebook you can run the eval programmatically.

s693
00:53:38.000 --> 00:53:43.520
Uh this is how to do this is these are the instructions on how to do it via the UI if you followed that.

s694
00:53:43.520 --> 00:53:51.040
The alternative is to do it in the code where you can just run it directly which is uh easier to do if you're following along in a notebook.

s695
00:53:51.040 --> 00:53:55.359
Um, a code evaluator returns a label and a score.

s696
00:53:55.359 --> 00:53:56.960
Like I said, there's no LLM needed.

s697
00:53:56.960 --> 00:53:57.920
There's no API call.

s698
00:53:57.920 --> 00:53:59.040
It's instant.

s699
00:53:59.040 --> 00:54:12.400
Uh, and uh if you're wondering why we're wrapping it in suppressed tracing, it's because uh the evaluation is itself uh a call to the uh agent SDK which would then get

s700
00:54:12.400 --> 00:54:17.520
stored as traces and I didn't want traces of my tracing happening because that gets too meta.

s701
00:54:17.520 --> 00:54:21.599
Uh so I told it to suppress tracing when I'm running an evaluation.

s702
00:54:21.599 --> 00:54:28.559
Uh if you look at this programmatic run uh you will see that 12 passed and one failed.

s703
00:54:28.559 --> 00:54:31.119
Uh there was a missing ticker for Amazon.

s704
00:54:31.119 --> 00:54:39.839
Uh the specific trace that went wrong uh is when I asked you'll note you'll notice I told you earlier that I made two queries to Amazon.

s705
00:54:39.839 --> 00:54:45.119
One where I asked about its profitability and one I asked were about uh AWS's profitability.

s706
00:54:45.119 --> 00:55:00.960
Uh if you click into the traces uh which I could attempt now um you'll see that the report that asked about AWS uh wrote a report that was entirely about AWS

s707
00:55:00.960 --> 00:55:05.599
only about the AWS branch of Amazon rather than uh all of Amazon.

s708
00:55:05.599 --> 00:55:08.960
Uh and so it failed to mention the Amazon stock ticker entirely.

s709
00:55:08.960 --> 00:55:10.880
It talked only about AWS.

s710
00:55:10.880 --> 00:55:13.520
So it's a really subtle failure because it did a bunch of research.

s711
00:55:13.520 --> 00:55:16.240
It found out a whole bunch of stuff about what was happening at Amazon.

s712
00:55:16.240 --> 00:55:19.920
It just didn't think about what was happening about Amazon as a company.

s713
00:55:19.920 --> 00:55:26.480
Uh only about AWS because I gave it that extra prompt like I also want to know about AWS profitability.

s714
00:55:26.480 --> 00:55:38.000
Um so uh looking at those results back in AX um they show up as annotations next to our traces.

s715
00:55:38.000 --> 00:55:43.280
So you can click into any one of these traces uh and you'll see evaluations.

s716
00:55:43.280 --> 00:55:51.359
I've run a lot more evaluations uh than you have so far, but you can see my mention sticker evaluation has run with the span.

s717
00:55:51.359 --> 00:55:52.559
It's run at the span level.

s718
00:55:52.559 --> 00:55:56.079
It has hit a pass uh and it's given a score of one.

s719
00:55:56.079 --> 00:56:02.240
Um the notebook has a little helper called log eval to ax which is doing that work for you.

s720
00:56:02.240 --> 00:56:08.960
takes uh the spans that it ran offline and the tests that it ran offline and pushes them back to AX for you.

s721
00:56:08.960 --> 00:56:22.240
Um so now every score now every eval has uh a mention sticker evaluation uh that you can look at and filter by uh and look for the failing failing ones.

s722
00:56:22.240 --> 00:56:23.520
So why does this matter?

s723
00:56:23.520 --> 00:56:25.520
Because it is your first line of defense.

s724
00:56:25.520 --> 00:56:35.119
Uh if your financial analyst agent writes a beautiful report about m Microsoft when the user asked about Tesla uh which is a thing that actually happened to me when I was building this

s725
00:56:35.119 --> 00:56:40.559
uh then no amount of answer of eloquence from your agent uh matters because the answer is wrong.

s726
00:56:40.559 --> 00:56:43.920
Uh so you can catch this with five lines of Python.

s727
00:56:43.920 --> 00:56:51.599
Um the ticker check catches uh really basic failures but it catches them really cheaply.

s728
00:56:51.599 --> 00:56:53.280
That is why it is important.

s729
00:56:53.280 --> 00:56:56.079
Um, Kodi valves aren't just toy examples.

s730
00:56:56.079 --> 00:57:05.599
Often, you're going to want to know that your output is JSON, uh, or that it has a certain length, or you're going to want to avoid forbidden phrases like as an AI language model.

s731
00:57:05.599 --> 00:57:07.920
Uh, and those are production critical checks.

s732
00:57:07.920 --> 00:57:10.720
A code eval doesn't have to be a simple string operation.

s733
00:57:10.720 --> 00:57:14.079
Kodval means the grading logic is deterministic.

s734
00:57:14.079 --> 00:57:15.839
Uh, but it can do complicated things.

s735
00:57:15.839 --> 00:57:17.200
It can query a database for you.

s736
00:57:17.200 --> 00:57:22.559
It could hit an API and get the actual stock price to make sure that it wasn't hallucinating a stock price.

s737
00:57:22.559 --> 00:57:29.359
uh anything with a grading always gives the same answer for the same input can be a code evaluator.

s738
00:57:29.520 --> 00:57:34.000
One important principle is to grade what the agent produced and not the path that it took.

s739
00:57:34.000 --> 00:57:36.079
Uh I referred to this earlier.

s740
00:57:36.079 --> 00:57:40.079
There's a common instinct to check that the agent followed a specific sequence of steps.

s741
00:57:40.079 --> 00:57:43.599
Did it call this tool first and then that tool in that order?

s742
00:57:43.599 --> 00:57:48.960
Uh and uh practitioners have found this to be in practice too rigid.

s743
00:57:48.960 --> 00:57:54.480
uh agents regularly find valid approaches uh that the eval designer didn't anticipate.

s744
00:57:54.480 --> 00:58:00.640
Uh so uh if you grade the path um you punish creativity.

s745
00:58:00.640 --> 00:58:02.559
Instead grade what the agent produced.

s746
00:58:02.559 --> 00:58:04.480
Did the output have the right information?

s747
00:58:04.480 --> 00:58:06.960
Did the final state match what you wanted.

s748
00:58:06.960 --> 00:58:09.839
Uh our ticker check doesn't care how the agent found the data.

s749
00:58:09.839 --> 00:58:12.559
It just checks is the ticker in the output.

s750
00:58:12.559 --> 00:58:17.359
Uh that is the right level of abstraction for most Kodi valves.

s751
00:58:17.440 --> 00:58:19.680
So now let's get to built-in evals.

s752
00:58:19.680 --> 00:58:20.640
LLM evals.

s753
00:58:20.640 --> 00:58:23.200
This is where it begins to get more interesting.

s754
00:58:23.200 --> 00:58:28.160
Uh code can check whether the ticker appears but it can't check whether the analysis is good.

s755
00:58:28.160 --> 00:58:29.760
Is the financial data accurate?

s756
00:58:29.760 --> 00:58:30.880
Is the report complete?

s757
00:58:30.880 --> 00:58:32.880
Are the recommendations well reasoned?

s758
00:58:32.880 --> 00:58:37.520
Uh these are semantic questions and for semantic questions you need a judge.

s759
00:58:37.520 --> 00:58:40.799
Every LLM as a judge has three parts.

s760
00:58:40.799 --> 00:58:45.040
Uh it has a judge model which is the LLM that is doing the grading for you.

s761
00:58:45.040 --> 00:58:48.960
It has a prompt template which as I mentioned is also called a rubric.

s762
00:58:48.960 --> 00:58:53.680
Uh which is the criteria that the judge applies the rules of what defines good.

s763
00:58:53.680 --> 00:58:56.880
Uh and it has the data which is the examples being evaluated.

s764
00:58:56.880 --> 00:59:00.000
AX keeps these three things separate which means that you can mix and match.

s765
00:59:00.000 --> 00:59:02.640
You can try the same criteria with different judge models.

s766
00:59:02.640 --> 00:59:04.799
The same judge model with different criteria.

s767
00:59:04.799 --> 00:59:07.440
Uh it is modular by design.

s768
00:59:07.440 --> 00:59:13.680
The good news is that AX ships with a B with a bunch of built-in eval so you don't have to write your own.

s769
00:59:13.680 --> 00:59:21.839
There are a bunch of types of evals that are always uh that are often going to apply to uh most AI applications.

s770
00:59:21.839 --> 00:59:26.079
So we've already written the prompts for those and you don't have to write them yourself.

s771
00:59:26.079 --> 00:59:30.079
Uh correctness, for instance, checks whether a response is factually accurate.

s772
00:59:30.079 --> 00:59:34.400
Faithfulness checks whether the response stays grounded in the source material.

s773
00:59:34.400 --> 00:59:36.400
Uh and there are evals for tool selection.

s774
00:59:36.400 --> 00:59:38.400
Did the agent pick the right tool?

s775
00:59:38.400 --> 00:59:40.079
Uh and tool invocation.

s776
00:59:40.079 --> 00:59:42.400
Did it pass the right arguments to that tool?

s777
00:59:42.400 --> 00:59:50.720
stuff like document relevance, refusal detection, many of the things that you'd want to check in an agentic application uh come out of the box.

s778
00:59:50.720 --> 00:59:58.000
Uh so the first eval that we're going to run uh is a correctness eval.

s779
00:59:58.000 --> 01:00:02.960
Um and spoiler alert, it's not going to work.

s780
01:00:02.960 --> 01:00:06.400
Um setting it up is very easy.

s781
01:00:06.400 --> 01:00:07.680
We give it an LLM.

s782
01:00:07.680 --> 01:00:11.680
In this case, uh I've chosen uh the anthropic LLM.

s783
01:00:11.680 --> 01:00:16.559
uh sonnet 4.6 because it is better than haiku which is the thing that is doing the work.

s784
01:00:16.559 --> 01:00:20.000
You often want to pick a bigger slower model to be your judge.

s785
01:00:20.000 --> 01:00:29.040
Uh and then I've just uh given correctness evaluated the LLM uh and uh run it right here.

s786
01:00:29.040 --> 01:00:48.000
Um like I said uh once we've run the evaluation with evaluate data frame uh we've also uh turned on suppressed tracing here so that our evaluation doesn't send a bunch of extra traces to uh AX.

s787
01:00:48.000 --> 01:00:53.200
Um it gives us this output which is a bunch of evaluation rows.

s788
01:00:53.200 --> 01:00:56.240
Um these are not very readable.

s789
01:00:56.240 --> 01:00:58.400
This is why you need AX to look at them.

s790
01:00:58.400 --> 01:01:03.839
Um the judge reads the input and output from each span and grades it against the rubric.

s791
01:01:03.839 --> 01:01:10.640
Um and we can see that every score is zero.

s792
01:01:10.640 --> 01:01:19.520
If I scroll across, status completed, name correctness, score zero.

s793
01:01:19.520 --> 01:01:29.520
The reason it's doing this is because correctness is judging against the uh LLM's definition of of what is correct and what is true.

s794
01:01:29.520 --> 01:01:34.160
Uh which means that it is relying on the LLM's training data.

s795
01:01:34.160 --> 01:01:46.880
What we're doing in this agent is a whole bunch of live web search to get current stock prices and up-to-date financial information, which means the agent is talking about things that happened in 2026,

s796
01:01:46.880 --> 01:01:51.040
but the LLM that's doing the judging only has information that stops in January.

s797
01:01:51.040 --> 01:02:02.720
So, if you read the explanations of our correctness eval inside uh inside of AX, you can scroll to evaluations, you can look at correctness, you can say that it is incorrect.

s798
01:02:02.720 --> 01:02:05.920
uh I will save you a bunch of reading of a very densely packed column.

s799
01:02:05.920 --> 01:02:11.359
What it's doing is complaining that there's a bunch of stuff that comes from the future that it couldn't possibly know.

s800
01:02:11.359 --> 01:02:14.319
Uh so our correctness eval here has completely failed.

s801
01:02:14.319 --> 01:02:16.079
This is not what you wanted to do.

s802
01:02:16.079 --> 01:02:19.680
So instead we are going to use a faithfulness eval.

s803
01:02:19.680 --> 01:02:25.680
Uh faithfulness is a better eval for our use case because uh the judge actually has context.

s804
01:02:25.680 --> 01:02:37.599
In faithfulness, you give the uh LLM as a judge the information that it is being based on uh and then say based on the same information that the jud that my agent collected,

s805
01:02:37.599 --> 01:02:40.000
did it do a good job of writing the report?

s806
01:02:40.000 --> 01:02:42.160
This is this way they are on a level playing field.

s807
01:02:42.160 --> 01:02:55.119
They both have the same amount of information and your slow complicated judge agent uh can judge whether your uh quick fast uh worker agent uh was uh doing a good job.

s808
01:02:55.119 --> 01:02:59.920
Um so let's look at the faithfulness cells.

s809
01:02:59.920 --> 01:03:07.359
Uh just like everything else, these can be done online as well as offline.

s810
01:03:07.359 --> 01:03:14.400
So uh there is a faithfulness built-in evaluator uh which isn't in this view.

s811
01:03:14.400 --> 01:03:22.400
There is a faithfulness built-in evaluator which you can pull up uh and you can give your input, output and your context to it.

s812
01:03:22.400 --> 01:03:25.039
Uh again, you can do this in code as well.

s813
01:03:25.039 --> 01:03:27.760
And if you're using the skills, this is probably what's going to happen.

s814
01:03:27.760 --> 01:03:30.400
Your skills will say, "Oh, I'm not going to use the UI.

s815
01:03:30.400 --> 01:03:31.680
I'm going to use the code."

s816
01:03:31.680 --> 01:03:34.400
Uh which is why I show it here.

s817
01:03:34.400 --> 01:03:39.599
Um so the first thing we do is we extract the research context from the traces.

s818
01:03:39.599 --> 01:03:43.280
As you'll remember, we do uh a two-step process.

s819
01:03:43.280 --> 01:03:55.760
So I take the output from the first step where it does the research and I attach that to all of my spans uh as uh the context for it the judge to work on.

s820
01:03:55.760 --> 01:03:58.640
Uh and then I run the faithfulness evaluator.

s821
01:03:58.640 --> 01:04:02.799
So uh here you can see spans with context.

s822
01:04:02.799 --> 01:04:05.200
These are the same spans that I just pulled out of AX.

s823
01:04:05.200 --> 01:04:08.319
I attached at the column level all of my context.

s824
01:04:08.319 --> 01:04:21.839
Uh and then uh I called evaluate dataf frame with those spans and I gave it my faithfulness evaluator which I instantiated here just like I instantiated the correctness evaluator.

s825
01:04:21.839 --> 01:04:25.599
This gives us uh much more interesting data.

s826
01:04:25.599 --> 01:04:29.440
This gives us unfaithful seven and faithful six.

s827
01:04:29.440 --> 01:04:38.720
Uh so uh roughly half the time my very simple report is hallucinating.

s828
01:04:38.720 --> 01:04:43.599
It is coming up with things that were not in the source data according to sonnet.

s829
01:04:43.599 --> 01:04:48.319
Um and that is this is a useful eval.

s830
01:04:48.319 --> 01:04:50.319
This is an eval where we're failing half the time.

s831
01:04:50.319 --> 01:04:56.880
This is a capability eval where we can climb the hill, modify our prompt and get it better at writing a grounded report.

s832
01:04:56.880 --> 01:04:59.839
This is exactly the kind of eval that you want.

s833
01:04:59.839 --> 01:05:07.119
Um, so that's two built-in evals and two very different signals.

s834
01:05:07.119 --> 01:05:16.160
And there's a really important lesson here, which is correctness gave us zero out of 13 and faithfulness uh told us that reports aren't grounded in their sources roughly half the time.

s835
01:05:16.160 --> 01:05:18.400
The difference isn't that one eval is better than the other.

s836
01:05:18.400 --> 01:05:20.720
The difference is uh the use case.

s837
01:05:20.720 --> 01:05:26.720
There are lots of use cases where that correctness eval that I showed you earlier works just fine for some use cases.

s838
01:05:26.720 --> 01:05:32.160
Uh it's just not in this specific use case where we were doing live real-time research.

s839
01:05:32.160 --> 01:05:37.119
Um so it's important to know the question that you're actually asking.

s840
01:05:38.880 --> 01:05:40.480
Built-in eval are your starting point.

s841
01:05:40.480 --> 01:05:43.359
They give you an immediate signal without any prompt engineering.

s842
01:05:43.359 --> 01:05:49.039
Uh but even with a good built-in like faithfulness, uh there are a bunch of things that it can't check.

s843
01:05:49.039 --> 01:05:54.559
uh our financial analyst uh should produce actionable recommendations.

s844
01:05:54.559 --> 01:05:56.799
It shouldn't just summarize data.

s845
01:05:56.799 --> 01:06:02.559
Um so not just you know give you a financial report but tell you what to do with the financial report.

s846
01:06:02.559 --> 01:06:04.400
Should I buy, sell or hold?

s847
01:06:04.400 --> 01:06:07.359
Uh no built-in template checks for that.

s848
01:06:07.359 --> 01:06:09.920
This AX is does not cover every single use case.

s849
01:06:09.920 --> 01:06:13.839
So you have to build a specific eval that checks for actionability.

s850
01:06:13.839 --> 01:06:17.599
You have to build an custom eval that is about your domain.

s851
01:06:17.599 --> 01:06:25.520
If you are writing real eval, you're absolutely going to have to write a custom eval with a definition of good that we didn't think of.

s852
01:06:25.520 --> 01:06:28.799
So that brings us to the next step, which is writing a custom eval rubric.

s853
01:06:28.799 --> 01:06:32.799
This is where you make eval truly specific to your application.

s854
01:06:32.799 --> 01:06:35.359
And writing a good eval is complicated.

s855
01:06:35.359 --> 01:06:39.760
So I'm going to spend a little time talking about what makes a good eval rubric.

s856
01:06:39.760 --> 01:06:43.520
Uh we recommend four parts of a custom rubric.

s857
01:06:43.520 --> 01:06:46.240
The first is to define the judge's role.

s858
01:06:46.240 --> 01:06:50.400
Um, you should spell out ex explicit pass and fail criteria.

s859
01:06:50.400 --> 01:06:52.480
You should label the data with XML tags.

s860
01:06:52.480 --> 01:06:56.240
Uh, and you should define the output choices outside the prompt itself.

s861
01:06:56.240 --> 01:07:06.559
Uh, I'm going to add one more thing on top of those which is actually kind of controversial amongst practitioners, which is I tend to include examples of good and bad output.

s862
01:07:06.559 --> 01:07:13.680
Uh, some practitioners believe uh that including examples will make your uh, eval overfit.

s863
01:07:13.680 --> 01:07:16.640
Uh but let's walk through all of those.

s864
01:07:16.640 --> 01:07:19.119
So the first part is defining the role.

s865
01:07:19.119 --> 01:07:26.079
Um you'll notice this uh what is by now a tired trope which is you are an expert financial analyst evaluator.

s866
01:07:26.079 --> 01:07:27.920
That is not the important part.

s867
01:07:27.920 --> 01:07:34.400
Tests show it makes some very small difference uh to whether or not uh your eval works.

s868
01:07:34.400 --> 01:07:37.920
The important thing is to give it the context of what it is looking at.

s869
01:07:37.920 --> 01:07:41.839
You are looking at a financial report and what I am expecting is financial analysis.

s870
01:07:41.839 --> 01:07:45.920
Those are the important pieces of context that you are giving the LLM there.

s871
01:07:45.920 --> 01:07:51.520
Um you're also telling it that it should be that what you want is something actionable.

s872
01:07:51.520 --> 01:07:53.680
So that brings us to part two.

s873
01:07:53.680 --> 01:07:57.680
So uh this is where most people underinvest.

s874
01:07:57.680 --> 01:08:01.039
Don't say a good response because that is uh an aspiration.

s875
01:08:01.039 --> 01:08:02.400
That is not a criterion.

s876
01:08:02.400 --> 01:08:12.079
A good response is helpful and accurate sounds like you're adding more detail, but you're still being just as vague because helpful and accurate don't have any meaning as far as the LLM is concerned.

s877
01:08:12.079 --> 01:08:15.760
Uh instead, you should be listing exactly what makes a report actionable.

s878
01:08:15.760 --> 01:08:19.199
So list exactly what makes it actionable and not actionable.

s879
01:08:19.199 --> 01:08:21.759
Every criterion should be specific and observable.

s880
01:08:21.759 --> 01:08:28.799
Something that if you were a human looking at this report, you'd be able to know whether or not it was there and you'd be able to say yes or no.

s881
01:08:28.799 --> 01:08:32.400
Uh so contains specific recommendations.

s882
01:08:32.400 --> 01:08:37.440
That's something you can definitely say yes or no. Includes forward-looking analysis, not just historical data.

s883
01:08:37.440 --> 01:08:40.960
That's a clear distinction that you can say yes or no to.

s884
01:08:40.960 --> 01:08:48.000
Uh and the important thing here is that each of these criteria maps to something we saw in our analysis.

s885
01:08:48.000 --> 01:09:00.719
If we when we were reading the reports uh we saw we actually observed when we read our traces or at least when I read our traces uh that uh these are ways that these reports are going wrong specifically that is not a coincidence.

s886
01:09:00.719 --> 01:09:04.239
Your error analysis is what tells you what criteria to write.

s887
01:09:04.239 --> 01:09:07.600
You don't come up with them from whole cloth because it's not going to occur to you.

s888
01:09:07.600 --> 01:09:11.600
Where you get that data from is from reading your traces and going oh it didn't do that.

s889
01:09:11.600 --> 01:09:15.199
My axial coding said it does this specific wrong thing.

s890
01:09:15.199 --> 01:09:22.319
Uh so specific criteria produce consistent judgments uh while vague criteria produce coin flips.

s891
01:09:22.319 --> 01:09:28.239
The third thing you should do in your rubric is you should label the data with XML tags especially if you are using anthropic.

s892
01:09:28.239 --> 01:09:31.600
Anthropic loves XML and finds it very easy to understand.

s893
01:09:31.600 --> 01:09:43.759
Um your rubric has variables the input and the output and you want to make sure that it can tell the difference between your instructions and the input and the output and XML tags are an excellent way to do that.

s894
01:09:43.759 --> 01:09:48.480
Um, so put a user query tag around the input, a financial report tag around the output.

s895
01:09:48.480 --> 01:09:52.960
Uh, clear boundaries reduce the chance that the judge confuses the question with the answer.

s896
01:09:52.960 --> 01:10:04.560
It is especially important if you're including examples uh to label the examples with XML tags because otherwise it will go, "Oh, this is exactly what I'm supposed to write and just spit out the example back to you."

s897
01:10:04.560 --> 01:10:07.440
Um, which brings us to the examples.

s898
01:10:07.440 --> 01:10:09.679
This is the part a lot of people skip.

s899
01:10:09.679 --> 01:10:12.719
Uh, in my opinion, it is the part that most improves quality.

s900
01:10:12.719 --> 01:10:16.080
LLMs are okay at following instructions.

s901
01:10:16.080 --> 01:10:18.400
They are increasingly good at following instructions.

s902
01:10:18.400 --> 01:10:26.480
But what they are really good at is copying an example that you've already given them and replacing the things that are relevant to the current situation.

s903
01:10:26.480 --> 01:10:32.560
So if you give it an example of exactly the kind of report that you're looking for, it's going to follow that format exactly.

s904
01:10:32.560 --> 01:10:36.560
That might or might not depending on your domain be a good idea.

s905
01:10:36.560 --> 01:10:44.159
If you if your example is too rigid, then the LLM will always give the same report and you have reduced its scope for creativity.

s906
01:10:44.159 --> 01:10:55.120
Uh but on the other hand, if it is being too nondeterministic and your report is being crazy, uh then giving it an example will ground it in what exactly you are expecting.

s907
01:10:55.760 --> 01:10:58.880
Uh here's what an actional example looks like in the template.

s908
01:10:58.880 --> 01:11:00.000
Uh it has everything.

s909
01:11:00.000 --> 01:11:03.440
It has uh percentage revenue growth, a PE ratio.

s910
01:11:03.440 --> 01:11:08.000
It identifies a concrete risk with a number to back it up and it gives a specific recommendation.

s911
01:11:08.000 --> 01:11:10.000
Uh that is what actionable looks like.

s912
01:11:10.000 --> 01:11:12.000
Here is a non-actionable result.

s913
01:11:12.000 --> 01:11:13.440
Uh it's not wrong.

s914
01:11:13.440 --> 01:11:15.520
Uh everything it says is correct.

s915
01:11:15.520 --> 01:11:16.960
Uh but it doesn't say anything useful.

s916
01:11:16.960 --> 01:11:18.159
It doesn't have any specific data.

s917
01:11:18.159 --> 01:11:19.840
It doesn't have any specific risks.

s918
01:11:19.840 --> 01:11:22.880
It just says investors should consider various factors.

s919
01:11:22.880 --> 01:11:24.960
Uh that's not a recommendation.

s920
01:11:24.960 --> 01:11:31.199
Uh so the judge now has a concrete reference for what you actually mean by the labels that you were giving it.

s921
01:11:31.199 --> 01:11:40.159
Uh and this can dramatically improve consistency especially on the edge cases where the report uh would have otherwise been somewhere in between.

s922
01:11:40.560 --> 01:11:43.840
Uh and this is uh an important part.

s923
01:11:43.840 --> 01:11:51.280
It is very tempting if you're writing this rubric to say and now spit out the word correct or incorrect because that is what you wanted to do.

s924
01:11:51.280 --> 01:11:53.360
You do not need to do that anymore.

s925
01:11:53.360 --> 01:12:03.920
Uh the way that it actually works under the hood is we have given the LLM a tool that says you call this tool when you are done and it will call that tool with correct or incorrect when you're done.

s926
01:12:03.920 --> 01:12:12.320
So you don't need to tell it what output and in fact telling it to spit out a specific string as output is more likely to confuse it and make your eval fail.

s927
01:12:12.320 --> 01:12:25.120
So just in the UI uh set your uh choices uh or when you are setting the when you're creating it programmatically set the choices uh and let the LLM do its thing.

s928
01:12:25.120 --> 01:12:29.600
Um when you are coming up with these choices it is best to keep those choices simple.

s929
01:12:29.600 --> 01:12:32.320
Uh binary is best two clear labels.

s930
01:12:32.320 --> 01:12:36.239
Uh numeric scales like a scale of 1 to five.

s931
01:12:36.239 --> 01:12:40.239
Uh they are tempting but the LLMs are bad at them.

s932
01:12:40.239 --> 01:12:42.480
what is the difference between a three and a four?

s933
01:12:42.480 --> 01:12:45.520
You would have to write a whole bunch of rules for the LLM to know that.

s934
01:12:45.520 --> 01:12:47.679
Uh it doesn't know that automatically.

s935
01:12:47.679 --> 01:12:53.920
Uh whereas you can give it a pass fail and it will be pretty good at deciding what is a pass and what is a fail.

s936
01:12:53.920 --> 01:12:58.080
Uh so if you really need more categories, you can do like a partial result.

s937
01:12:58.080 --> 01:13:03.120
You can say a 0.5 score is where it did a a half-ass job.

s938
01:13:03.120 --> 01:13:08.080
Uh but by default, go for a zero or one.

s939
01:13:08.080 --> 01:13:11.280
Um, one more practical tip is chain of thought for judges.

s940
01:13:11.280 --> 01:13:16.239
Ask the judge to reason before it scores and have it explain its thinking first.

s941
01:13:16.239 --> 01:13:20.000
Uh, this measurably improves uh, accuracy.

s942
01:13:20.000 --> 01:13:25.440
Uh, when the judge has to articulate why something is actionable or not or why it is correct or not.

s943
01:13:25.440 --> 01:13:30.320
Uh, it makes fewer mistakes than when it just picks a label uh, without thinking first.

s944
01:13:30.320 --> 01:13:33.360
Uh, this explanation is also very helpful for you.

s945
01:13:33.360 --> 01:13:35.679
Uh, but it also helps the judge think better.

s946
01:13:35.679 --> 01:13:38.640
Just one of those weird things about LLMs.

s947
01:13:38.640 --> 01:13:45.760
Uh so if you are coding along, now is the time to find your actionability template.

s948
01:13:45.760 --> 01:13:51.760
Uh this contains all of the example all of the instructions that I uh was just talking about.

s949
01:13:51.760 --> 01:13:53.520
Uh can you do better than me?

s950
01:13:53.520 --> 01:13:59.199
Can you oneshot your way to a perfect eval uh by modifying this eval template?

s951
01:13:59.199 --> 01:14:05.679
Um, as with uh all things in AX, you can do this via the UI as well.

s952
01:14:05.679 --> 01:14:10.960
So, you can create an online evaluator uh and you can create a new LLM as a judge.

s953
01:14:10.960 --> 01:14:21.760
In this case, I've already got it, my actionability evaluator uh which has literally exactly the same text uh that I just showed you in the notebook.

s954
01:14:21.760 --> 01:14:27.199
Um, and as before, I had to pick a trace uh scope.

s955
01:14:27.199 --> 01:14:30.960
Uh, and because it's an LLM as a judge, I had to set up a template.

s956
01:14:30.960 --> 01:14:35.679
Uh, sorry, I had to set up an LLM for it to use in the UI.

s957
01:14:35.679 --> 01:14:37.840
Uh, which means that you need to give it your anthropic key.

s958
01:14:37.840 --> 01:14:43.440
Again, uh, and I picked my inputs and outputs via the single trace method.

s959
01:14:43.440 --> 01:14:45.040
Uh, and then I saved it.

s960
01:14:45.040 --> 01:14:52.880
One of the things that you can do with online evals uh is you can set them to run continuously against new incoming traces.

s961
01:14:52.880 --> 01:14:55.360
This is something you'll almost certainly do in production.

s962
01:14:55.360 --> 01:15:01.600
Uh and if your evals are expensive or you have a whole lot of traffic, you won't want to do it on all of your traffic.

s963
01:15:01.600 --> 01:15:06.719
You can uh set it to serve to uh rate only a percentage.

s964
01:15:06.719 --> 01:15:07.840
That's the sampling rate.

s965
01:15:07.840 --> 01:15:16.320
So you can say, you know, just look at 1% of my traffic and that will still give you uh a whole bunch of eval data without breaking your budget.

s966
01:15:17.440 --> 01:15:29.920
Uh so back to the code version, wiring it up is just like wiring up the other action the other uh evaluators that we've given.

s967
01:15:29.920 --> 01:15:32.640
Uh we've created a classification evaluator.

s968
01:15:32.640 --> 01:15:34.159
We've given it a name actionability.

s969
01:15:34.159 --> 01:15:35.679
We've given it an LLM.

s970
01:15:35.679 --> 01:15:39.520
Uh we've given it that template that I just showed you and we've given it those choices.

s971
01:15:39.520 --> 01:15:45.120
This is us programmatically doing what we would have done in the UI where we said your choices are correct or incorrect.

s972
01:15:45.120 --> 01:15:48.480
Uh actionable or not actionable scores of one or zero.

s973
01:15:48.480 --> 01:15:52.480
Uh and then uh we ran evaluate data frame.

s974
01:15:52.480 --> 01:16:00.960
That gives us uh the data that I was skipping over because I was already run earlier.

s975
01:16:02.400 --> 01:16:05.120
So you can see actionability actionable.

s976
01:16:05.120 --> 01:16:15.120
If we look at our evaluations tab, we can scroll down to our actionability and we get this really great explanation about why the report is actionable or not actionable.

s977
01:16:15.120 --> 01:16:32.159
Uh lots of our reports scored actionable uh but several are not which is awesome because it means that we have again we have a capability eval uh and a hill that we can climb.

s978
01:16:32.159 --> 01:16:44.640
Um, so in the notebook, uh, this is where you log your results back to AX or if you've run it online in the UI, your evals, your eval results are already in the UI as you've seen.

s979
01:16:44.640 --> 01:16:50.159
Um, so you can see now in in AX that every span has multiple eval scores.

s980
01:16:50.159 --> 01:16:54.080
It has mention sticker, it has correctness, it has faithfulness, it has actionability.

s981
01:16:54.080 --> 01:17:03.600
uh and you can filter and select uh for specific uh eval results.

s982
01:17:03.600 --> 01:17:17.920
So, eval.actionability.label equals not actionable.

s983
01:17:21.920 --> 01:17:23.920
that finds you the failing set.

s984
01:17:23.920 --> 01:17:29.040
This is the set that you're like, okay, my eva my eval says that my code messed up here.

s985
01:17:29.040 --> 01:17:33.920
I can turn this into a data set and I can start running uh new prompts against this.

s986
01:17:33.920 --> 01:17:37.360
I can run experiments against this as we're going to see.

s987
01:17:37.360 --> 01:17:49.040
Um before we move on, I want to flag a few a few common antiatterns uh that turn evals from a useful tool into noise.

s988
01:17:49.040 --> 01:17:51.760
Uh the first one is to treat your eval prompts like code.

s989
01:17:51.760 --> 01:17:52.719
And I mean that literally.

s990
01:17:52.719 --> 01:17:55.280
They're very sensitive to war to wording.

s991
01:17:55.280 --> 01:18:01.120
Small shifts in your eval prompt uh can lead to dramatic changes in your eval results.

s992
01:18:01.120 --> 01:18:02.480
Uh so you should version them.

s993
01:18:02.480 --> 01:18:05.280
You should test them on examples where you know the right answer.

s994
01:18:05.280 --> 01:18:13.360
If the judge if the LLM as a judge disagrees with your human labels uh on 40% of examples, then your eval is probably wrong.

s995
01:18:13.360 --> 01:18:14.960
It's not that the your results are wrong.

s996
01:18:14.960 --> 01:18:17.440
It is that you have written the eval incorrectly.

s997
01:18:17.440 --> 01:18:27.520
uh AX has a prompt playground where you can iterate on uh rubrics without touching your code and it's very useful for doing this kind of iteration of your eval.

s998
01:18:27.760 --> 01:18:31.600
Uh the second antiattern is the god evaluator.

s999
01:18:31.600 --> 01:18:36.880
Um it is tempting to build one big mega evaluator that checks everything at once.

s1000
01:18:36.880 --> 01:18:41.760
It'll look for accuracy and tone and completeness and policy compliance and formatting all in a single prompt.

s1001
01:18:41.760 --> 01:18:44.640
Don't do this because it is a nightmare to calibrate.

s1002
01:18:44.640 --> 01:18:49.199
uh when your god evaluator says fail, which of those five dimensions failed?

s1003
01:18:49.199 --> 01:18:49.920
You don't know.

s1004
01:18:49.920 --> 01:18:52.640
So suddenly you're like, "Oh, well, I'll have it spit out labels."

s1005
01:18:52.640 --> 01:18:55.360
And then you're like, "Oh, it's spitting out 15 different labels."

s1006
01:18:55.360 --> 01:18:57.520
Suddenly it's way too complicated.

s1007
01:18:57.520 --> 01:19:01.199
Uh you can't tell, you can't judge by it, you can't score by it.

s1008
01:19:01.199 --> 01:19:07.760
It is much much simpler and easier to split this up into uh one evaluator per dimension.

s1009
01:19:07.760 --> 01:19:09.199
I've already done this today.

s1010
01:19:09.199 --> 01:19:10.159
I have a ticker check.

s1011
01:19:10.159 --> 01:19:10.880
I have faithfulness.

s1012
01:19:10.880 --> 01:19:11.760
I have actionability.

s1013
01:19:11.760 --> 01:19:13.600
Three separate focused evals.

s1014
01:19:13.600 --> 01:19:16.159
and each one tells you something specific.

s1015
01:19:16.159 --> 01:19:21.040
Uh the combination gives you a richer picture than any single eval ever could.

s1016
01:19:21.040 --> 01:19:26.719
If you find yourself writing a rubric that lists six different things that the response should do, then stop.

s1017
01:19:26.719 --> 01:19:30.080
That is six evaluators, not one.

s1018
01:19:30.080 --> 01:19:33.760
Uh the third one is know your guardrails and your northstar metrics.

s1019
01:19:33.760 --> 01:19:35.280
Some evals are guard rails.

s1020
01:19:35.280 --> 01:19:36.159
They're ship blockers.

s1021
01:19:36.159 --> 01:19:39.520
If an agent hallucinates a stock price, that is a hard fail.

s1022
01:19:39.520 --> 01:19:40.800
Don't ship on that.

s1023
01:19:40.800 --> 01:19:42.480
Uh others are northstar metrics.

s1024
01:19:42.480 --> 01:19:44.000
they are aspirational.

s1025
01:19:44.000 --> 01:19:47.840
So always recommend complimentary investments is a nice to have in this case.

s1026
01:19:47.840 --> 01:19:49.679
It is not a dealbreaker.

s1027
01:19:49.679 --> 01:19:52.320
Uh so know which of your eval is which.

s1028
01:19:52.320 --> 01:19:56.880
It changes how you act on the results and later it changes how you set up your production monitors.

s1029
01:19:56.880 --> 01:20:00.640
A guardrail uh regression should page someone in the middle of the night.

s1030
01:20:00.640 --> 01:20:05.199
A northstar dip is something that you look at in a weekly report.

s1031
01:20:06.400 --> 01:20:09.679
Which brings us to the next section which is can you trust your judges?

s1032
01:20:09.679 --> 01:20:11.360
This is a question we've been dancing around.

s1033
01:20:11.360 --> 01:20:13.120
I've mentioned it a couple of times.

s1034
01:20:13.120 --> 01:20:16.000
Uh it's time to evaluate your evaluator.

s1035
01:20:16.000 --> 01:20:20.880
We built an LLM judge and it told us that some reports are not actionable.

s1036
01:20:20.880 --> 01:20:22.239
Uh but can we trust it?

s1037
01:20:22.239 --> 01:20:25.840
How do we know that the judge is correct?

s1038
01:20:25.840 --> 01:20:29.840
Uh the key mental model here is that your judge is a classifier.

s1039
01:20:29.840 --> 01:20:31.520
So you can treat it as a classifier.

s1040
01:20:31.520 --> 01:20:37.120
It takes an input in this case a financial report and it makes a prediction actionable or not actionable.

s1041
01:20:37.120 --> 01:20:38.560
That prediction can be right or wrong.

s1042
01:20:38.560 --> 01:20:44.880
So just like any classifier, you can measure its performance by comparing its predictions against ground truth.

s1043
01:20:44.880 --> 01:20:47.440
In this case, a human's judgment.

s1044
01:20:47.440 --> 01:20:50.560
Uh you can check the judge's homework.

s1045
01:20:50.560 --> 01:20:54.320
Um the thing about applying judgment is that it is a ton of work.

s1046
01:20:54.320 --> 01:20:57.040
You have to read actually every single output.

s1047
01:20:57.040 --> 01:20:59.679
Uh you have to think hard and apply a label.

s1048
01:20:59.679 --> 01:21:06.719
Uh AX does what it can to help by letting you define annotations and attaching them to spans uh right in the UI.

s1049
01:21:06.719 --> 01:21:10.159
So, let me show you what that looks like.

s1050
01:21:11.120 --> 01:21:17.360
When you're looking at a span, you can click annotate span and you can add and remove annotation configs.

s1051
01:21:17.360 --> 01:21:20.320
In this case, I've already created one called human actionable.

s1052
01:21:20.320 --> 01:21:23.679
And you can set this span as being actionable or not actionable.

s1053
01:21:23.679 --> 01:21:31.840
And then you can just use these little arrows to flip through all of your traces and set them as actionable or not actionable, which I have conveniently already done.

s1054
01:21:31.840 --> 01:21:36.880
Uh this is the kind of work that you can give to a nontechnical member of staff.

s1055
01:21:36.880 --> 01:21:38.400
That is why the UI looks like this.

s1056
01:21:38.400 --> 01:21:42.320
They can just click in, read the report and click a checkbox.

s1057
01:21:42.320 --> 01:21:43.440
They don't have to write any code.

s1058
01:21:43.440 --> 01:21:51.280
They don't need to know anything other than their domain expertise of whether or not this report should be considered actionable or not actionable.

s1059
01:21:52.320 --> 01:21:56.480
What we're doing here when we add human annotations is building a golden data set.

s1060
01:21:56.480 --> 01:22:03.600
Building golden data sets is incredibly helpful uh because it's how you measure uh whether your evaluator is actually doing its job.

s1061
01:22:03.600 --> 01:22:07.440
Uh the way to do it is the same thing uh the LLM did.

s1062
01:22:07.440 --> 01:22:10.400
Give yourself real concrete criteria and stick to them.

s1063
01:22:10.400 --> 01:22:12.480
So don't say this was good, this was bad.

s1064
01:22:12.480 --> 01:22:15.920
Uh be specific the same way that you've told the LLM to be specific.

s1065
01:22:15.920 --> 01:22:17.600
You failed at this specific thing.

s1066
01:22:17.600 --> 01:22:21.040
You asked I this thing was supposed to be present and it wasn't.

s1067
01:22:21.040 --> 01:22:26.560
uh and eliminate the chance to get lazy uh when you get tired of labeling.

s1068
01:22:26.560 --> 01:22:36.239
Um as I said uh the labeling is extremely uh timeconuming so I did it already.

s1069
01:22:36.239 --> 01:22:42.719
Um once I've labeled some spans in the UI uh I can reexport the spans into the notebook.

s1070
01:22:42.719 --> 01:22:51.040
Um, and this is a nice thing about AX is that the human annotations that I added by hand come back as columns uh in the export.

s1071
01:22:51.040 --> 01:23:04.560
Um, so we filter down to just the rows that I labeled here.

s1072
01:23:04.960 --> 01:23:15.600
Uh, and now we're going to compare uh what I said was actionable and not actionable to what the judge says is actionable or not actionable.

s1073
01:23:15.600 --> 01:23:25.679
Um, before we talk before we do that though, uh, a little bit more about golden data sets, uh, keep your tasks unambiguous.

s1074
01:23:25.679 --> 01:23:30.000
If your agent scores 0% consistently, that's almost always a broken task.

s1075
01:23:30.000 --> 01:23:34.239
That's almost always, uh, you've written your rubric incorrectly.

s1076
01:23:34.239 --> 01:23:44.000
uh if you've made your task something that only a human can do or worse that somebody something nobody could really do uh then you're going to get a zero score.

s1077
01:23:44.000 --> 01:23:49.440
Um so for each task create a reference solution a known working output that passes all of your graders.

s1078
01:23:49.440 --> 01:23:54.400
This proves that the task is solvable and you should also test in both directions.

s1079
01:23:54.400 --> 01:23:58.639
You should have cases where a behavior should occur and cases where it shouldn't.

s1080
01:23:58.639 --> 01:24:09.280
So if you test for instance does it search the web uh then you will end up with a uh an agent that always searches the web even when it doesn't need to.

s1081
01:24:09.280 --> 01:24:16.880
Whereas you should also have a test that says did it not search the web for this very obvious answer where it didn't need to do that.

s1082
01:24:16.880 --> 01:24:24.080
Uh and if you're doing this for real you should split your label data that you just created uh into a development set and a test set.

s1083
01:24:24.080 --> 01:24:33.120
Uh so use maybe 75% of your labels to it to to iterate on your eval uh tweaking criteria, adjusting examples until the judge agrees with you.

s1084
01:24:33.120 --> 01:24:40.159
Then hold out the remaining 25% to run the judge and see that it actually gets the scores that you are expecting.

s1085
01:24:40.159 --> 01:24:47.440
Uh that's how you know that the judge is actually generalizing to new examples instead of just fitting to the specific data set that you gave it.

s1086
01:24:47.440 --> 01:24:51.040
Uh this is the same principle as train tests, splits in machine learning.

s1087
01:24:51.040 --> 01:24:55.440
Don't overfit your evaluator to your golden data set.

s1088
01:24:56.639 --> 01:25:00.239
Now let's look at our actionability judge.

s1089
01:25:01.199 --> 01:25:08.560
I have run the judge on some examples and it has come up with uh a list of where they agree and disagree.

s1090
01:25:08.560 --> 01:25:17.440
Uh spoiler alert I just hit actionable and not actionable at random uh so that I would get lots of disagreement because actually Sonnet is pretty good at this stuff.

s1091
01:25:17.440 --> 01:25:23.840
Um that gives us uh an agreement rate of 46% six out of 13 times.

s1092
01:25:23.840 --> 01:25:32.080
Um in real life uh this is exactly where you dig in.

s1093
01:25:32.080 --> 01:25:37.040
This is where uh you would ask is my rubric wrong or were my human labels wrong.

s1094
01:25:37.040 --> 01:25:43.679
When the judge disagrees with you, you should read its explanation and decide whether or not the judge was right or you were right.

s1095
01:25:43.679 --> 01:25:46.880
Uh that is the most valuable output of meta evaluation.

s1096
01:25:46.880 --> 01:25:51.600
It usually reveals an ambiguity in the rules that you set down.

s1097
01:25:51.600 --> 01:25:57.840
To fix a rubric, you should read the explanations on the disagreements, find the ambiguity, uh, and tighten the criteria.

s1098
01:25:57.840 --> 01:26:05.199
So, instead of just includes forward-looking analysis, I'd write includes forward-looking analysis with specific recommendations or guidance.

s1099
01:26:05.199 --> 01:26:06.480
That's more precise.

s1100
01:26:06.480 --> 01:26:12.239
Uh, it rules out the case where the report talks about the future, but never tells the judge what to do about it.

s1101
01:26:12.239 --> 01:26:16.960
Uh then I'd rerun the judge on the same examples and see if the disagreement goes goes away.

s1102
01:26:16.960 --> 01:26:18.960
Uh this is rubric iteration.

s1103
01:26:18.960 --> 01:26:21.199
Your eval prompt is an application.

s1104
01:26:21.199 --> 01:26:23.520
Uh it is another LLM application.

s1105
01:26:23.520 --> 01:26:26.880
Uh so it needs testing and iteration just like your agent does.

s1106
01:26:26.880 --> 01:26:31.760
Uh and that is what I meant when I said that you should treat your evals like code.

s1107
01:26:32.800 --> 01:26:37.679
So now I'm going to talk very briefly about an ML concept called precision and recall.

s1108
01:26:37.679 --> 01:26:45.440
Uh, precision asks, when the judge says not actionable, how often is it really not actionable?

s1109
01:26:45.440 --> 01:26:52.000
Uh, and recall asks, of all the reports that are actually not actionable, how many did the judge catch?

s1110
01:26:52.000 --> 01:26:57.679
Uh, with a tiny sample like uh 12 examples, these numbers are going to judge jump around a lot.

s1111
01:26:57.679 --> 01:27:05.440
Uh, but even with the 12 that I have, uh, you can see whether the judge is in the right ballpark or completely off.

s1112
01:27:05.440 --> 01:27:10.800
In most scenarios, you want to uh you want to prioritize recall.

s1113
01:27:10.800 --> 01:27:14.719
These two measurements are at odds with each other.

s1114
01:27:14.719 --> 01:27:23.360
If you are if you prioritize recall, then you're going to get false positives where the judge has said something is wrong when nothing is wrong.

s1115
01:27:23.360 --> 01:27:27.679
That is much better than accidentally letting through things that are wrong.

s1116
01:27:27.679 --> 01:27:32.880
Uh however, some in some cases when you are tuning, you will say actually I want precision.

s1117
01:27:32.880 --> 01:27:36.639
I want it to be absolutely right when it's right and I don't care if it misses some things.

s1118
01:27:36.639 --> 01:27:43.760
There are domains where that is the output that you want, but most of the time uh you want to prioritize recall.

s1119
01:27:43.760 --> 01:27:50.000
Um a few known pitfalls with using an LLM as a judge.

s1120
01:27:50.000 --> 01:27:52.159
Um one is position bias.

s1121
01:27:52.159 --> 01:27:58.800
If you present it with two options to judge between, uh the judge tend to favor tends to favor whichever comes first.

s1122
01:27:58.800 --> 01:28:00.639
Uh this depends on the model.

s1123
01:28:00.639 --> 01:28:07.120
Sometimes the model depends uh prefers whichever comes last but it always has some kind of preference to position.

s1124
01:28:07.120 --> 01:28:09.920
Um there's also length bias.

s1125
01:28:09.920 --> 01:28:11.920
Uh LLMs like longer responses.

s1126
01:28:11.920 --> 01:28:17.840
They tend they tend to score those higher and that say that they are better even if the length extra length is just filler.

s1127
01:28:17.840 --> 01:28:20.400
Uh and the last one I love which is confidence bias.

s1128
01:28:20.400 --> 01:28:25.280
The judge gets fooled uh by a response that sounds confident just like humans do.

s1129
01:28:25.280 --> 01:28:33.520
If your LLM says things that are wrong, but it says them in a really confident tone, uh, your LLM judge is more likely to believe them.

s1130
01:28:33.520 --> 01:28:36.000
Um, there's also self-preference bias.

s1131
01:28:36.000 --> 01:28:41.920
If you use the same model to do the generation as you use to do the judging, they tend to like their own output.

s1132
01:28:41.920 --> 01:28:45.679
So, if possible, use both say OpenAI and Anthropic.

s1133
01:28:45.679 --> 01:28:53.040
Use one to do your agent and one to do the judging and they are less likely to agree with each other.

s1134
01:28:53.360 --> 01:29:00.320
uh self-preference bias is why I used sonnet as a judge uh for output generated by haiku.

s1135
01:29:00.320 --> 01:29:06.080
Uh but it works even better if you use a different uh labs model entirely.

s1136
01:29:06.719 --> 01:29:09.840
And remember that the benchmark here is human performance.

s1137
01:29:09.840 --> 01:29:14.800
Humans are not going to be perfect at judging whether things are or are not actionable.

s1138
01:29:14.800 --> 01:29:17.280
They are going to get it wrong quite a lot of the time.

s1139
01:29:17.280 --> 01:29:23.360
What you're trying to do is come up with an eval that is reasonably close to human judgment.

s1140
01:29:23.360 --> 01:29:26.560
Uh human judgment also varies from human to human.

s1141
01:29:26.560 --> 01:29:33.360
Human inter interrator reliability, so two humans judging the same thing uh is often as low as 02 or.3.

s1142
01:29:33.360 --> 01:29:39.760
So two experts with the same output and the same rules disagree a surprising amount of the time.

s1143
01:29:39.760 --> 01:29:46.639
So if your judge, if your LLM judge achieves that uh higher consistency than humans, then it is doing better than humans.

s1144
01:29:46.639 --> 01:29:52.320
the judge uh sometimes disagrees with me is not by itself a reason to distrust it.

s1145
01:29:52.320 --> 01:29:56.159
You what you're looking for is a judge that disagrees with you all of the time.

s1146
01:29:56.159 --> 01:29:57.280
That is the danger signal.

s1147
01:29:57.280 --> 01:30:05.440
A judge that disagrees with you some of the time is a judge uh correctly handling a genuinely ambiguous task.

s1148
01:30:06.880 --> 01:30:09.440
Uh and also failures should seem fair.

s1149
01:30:09.440 --> 01:30:12.639
That is a principle from Anthropic Eval teams that I want to leave you with.

s1150
01:30:12.639 --> 01:30:17.440
When a task fails, uh, it should be clear what the agent got wrong and why.

s1151
01:30:17.440 --> 01:30:25.920
If you look at a failing trace and you think that answer looks fine to me, the problem is probably the eval and not the agent.

s1152
01:30:26.800 --> 01:30:34.719
So, uh, now we've built evals, we've tested them, uh, let's use them to actually improve our agent.

s1153
01:30:34.719 --> 01:30:37.280
This is how you close the loop.

s1154
01:30:37.280 --> 01:30:41.440
Uh, here's the problem with one-off fixes.

s1155
01:30:41.440 --> 01:30:47.600
Um, if you found some failures, you've read some explanations, you know what to improve, you change the prompt, and then what?

s1156
01:30:47.600 --> 01:30:50.320
How do you know uh whether the fix actually worked?

s1157
01:30:50.320 --> 01:30:53.120
How do you know if it didn't break something that was working before?

s1158
01:30:53.120 --> 01:30:58.480
If you just run the agent again on a couple of examples and eyeball it, then you're back to vibes.

s1159
01:30:58.480 --> 01:31:02.239
Uh, you need a structured way to compare before and after.

s1160
01:31:02.239 --> 01:31:06.960
And as I mentioned earlier, this is what experiments are for.

s1161
01:31:06.960 --> 01:31:16.320
As I showed you earlier, uh you can filter your traces down to just the ones that are failing and you can turn them into a data set like this.

s1162
01:31:16.320 --> 01:31:21.600
So you select all of your traces, you click add to data set, and you create a new data set.

s1163
01:31:21.600 --> 01:31:25.920
In this case, uh I've called it AIEWF financial demo fails.

s1164
01:31:25.920 --> 01:31:29.520
Uh and I've already created my failure data set.

s1165
01:31:29.520 --> 01:31:39.679
But this gives you a smaller set of places where you know that your eval is failing so that you can then run uh your changed prompt against just them.

s1166
01:31:39.679 --> 01:31:41.679
This is faster and cheaper.

s1167
01:31:41.679 --> 01:31:45.760
Um this is a regression test set.

s1168
01:31:45.760 --> 01:31:59.120
uh sorry you should also save a regression test set rather you should get the ones where it is not failing and make sure that those are also a data set that you can check less frequently to make sure that when you've changed your model sorry when you've changed your rubric

s1169
01:31:59.120 --> 01:32:17.360
uh that you haven't accidentally made it fail in places that it was succeeding before um and your data sets are not static um pre-production when you don't have users yet your data sets are probably going to be synthetic queries you generated yourself

s1170
01:32:17.360 --> 01:32:20.080
or queries that you got an LLM to generate for you.

s1171
01:32:20.080 --> 01:32:20.800
That is fine.

s1172
01:32:20.800 --> 01:32:22.080
It gets you off the ground.

s1173
01:32:22.080 --> 01:32:28.320
As soon as real traffic starts coming in, you should turn your uh that into your eval sets.

s1174
01:32:28.320 --> 01:32:31.360
Uh real failures should replace imagined ones.

s1175
01:32:31.360 --> 01:32:36.400
The union of your failure cases and your passing cases is what people call your golden data set.

s1176
01:32:36.400 --> 01:32:42.320
Uh the curated labeled set you trust as ground truth for measuring quality.

s1177
01:32:42.320 --> 01:32:45.840
Uh now we're going to improve our agent.

s1178
01:32:45.840 --> 01:32:49.360
Uh but we are not going to handw write the fix.

s1179
01:32:49.360 --> 01:32:52.880
Uh as I mentioned, you can do all of this with skills.

s1180
01:32:52.880 --> 01:32:54.400
You can do all of this with a coding agent.

s1181
01:32:54.400 --> 01:32:59.679
I am assuming it being 2026 that absolutely everyone is using a coding agent to do all of their coding.

s1182
01:32:59.679 --> 01:33:11.760
Uh so why would you modify your prompts by yourself when you can get cloud code to pull down your failing traces uh modify your code for you uh and improve your application for you?

s1183
01:33:11.760 --> 01:33:13.760
That is what we're going to do here.

s1184
01:33:13.760 --> 01:33:17.679
Um Claude will pull down all of the failing traces.

s1185
01:33:17.679 --> 01:33:18.960
It will read all of the explanations.

s1186
01:33:18.960 --> 01:33:24.960
It will turn the explanations into action action that it takes on your codebase.

s1187
01:33:24.960 --> 01:33:27.760
Um and here's the important thing.

s1188
01:33:27.760 --> 01:33:28.800
Nothing here is guessed.

s1189
01:33:28.800 --> 01:33:31.040
Claude isn't inventing improvements out of thin air.

s1190
01:33:31.040 --> 01:33:33.199
You haven't just told it get better at this.

s1191
01:33:33.199 --> 01:33:35.520
You haven't just told it make no mistakes.

s1192
01:33:35.520 --> 01:33:40.880
Uh every change it proposes is traces back to a specific failure explanation.

s1193
01:33:40.880 --> 01:33:48.719
Uh so if the judge said that it lacks specific recommendations, uh Claude will read that and rewrite the prompt so that it does.

s1194
01:33:48.719 --> 01:33:54.960
Uh if the judge said that it presents risks without supporting evidence, then it will rewrite the prompt so that it fixes that too.

s1195
01:33:54.960 --> 01:33:57.280
This is datadriven prompt engineering.

s1196
01:33:57.280 --> 01:33:59.440
uh and now we can automate it.

s1197
01:33:59.440 --> 01:34:05.040
Uh so notice what we've passed in.

s1198
01:34:05.040 --> 01:34:08.960
Uh we've passed in our requirements, what we were trying to get done.

s1199
01:34:08.960 --> 01:34:17.360
Uh and we've passed in uh an improvement prompt telling it uh what you should be working on and how to get better at it.

s1200
01:34:17.360 --> 01:34:23.679
This is yet another LLM application and yet another thing that is nondeterministic and yet another way that it can go wrong.

s1201
01:34:23.679 --> 01:34:31.679
You could, if we had more time, meta-evaluate your meta evaluator of your meta evaluator and make sure that your meta evaluator is doing the right thing.

s1202
01:34:31.679 --> 01:34:36.719
But we're not doing that here because it gets too complicated and also we only have so much time.

s1203
01:34:36.719 --> 01:34:42.880
Um, but uh good requirements keep claud.

s1204
01:34:42.880 --> 01:34:50.639
So you should be spending time making sure that this uh evaluator uh is as accurate as you can make it.

s1205
01:34:50.639 --> 01:35:05.440
Uh so in this case we are taking this prompt and we are wiring them into uh our two pass uh we're taking the recommendations that we got from Claude based on pulling the

s1206
01:35:05.440 --> 01:35:18.320
uh failing traces uh and we are passing them back to anthropic and saying based on these explanations write a better prompt write a better research prompt write a better write prompt and then we're going to feed them back into our agent.

s1207
01:35:18.320 --> 01:35:24.159
Um, and now we're going to run an experiment.

s1208
01:35:24.159 --> 01:35:28.480
So, uh, over in the UI, we have data sets and experiments.

s1209
01:35:28.480 --> 01:35:31.199
We have my demo financial fails.

s1210
01:35:31.199 --> 01:35:44.239
Uh, we can, assuming I can scroll, wire up an improved agent, uh, using those new improved prompts.

s1211
01:35:44.239 --> 01:35:47.040
So, you can see the improved research prompt and the improved write prompt.

s1212
01:35:47.040 --> 01:35:50.080
this is otherwise exactly the same agent that it was before.

s1213
01:35:50.080 --> 01:35:55.199
Uh and you can create an evaluator uh with an experiment.

s1214
01:35:55.199 --> 01:36:01.840
Uh so what we're doing here is we're giving it a task.

s1215
01:36:01.840 --> 01:36:03.360
A task can be anything.

s1216
01:36:03.360 --> 01:36:07.040
So a task could be some subcomponent of what your agent did.

s1217
01:36:07.040 --> 01:36:17.120
So if we were running an evaluator that was about uh tool calls then your task could be just call this tool and see what the output of the tool is and then you would evaluate that.

s1218
01:36:17.120 --> 01:36:27.600
In this case my task is run the entire agent again from start to finish but it doesn't need to be uh and then we are running the same evaluation results

s1219
01:36:27.600 --> 01:36:32.719
uh against our uh against this new running of the task.

s1220
01:36:32.719 --> 01:36:34.320
So, we've changed the agent.

s1221
01:36:34.320 --> 01:36:36.719
We've rerun the agent, and we're rerunning the eval.

s1222
01:36:36.719 --> 01:36:39.280
And we're turning that into an experiment.

s1223
01:36:39.280 --> 01:36:41.520
This is what the experiment looks like.

s1224
01:36:41.520 --> 01:36:42.960
Experiments.run.

s1225
01:36:42.960 --> 01:36:44.239
We've given it a name.

s1226
01:36:44.239 --> 01:36:46.560
We've given it the data set that it should work on.

s1227
01:36:46.560 --> 01:36:48.159
Uh, we've given it the task.

s1228
01:36:48.159 --> 01:36:49.600
We've given it the evaluator.

s1229
01:36:49.600 --> 01:36:51.679
And we've told it to run.

s1230
01:36:51.679 --> 01:36:59.440
Uh, this gives us the results which are here.

s1231
01:37:00.400 --> 01:37:08.560
So for every single run uh we've got the output that it got and the new eval score.

s1232
01:37:08.560 --> 01:37:16.080
So uh as you can see this was a data set if you recall where previously all of them were not actionable.

s1233
01:37:16.080 --> 01:37:19.280
Uh the eval has run and you can see that some of them are now actionable.

s1234
01:37:19.280 --> 01:37:28.080
So we have improved and if we look at the overall score here actually I've nailed it because I had you know a lot of chances to get this right.

s1235
01:37:28.080 --> 01:37:30.000
I have oneshotted my prompt.

s1236
01:37:30.000 --> 01:37:36.000
uh from getting being wrong 50% of the time to being right right 100% of the time.

s1237
01:37:36.000 --> 01:37:38.880
Uh real eval won't work like that.

s1238
01:37:38.880 --> 01:37:42.320
Uh real iteration won't won't run so smoothly.

s1239
01:37:42.320 --> 01:37:43.760
That is why this is a graph.

s1240
01:37:43.760 --> 01:37:51.280
What you're expecting to do is run experiment after experiment and slowly watch the graph climb as your evals get more and more capable.

s1241
01:37:51.280 --> 01:37:54.960
That is the hill that I'm talking about that you're trying to climb.

s1242
01:37:54.960 --> 01:38:02.880
Uh so the key abstraction behind experiments is that a task is just a Python function.

s1243
01:38:02.880 --> 01:38:04.400
It takes an example from the data set.

s1244
01:38:04.400 --> 01:38:07.199
It runs whatever you run want and it returns an output.

s1245
01:38:07.199 --> 01:38:10.719
So AX doesn't care uh what the task does internally.

s1246
01:38:10.719 --> 01:38:17.840
So it could run the full agent or just one component like I said or it could call an API as long as it takes an input and returns an output.

s1247
01:38:17.840 --> 01:38:20.000
You can run experiments on it.

s1248
01:38:20.000 --> 01:38:23.520
The power of experiments is controlled comparison.

s1249
01:38:23.520 --> 01:38:29.679
So you take the same inputs, the same evaluators and the only thing that changed is the agent's prompts.

s1250
01:38:29.679 --> 01:38:33.040
Uh that means any differences in scores is attributable to your change.

s1251
01:38:33.040 --> 01:38:36.800
You're not wondering did it score higher because of my prompt change?

s1252
01:38:36.800 --> 01:38:41.840
Uh or did it score higher because the web search happened to return better results this time.

s1253
01:38:41.840 --> 01:38:45.520
Well, actually you're still wondering that a little bit because the agent is nondeterministic.

s1254
01:38:45.520 --> 01:38:48.560
So it could have done 10 web searches instead of five web searches.

s1255
01:38:48.560 --> 01:38:54.080
But you've eliminated a major source of variation which is the test cases.

s1256
01:38:55.280 --> 01:39:04.719
So like I said, uh my agent has one shot at this uh which is great, but in reality uh it's not going to be that simple.

s1257
01:39:04.719 --> 01:39:15.600
Um the eval iterate cycle that we're talking about here where you go through uh run an experiment, change your prompt, run the experiment again over and over and over.

s1258
01:39:15.600 --> 01:39:17.840
That is where the real value lies.

s1259
01:39:17.840 --> 01:39:20.639
Uh not in the score itself, but in the cycle.

s1260
01:39:20.639 --> 01:39:30.480
You run the evals, you look at the failures, you read the explanations, uh, you identify a pattern, and you adjust your prompt or as in this case, you get Claude to adjust your prompt for you.

s1261
01:39:30.480 --> 01:39:36.400
Um, did it get better is a question that can only be answered by evals.

s1262
01:39:36.400 --> 01:39:42.320
You have taken the definition of good and you have codified it in a way that is incredibly useful.

s1263
01:39:42.320 --> 01:39:48.000
Uh, so this allows you to move from I think it's working to I can prove it's working.

s1264
01:39:48.000 --> 01:39:52.639
A practical question that comes up fast uh is how many samples do you need?

s1265
01:39:52.639 --> 01:39:58.400
Um for workshop scale experiments like today, 12 to 20 examples gives you uh directional signal.

s1266
01:39:58.400 --> 01:40:01.600
For actual shipping decisions, you're going to want uh more.

s1267
01:40:01.600 --> 01:40:04.560
You're going to want something like 200 to 400 is a good target.

s1268
01:40:04.560 --> 01:40:10.960
But note that 200 to 400 is still like a humansized number that you could possibly accumulate over a couple of weeks.

s1269
01:40:10.960 --> 01:40:15.920
Uh it's not tens of thousands or millions like you would use if you were training an ML model.

s1270
01:40:15.920 --> 01:40:18.960
um there are diminishing returns.

s1271
01:40:18.960 --> 01:40:23.600
Uh to have your your margin of error, you have to quadruple your sample size.

s1272
01:40:23.600 --> 01:40:30.960
Uh so at some level, at some point, you have enough signal to be able to get along uh to get along.

s1273
01:40:30.960 --> 01:40:33.920
Um when you're iterating, where do you invest?

s1274
01:40:33.920 --> 01:40:35.840
Uh there's a hierarchy.

s1275
01:40:35.840 --> 01:40:38.560
Um data quality fixes have the highest impact.

s1276
01:40:38.560 --> 01:40:44.960
If your agent is searching the wrong sources or your knowledge base has stale information, then no amount of prompt iteration is going to help.

s1277
01:40:44.960 --> 01:40:46.880
you should fix the data first.

s1278
01:40:46.880 --> 01:40:48.960
Uh then prompting improvements are next.

s1279
01:40:48.960 --> 01:40:56.159
Few shot examples, explicit instructions, the kind of stuff that I mentioned today, uh constraints on what the agent should and shouldn't do.

s1280
01:40:56.159 --> 01:41:00.480
These are often the highest ROI change uh to your agent.

s1281
01:41:00.480 --> 01:41:03.760
Um model selection comes third.

s1282
01:41:03.760 --> 01:41:07.360
Sometimes a more capable model solves problems that prompting can't.

s1283
01:41:07.360 --> 01:41:09.760
Uh but it will also cost more and be slower.

s1284
01:41:09.760 --> 01:41:11.840
So that is a trade-off that you have to make.

s1285
01:41:11.840 --> 01:41:15.199
Uh the thing to not spend a lot of time on is hyperparameter tuning.

s1286
01:41:15.199 --> 01:41:20.480
Things like temperature and top p. It's very tempting to just go for them because they're easy to tweak.

s1287
01:41:20.480 --> 01:41:31.199
Uh they very seldom make a high lever change like prompting would or improving the quality of your data would so you can try them but they should be the last thing you try.

s1288
01:41:31.199 --> 01:41:33.679
A practice worth knowing is eval driven development.

s1289
01:41:33.679 --> 01:41:36.320
Write the eval before you build the feature.

s1290
01:41:36.320 --> 01:41:44.880
Uh if you want your agent to always verify customer identity before it does something before processing a refund for instance then write the eval that checks for that first.

s1291
01:41:44.880 --> 01:41:49.440
That gives you a clear measurable definition of what good looks like of what done looks like.

s1292
01:41:49.440 --> 01:41:53.760
Then you can build a feature uh until the eval passes.

s1293
01:41:53.760 --> 01:41:58.000
This is the same philosophy as test-driven development.

s1294
01:41:59.040 --> 01:42:06.800
Uh and something I love about eval driven development is that the people closest to product requirements are best positioned to define success.

s1295
01:42:06.800 --> 01:42:13.760
So product managers, customer service uh people, even salespeople can contribute to eval tasks.

s1296
01:42:13.760 --> 01:42:17.440
They can tell you what looks good and what doesn't look good in practice.

s1297
01:42:17.440 --> 01:42:18.960
Uh they don't need to write code.

s1298
01:42:18.960 --> 01:42:23.199
They just need to describe in natural language what good looks like to them.

s1299
01:42:23.199 --> 01:42:26.960
And you can boil that down into your Eval rubric.

s1300
01:42:26.960 --> 01:42:36.320
Uh, and so far, uh, I've shown you how to do things both offline and online.

s1301
01:42:36.320 --> 01:42:39.280
Uh, we've been doing everything in a notebook.

s1302
01:42:39.280 --> 01:42:43.679
Uh, but you're not really done when your dev experiment tra pass passes.

s1303
01:42:43.679 --> 01:42:48.639
Uh, you're done when your agent uh, keeps performing in production on traffic that you've never seen.

s1304
01:42:48.639 --> 01:42:58.480
Um and it's worth spending a couple of minutes talking about that even though uh we're going to stick in the notebook uh for this workbook for this work session.

s1305
01:42:58.480 --> 01:43:05.840
Um the first thing is online eval uh like I said uh and like I showed you earlier online evals are exactly what they sound like.

s1306
01:43:05.840 --> 01:43:14.560
They take the evaluators that you wrote today uh the ticker check the faithfulness the actionability uh and run them automatically on incoming production traces.

s1307
01:43:14.560 --> 01:43:17.199
The same a val running on new data.

s1308
01:43:17.199 --> 01:43:19.920
Uh I showed you how to run this on a trace.

s1309
01:43:19.920 --> 01:43:22.239
AX lets you do this at three scopes.

s1310
01:43:22.239 --> 01:43:24.639
You can run your evals at the span level.

s1311
01:43:24.639 --> 01:43:30.320
So you can say this particular LLM call, this particular tool call, run a trace on it every single time.

s1312
01:43:30.320 --> 01:43:32.320
Sorry, run a eval on it every single time.

s1313
01:43:32.320 --> 01:43:39.600
Or you can run it at the trace level where it has the full tree to look at uh and it can evaluate everything from start to finish.

s1314
01:43:39.600 --> 01:43:44.000
Um you pick the scope that matches the question that you're asking.

s1315
01:43:44.000 --> 01:43:50.560
Um, and you don't need to run evals on every single prediction trace I uh production trace rather.

s1316
01:43:50.560 --> 01:43:55.440
Um, LLM judges cost money and at production uh scale those costs can add up.

s1317
01:43:55.440 --> 01:43:58.639
So you sample 10% of your traffic or 1% of your traffic.

s1318
01:43:58.639 --> 01:44:08.560
Uh, and that gives you a directional signal without you having to run an expensive LLM as a judge on every single iteration, every single uh, instance of your application.

s1319
01:44:08.560 --> 01:44:19.280
Um if you don't want to write your evals by hand uh we have good news for you which is that uh we have built an agent into AX itself.

s1320
01:44:19.280 --> 01:44:20.880
It is called Alex.

s1321
01:44:20.880 --> 01:44:34.960
Um, you can describe your eval in plain English or you can get your non-technical uh colleagues to describe a pro describe the definition of good in plain English and Alex will actually manipulate the UI for you

s1322
01:44:34.960 --> 01:44:42.320
uh create an eval write the rubric set all the scores correctly get all the configuration correct and then just tell you that it's done.

s1323
01:44:42.320 --> 01:44:45.600
Uh this is especially useful for non-engineers.

s1324
01:44:45.600 --> 01:44:50.000
Um and the full loop in production looks like this.

s1325
01:44:50.000 --> 01:44:57.199
So you go from application uh which produces traces to online evals which grade those traces and produce labels and explanations.

s1326
01:44:57.199 --> 01:45:01.360
The eval labels feed into monitors which we didn't cover today.

s1327
01:45:01.360 --> 01:45:05.119
Um the monitors can alert your team when something goes wrong.

s1328
01:45:05.119 --> 01:45:14.480
Um the team can investigate, find the failing traces, save them as a regression data set, um improve the agent, run an experiment to verify and ship the fix.

s1329
01:45:14.480 --> 01:45:16.719
And then the loop can run again.

s1330
01:45:16.719 --> 01:45:24.639
Um, as new production traffic accumulates, you will accumulate new failure modes, you will accumulate new edge cases.

s1331
01:45:24.639 --> 01:45:31.840
Uh, you're always going to be uh iterating on your evals and iterating on your production monitoring.

s1332
01:45:32.960 --> 01:45:34.800
But it is a payoff that compounds.

s1333
01:45:34.800 --> 01:45:38.320
Every failure that your online evals catch becomes a new test case.

s1334
01:45:38.320 --> 01:45:42.320
You save it to a data set and now it's a regression test for your next round of changes.

s1335
01:45:42.320 --> 01:45:46.480
Over time, this creates a data set that is unique to your application.

s1336
01:45:46.480 --> 01:45:53.040
Uh so your specific failure modes, your specific users, your specific quality bar, nobody else has that data.

s1337
01:45:53.040 --> 01:45:58.719
Uh and that is a competitive advantage that grows every single time your agent runs.

s1338
01:45:59.360 --> 01:46:03.440
There's one last pattern that I want to leave you with uh which is the most powerful one I know.

s1339
01:46:03.440 --> 01:46:05.760
We did the simplest version today.

s1340
01:46:05.760 --> 01:46:09.440
We fed the judges explanations back to Claude and had it rewrite two prompts.

s1341
01:46:09.440 --> 01:46:11.840
This works beautifully for two prompts.

s1342
01:46:11.840 --> 01:46:13.920
Uh but a real system isn't two prompts.

s1343
01:46:13.920 --> 01:46:18.960
prompts scattered across many files, retrieval settings, tool definitions, all tangled together.

s1344
01:46:18.960 --> 01:46:20.960
Rewriting that isn't a single API call.

s1345
01:46:20.960 --> 01:46:26.480
It's a job for a coding agent uh that can see your entire repository.

s1346
01:46:26.480 --> 01:46:28.320
Um so this is the pattern.

s1347
01:46:28.320 --> 01:46:37.520
You want to export your failing traces from ax along with their explanations uh and hand the whole batch to cloud code or cursor or another coding agent as context.

s1348
01:46:37.520 --> 01:46:47.360
Uh this is where the Arise skills plugin that I mentioned at the beginning uh works really well because you don't have because you don't have to know how to call our API.

s1349
01:46:47.360 --> 01:46:49.360
You don't have to know how to do any of that stuff.

s1350
01:46:49.360 --> 01:46:57.920
You give it the skills and then tell cloud code, hey, pull down some traces, see what's wrong, and make make my application better and it works.

s1351
01:46:57.920 --> 01:47:02.719
Uh there should be two guardrails on that approach first.

s1352
01:47:02.719 --> 01:47:06.239
However, first you should feed it your requirements too and not just the failure explanations.

s1353
01:47:06.239 --> 01:47:10.800
So the goal isn't make the eval pass because otherwise it will cheat.

s1354
01:47:10.800 --> 01:47:16.000
It will include all of your test data into its eval uh and automatically pass all of your evals.

s1355
01:47:16.000 --> 01:47:25.440
So you have to make uh um your you have to make sure that your requirements to your cloud code are clear that you don't want it just to pass the evals.

s1356
01:47:25.440 --> 01:47:29.360
You want it to do this specific thing which then makes the evals pass.

s1357
01:47:29.360 --> 01:47:34.159
Um and second, you should tell it to find themes and not chase individual failures.

s1358
01:47:34.159 --> 01:47:46.159
It's very tempting for claude code to look at one particular failed trace and write a whole bunch of code to fix that particular failed trace when there are actually 10 failing traces that are all the same co cause that it should be focused on instead.

s1359
01:47:46.159 --> 01:47:50.960
Uh I'm not going to do all of that time all of that because I only have 12 minutes left.

s1360
01:47:50.960 --> 01:48:00.639
Uh but um once you have eval this is one of the highest leverage things that you can do with them.

s1361
01:48:01.119 --> 01:48:03.679
So let's step back and look at the whole shape of it.

s1362
01:48:03.679 --> 01:48:05.440
Uh production produces traces.

s1363
01:48:05.440 --> 01:48:08.320
Online eval them and produce explanations.

s1364
01:48:08.320 --> 01:48:11.600
A coding agent reads the explanations and proposes fixes.

s1365
01:48:11.600 --> 01:48:14.960
And experiments verify that the fixes work and then you ship.

s1366
01:48:14.960 --> 01:48:20.719
The production traffic that comes back gets graded by the same online evals that started the loop.

s1367
01:48:20.719 --> 01:48:25.119
Uh and this is the software development life cycle closing in on itself.

s1368
01:48:25.119 --> 01:48:31.119
uh your AI software is helping you improve your AI software and AX is the substrate uh that ties it all together.

s1369
01:48:31.119 --> 01:48:38.480
Your traces, the evals, the explanations, the data sets, the experiments, the online evals and the monitors.

s1370
01:48:39.119 --> 01:48:43.520
So just to recap, we instrumented an agent with two lines of code.

s1371
01:48:43.520 --> 01:48:45.119
We traced it and we read our data.

s1372
01:48:45.119 --> 01:48:48.000
We wrote a code eval and built-in LLM evals.

s1373
01:48:48.000 --> 01:48:49.840
We wrote a custom rubric from scratch.

s1374
01:48:49.840 --> 01:48:52.480
We validated our judge against human labels.

s1375
01:48:52.480 --> 01:48:54.480
Uh we saved failures at the data set.

s1376
01:48:54.480 --> 01:48:56.719
We had Claude improve the prompts.

s1377
01:48:56.719 --> 01:48:59.199
We ran an experiment to prove that it worked.

s1378
01:48:59.199 --> 01:49:05.280
Uh and we saw how AX takes all of that into production uh with online evals.

s1379
01:49:05.280 --> 01:49:09.119
Uh this is the full loop and now you know all of it.

s1380
01:49:09.119 --> 01:49:11.440
However, you don't have to do all of it at once.

s1381
01:49:11.440 --> 01:49:16.000
This has been two hours of extremely dense information and I'm aware of that.

s1382
01:49:16.000 --> 01:49:18.239
You can start very very small.

s1383
01:49:18.239 --> 01:49:20.480
Uh you can start by reading your traces.

s1384
01:49:20.480 --> 01:49:29.040
Just add those two lines of code and start looking at your traces and that is already that is immediately going to give you information about what your agent is up to that you didn't know.

s1385
01:49:29.040 --> 01:49:34.880
Uh 15 minutes of reading real outputs will teach you more about your application than an hour of building dashboards.

s1386
01:49:34.880 --> 01:49:36.560
Then write one code eval.

s1387
01:49:36.560 --> 01:49:40.320
Code evals are easier than LME evals and they're faster and they're cheaper.

s1388
01:49:40.320 --> 01:49:42.800
Uh and make it for the thing that matters most.

s1389
01:49:42.800 --> 01:49:45.600
So correctness, faithfulness, whatever fits your use case.

s1390
01:49:45.600 --> 01:49:51.040
Run them, look at the results, see what patterns emerge and build from there.

s1391
01:49:51.040 --> 01:49:52.639
Evals are infrastructure.

s1392
01:49:52.639 --> 01:49:53.760
They are not an afterthought.

s1393
01:49:53.760 --> 01:49:56.560
Some teams create evals at the very start of development.

s1394
01:49:56.560 --> 01:49:59.440
Add others add them later once they are at scale.

s1395
01:49:59.440 --> 01:50:04.960
Uh but either way uh the important thing is that you treat evals as a core part of your system.

s1396
01:50:04.960 --> 01:50:10.639
They should be as routine as unit tests and the value compounds but only if you keep investing.

s1397
01:50:10.639 --> 01:50:18.320
Each time a regression shows up before it show reaches your users instead of after you'll understand why all of this work is worth doing.

s1398
01:50:18.320 --> 01:50:20.000
Don't hope for great.

s1399
01:50:20.000 --> 01:50:27.199
You can get to great systematically by specifying it, measuring it, and improving towards it.

s1400
01:50:27.199 --> 01:50:30.560
So, now is the time to try this for real on an app of your own.

s1401
01:50:30.560 --> 01:50:35.760
Uh, you've already got an AX account, I hope, by the time the Wi-Fi kicked in.

s1402
01:50:35.760 --> 01:50:43.840
Uh, the docs at arise.com uh have companion notebooks with runnable code for everything that we covered today.

s1403
01:50:43.840 --> 01:50:52.320
Uh, and if you want your coding agent to do the heavy lifting, you can install the Arise plugins, uh, the Arise skills plugin with one npx command.

s1404
01:50:52.320 --> 01:50:55.920
Uh, and that is in that link is in today's notebook.

s1405
01:50:55.920 --> 01:51:04.639
Uh, if you've made it all the way to the end of this, you can also get a free year of Arise Pro uh, using that code that's at the on the screen right now.

s1406
01:51:04.639 --> 01:51:05.679
And that is it.

s1407
01:51:05.679 --> 01:51:09.040
Thank you all for your time and attention.