WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=SHMCGrNJJGc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/SHMCGrNJJGc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:13.759
My name is Lauri Voss.

s3
00:00:13.759 --> 00:00:16.160
I'm head of developer relations at Arise AI.

s4
00:00:16.160 --> 00:00:21.199
Uh I used to do JavaScript at npm, Inc. way back in the day, so you may remember me from then.

s5
00:00:21.199 --> 00:00:28.080
Uh but these days I think about AI applications and how to make them run and how to make them actually work.

s6
00:00:28.080 --> 00:00:31.679
Um, I had this slide up for the 15 minutes before.

s7
00:00:31.679 --> 00:00:32.880
I hope you've seen it already.

s8
00:00:32.880 --> 00:00:36.000
If not, now is the time to get started.

s9
00:00:36.000 --> 00:00:40.960
Um, this is a moderately hands-on workshop.

s10
00:00:40.960 --> 00:00:43.520
There's I've already built an agent for you.

s11
00:00:43.520 --> 00:00:45.200
It's sitting there in that repo.

s12
00:00:45.200 --> 00:00:48.879
Uh, you will get to play with the prompts and run actual evals.

s13
00:00:48.879 --> 00:00:52.000
Um, but it's got checkpoints all the way through.

s14
00:00:52.000 --> 00:00:59.920
So if you get lost, you can just switch to the right checkpoint and uh uh jump to the part where it's already working.

s15
00:00:59.920 --> 00:01:02.640
Um you're going to need node 22 or later.

s16
00:01:02.640 --> 00:01:05.840
All of this all of these examples are going to be in Typescript.

s17
00:01:05.840 --> 00:01:10.240
Um because you know it's me, I'm going to use JavaScript if I can.

s18
00:01:10.240 --> 00:01:16.479
Uh and uh you're going to be installing and running Phoenix on your own laptop.

s19
00:01:16.479 --> 00:01:22.240
uh on conference Wi-Fi that can be a challenge, but I tried it this morning and it came through pretty quickly.

s20
00:01:22.240 --> 00:01:24.960
So, uh UVX is magic.

s21
00:01:24.960 --> 00:01:27.680
Um you're not going to need to run any Python in this thing.

s22
00:01:27.680 --> 00:01:33.280
All of the Python is going to be run for you on your behalf by JavaScript, which is nice.

s23
00:01:33.280 --> 00:01:36.320
Um I think that's it for the setup.

s24
00:01:36.320 --> 00:01:44.320
Um so, what we're going to do today is um uh oh yeah, sorry, one more bit of setup.

s25
00:01:44.320 --> 00:01:51.920
Once Phoenix is installed, uh you should open up your browser on localhost 606 so that you can see what it looks like.

s26
00:01:51.920 --> 00:01:58.640
Um it's going to be empty right now, but we are going to run the commands that will fill it with data.

s27
00:01:58.640 --> 00:02:03.360
Um you will also need an anthropic key.

s28
00:02:03.360 --> 00:02:05.680
All of my examples are using Claude.

s29
00:02:05.680 --> 00:02:13.440
Uh you can do all of this stuff using any LLM provider in the world, but I had to pick an example, so I picked the one that I use every day.

s30
00:02:13.440 --> 00:02:15.760
So, this one is using Claude.

s31
00:02:15.760 --> 00:02:20.319
Um, who needs help with the setup?

s32
00:02:20.319 --> 00:02:21.440
Show of hands.

s33
00:02:21.440 --> 00:02:22.480
Just shout at me.

s34
00:02:22.480 --> 00:02:24.720
It's fine.

s35
00:02:25.040 --> 00:02:25.440
All right.

s36
00:02:25.440 --> 00:02:26.720
So, the first 10.

s37
00:02:26.720 --> 00:02:29.040
What's up?

s38
00:02:31.040 --> 00:02:33.280
Just getting started.

s39
00:02:33.280 --> 00:02:35.599
Okay, great.

s40
00:02:35.599 --> 00:02:36.000
Yeah.

s41
00:02:36.000 --> 00:02:44.000
So, the first 10 minutes are theory, which will give you some time to catch up on the setup uh while I'm talking.

s42
00:02:47.360 --> 00:02:49.599
So, let's get started with the theory.

s43
00:02:49.599 --> 00:02:53.120
Um, here's the situation I'm assuming most of you are in.

s44
00:02:53.120 --> 00:02:56.239
You have already built an application that has an LLM in it.

s45
00:02:56.239 --> 00:02:58.160
Uh, maybe a chatbot, maybe an agent.

s46
00:02:58.160 --> 00:03:01.120
You tried it a bunch of times and it looks pretty good.

s47
00:03:01.120 --> 00:03:06.640
Uh, then you shipped it and somebody used it in a way that you weren't expecting and it broke.

s48
00:03:06.640 --> 00:03:10.000
Um, or you changed the prompt to fix one thing.

s49
00:03:10.000 --> 00:03:13.120
uh and you have this nagging feeling that you broke something else.

s50
00:03:13.120 --> 00:03:15.120
Uh and there's no way of knowing.

s51
00:03:15.120 --> 00:03:17.360
That is the problem that today is about.

s52
00:03:17.360 --> 00:03:19.920
That is the problem that evals are here to solve.

s53
00:03:19.920 --> 00:03:23.200
How do you know if your agent is any good?

s54
00:03:23.200 --> 00:03:26.319
How do you know if your change made things better or worse?

s55
00:03:26.319 --> 00:03:34.239
Uh and how do you stop it getting worse when you make changes uh without having to read every output yourself?

s56
00:03:36.159 --> 00:03:46.640
My promise by the end of today is that you're going to have a working agent traced end to end with code evals with LLM judge evals and with a custom eval that you wrote yourself.

s57
00:03:46.640 --> 00:03:51.120
Uh and you'll have checked whether your judge can be trusted.

s58
00:03:51.120 --> 00:03:54.959
You'll have run an experiment to prove that your changes worked.

s59
00:03:54.959 --> 00:04:01.599
Uh and you'll have have all of it running as a test suite with npm tests so you can put it into npmci.

s60
00:04:01.599 --> 00:04:04.560
Um, that is the bit that I'm most excited about.

s61
00:04:04.560 --> 00:04:06.400
So, it's going to be the last thing that we do.

s62
00:04:06.400 --> 00:04:08.239
I see some people have walked in late.

s63
00:04:08.239 --> 00:04:14.319
So, let me just go back to the setup slide very quickly so that they're not completely lost.

s64
00:04:17.840 --> 00:04:19.199
Right.

s65
00:04:19.199 --> 00:04:23.199
Remember this repo because you're going to want this repo.

s66
00:04:32.160 --> 00:04:34.080
How is the Wi-Fi working for people?

s67
00:04:34.080 --> 00:04:36.479
Is it Are you getting places?

s68
00:04:36.479 --> 00:04:37.280
Excellent.

s69
00:04:37.280 --> 00:04:38.720
Okay.

s70
00:04:38.720 --> 00:04:50.080
I did this at World's Fair and uh every and I accidentally had a repo that was full of product images that looked really great in the demo, but it was 400 megabytes of product images and it just complete failure.

s71
00:04:50.080 --> 00:04:53.680
Nobody was able to get even the repo down.

s72
00:04:54.080 --> 00:04:57.280
So, I've learned my lesson and today has no images.

s73
00:04:57.280 --> 00:05:01.360
Uh, so let's cover the basics.

s74
00:05:01.360 --> 00:05:02.639
What is an eval?

s75
00:05:02.639 --> 00:05:20.400
The shortest version, the one that I use whenever I'm talking to somebody who's been an engineer for a while is that uh there are these things called traces and they are basically logs and there's these things called evals and they're basically tests and evals are using are running tests on your traces.

s76
00:05:20.400 --> 00:05:30.320
Um the difference between an eval and a unit test is that uh evals are for outputs that are never exactly the same twice, which is to say the non-deterministic

s77
00:05:30.320 --> 00:05:32.800
outputs that you get from LLMs.

s78
00:05:32.800 --> 00:05:37.600
Um traces record what your AI did at an incredible level of detail.

s79
00:05:37.600 --> 00:05:43.600
So every agent step, every tool call, every LLM call, the inputs and outputs at every single step.

s80
00:05:43.600 --> 00:05:46.000
A trace is made up of what we call spans.

s81
00:05:46.000 --> 00:05:48.320
Each span is one step in the execution.

s82
00:05:48.320 --> 00:05:50.880
You can think of it as just a line in the log.

s83
00:05:50.880 --> 00:05:56.960
Um, and a so a whole agent turn is a span that has other spans nested inside of it.

s84
00:05:56.960 --> 00:06:04.639
Uh, and each one records its input, its output, how long it took, how many tokens it used, uh, and sometimes how much it cost.

s85
00:06:04.639 --> 00:06:10.160
So traces tell you what happened and evals tell you whether it was any good and that is the whole mental model for today.

s86
00:06:10.160 --> 00:06:14.400
This is very far from being rocket science.

s87
00:06:15.280 --> 00:06:17.919
Looks good to me is not a test suite.

s88
00:06:17.919 --> 00:06:21.280
Um, but it's how most AI features get shipped.

s89
00:06:21.280 --> 00:06:24.479
You build it, you try it a few times, does it look right?

s90
00:06:24.479 --> 00:06:25.360
Yeah, that looks good.

s91
00:06:25.360 --> 00:06:26.639
You ship it.

s92
00:06:26.639 --> 00:06:30.880
Uh, and it probably does look good on the three things that you tried.

s93
00:06:30.880 --> 00:06:33.039
Um, but three tries is not a test suite.

s94
00:06:33.039 --> 00:06:35.360
And the usual fix is unit tests.

s95
00:06:35.360 --> 00:06:43.600
Um, but unit tests don't work here because the same prompt is going to give a different answer text every single time you run your application.

s96
00:06:43.600 --> 00:06:46.479
Uh, so two completely different answers can both be right.

s97
00:06:46.479 --> 00:06:49.759
a thousand different answers can both can all be right.

s98
00:06:49.759 --> 00:06:51.840
Uh just differently right.

s99
00:06:51.840 --> 00:06:54.720
So there's no expected string to assert against.

s100
00:06:54.720 --> 00:06:58.000
Uh so humans tend to fall back on human judgment.

s101
00:06:58.000 --> 00:07:01.169
They look at the strings being output and they're like well this looks fine.

s102
00:07:01.169 --> 00:07:01.440
[snorts]

s103
00:07:01.440 --> 00:07:03.360
Um but that doesn't scale.

s104
00:07:03.360 --> 00:07:09.840
It doesn't catch regressions and it certainly doesn't run in CI where there are no humans to watch things.

s105
00:07:09.919 --> 00:07:16.080
Uh what that looks like in practice is you change your system prompts to fix the tone and the tone gets better.

s106
00:07:16.080 --> 00:07:18.880
Uh but now the bot is making up product features.

s107
00:07:18.880 --> 00:07:21.520
Uh and you don't find out until a customer does.

s108
00:07:21.520 --> 00:07:25.039
Every prompt change can affect every kind of input your users send.

s109
00:07:25.039 --> 00:07:27.280
Uh and you can't test all of those by hand.

s110
00:07:27.280 --> 00:07:33.360
So you end up playing a game of whack-a-ole where you fix one thing and something else breaks and you fix that and something else breaks.

s111
00:07:33.360 --> 00:07:41.599
uh evals fix that by giving you a number that you can track and can compare and act on.

s112
00:07:42.960 --> 00:07:50.880
This matters more every month because uh every couple of weeks somebody releases a new model and you're like, "Oh, I want to switch to the new model."

s113
00:07:50.880 --> 00:08:00.879
Uh but you have no way of knowing which things the new model is going to be better at uh and which things the new model is going to be worse at unless you have a set of evals that you can run things through.

s114
00:08:00.879 --> 00:08:03.120
Uh, and that's exactly what we're going to do today.

s115
00:08:03.120 --> 00:08:17.599
We're going to start with uh Claude 4.5 and then we are going to upgrade it to Claude uh Haiku 5.5 um uh which was conveniently released like a week ago making this demo extremely easy to put together.

s116
00:08:17.599 --> 00:08:20.080
I was like I want to upgrade the model for this demo.

s117
00:08:20.080 --> 00:08:22.639
So I'm going to make that the demo.

s118
00:08:22.639 --> 00:08:31.440
Uh so with eval you run the suite you can compare the numbers and by the end of the afternoon you know uh whether you can upgrade your model.

s119
00:08:31.440 --> 00:08:33.440
Um and there are a few kinds of evals.

s120
00:08:33.440 --> 00:08:34.959
The first kind is code evals.

s121
00:08:34.959 --> 00:08:38.399
Code evals are fast and free and a little bit brittle.

s122
00:08:38.399 --> 00:08:41.440
They are very very similar to unit tests.

s123
00:08:41.440 --> 00:08:46.959
They are just ordinary functions where you basically assert a string.

s124
00:08:46.959 --> 00:08:54.320
There are a bunch of techniques you can use to make them more flexible than a straight unit test that asserts a basic string.

s125
00:08:54.320 --> 00:08:56.959
Uh but fundamentally that's what they're doing.

s126
00:08:56.959 --> 00:09:01.600
So the things that you can test with a Kodi valal are stuff like did the output parse as JSON?

s127
00:09:01.600 --> 00:09:03.360
Is it under 500 words?

s128
00:09:03.360 --> 00:09:05.680
Did it mention the thing that I asked about?

s129
00:09:05.680 --> 00:09:09.760
The advantage of them is that they run in milliseconds and they cost nothing to run.

s130
00:09:09.760 --> 00:09:17.040
Um, but there are lots of valid answers that a string match can't tell you about.

s131
00:09:17.040 --> 00:09:22.560
Um, and for that you need the second kind of eval which is the LLM is a judge eval.

s132
00:09:22.560 --> 00:09:26.320
Uh, which is what most people think about when they're thinking about evals.

s133
00:09:26.320 --> 00:09:31.166
In this one you use a second LLM to grade the output of the first LLM.

s134
00:09:31.166 --> 00:09:31.519
[snorts]

s135
00:09:31.519 --> 00:09:34.959
Um, you grade it against what is called a rubric.

s136
00:09:34.959 --> 00:09:43.279
There are a bunch of terms in uh eval land that are borrowed from ML uh that I find obscure the meaning of what's going on.

s137
00:09:43.279 --> 00:09:46.800
One of them is traces and one of them is evals and the other one is rubric.

s138
00:09:46.800 --> 00:09:48.720
I'm like rubric just means rules.

s139
00:09:48.720 --> 00:09:50.959
Why do we have a special word for it?

s140
00:09:50.959 --> 00:09:56.959
Anyway, uh L&M judges understand meaning not just strings.

s141
00:09:56.959 --> 00:09:58.800
So stuff like is this accurate?

s142
00:09:58.800 --> 00:10:00.560
Does it stick to the source documents?

s143
00:10:00.560 --> 00:10:02.720
Is the tone right for a customer?

s144
00:10:02.720 --> 00:10:10.240
Um, the catch is that because the LLM judge isn't itself an LLM, it has all of the flaws that LLMs have.

s145
00:10:10.240 --> 00:10:11.920
So, it can be unpredictable.

s146
00:10:11.920 --> 00:10:13.760
It is never the same twice.

s147
00:10:13.760 --> 00:10:15.839
Uh, and it can make mistakes.

s148
00:10:15.839 --> 00:10:20.000
Uh, they also cost money because you're using an LLM.

s149
00:10:20.000 --> 00:10:22.240
Uh, and they can be confidently wrong.

s150
00:10:22.240 --> 00:10:23.360
So, you have to check them.

s151
00:10:23.360 --> 00:10:26.560
And that is one of the things that we're going to do today.

s152
00:10:26.560 --> 00:10:30.560
Then there's a third kind of eval uh, which is the human eval.

s153
00:10:30.560 --> 00:10:35.920
somebody who knows the domain, who can read the output and grade it as an expert.

s154
00:10:35.920 --> 00:10:39.519
Uh, humans are the gold standard for quality.

s155
00:10:39.519 --> 00:10:42.160
Um, they're also slow and expensive.

s156
00:10:42.160 --> 00:10:44.480
Uh, and they get bored and start skimming.

s157
00:10:44.480 --> 00:10:48.560
So, uh, you want to use all three at the same time.

s158
00:10:48.560 --> 00:10:50.320
These are not competing approaches.

s159
00:10:50.320 --> 00:10:51.680
They are complimentary approaches.

s160
00:10:51.680 --> 00:10:56.880
And we're going to talk about a little bit about how you use them in a complimentary way later.

s161
00:10:56.959 --> 00:10:59.680
So which eval do you use when?

s162
00:10:59.680 --> 00:11:02.880
Use code when the answer is deterministic.

s163
00:11:02.880 --> 00:11:07.200
So things like formats and lengths and required fields and band phrases.

s164
00:11:07.200 --> 00:11:10.480
Uh but use an LLM judge when you need understanding.

s165
00:11:10.480 --> 00:11:11.279
Is it accurate?

s166
00:11:11.279 --> 00:11:12.079
Is it grounded?

s167
00:11:12.079 --> 00:11:13.360
Is it useful?

s168
00:11:13.360 --> 00:11:17.920
Uh and use humans for the kinds of failures you haven't seen before.

s169
00:11:17.920 --> 00:11:19.600
Stuff that you haven't anticipated.

s170
00:11:19.600 --> 00:11:23.680
And to check that your LLM judges are judging correctly.

s171
00:11:24.320 --> 00:11:31.120
Um, all of that applies to any LLM app, but agents in particular make all of this stuff harder because agents are more complicated.

s172
00:11:31.120 --> 00:11:35.600
A single simp a single LLM call is relatively simple.

s173
00:11:35.600 --> 00:11:37.680
You give it an input, you get an output.

s174
00:11:37.680 --> 00:11:41.680
Uh, but an agent tends to chain a whole bunch of steps together.

s175
00:11:41.680 --> 00:11:53.760
So, one of the things that can go wrong is very early on in the chain, the agent makes a mistake and it cascades through all of the rest of the steps in your agent and each one gets progressively more off the rails.

s176
00:11:53.760 --> 00:11:59.120
Um, and one of the biggest problems is being confidently wrong.

s177
00:11:59.120 --> 00:12:01.920
Confidently wrong is worse than obviously wrong.

s178
00:12:01.920 --> 00:12:04.480
Uh, so I talked about the cascading failure.

s179
00:12:04.480 --> 00:12:12.480
Um, one example of that is say you've written a financial analysis agent, which is the kind of agent that I'm going to show you today.

s180
00:12:12.480 --> 00:12:16.079
Um, and you ask it for a report on Tesla.

s181
00:12:16.079 --> 00:12:21.760
It searches for Tesla and instead of getting results about the car company, it gets results about the 18th century inventor.

s182
00:12:21.760 --> 00:12:26.880
It writes an beautiful, perfectly formatted report about an 18th century inventor.

s183
00:12:26.880 --> 00:12:32.399
Uh, and none of your checks fail because you're like, it's well formatted.

s184
00:12:32.399 --> 00:12:33.760
It's in a great tone.

s185
00:12:33.760 --> 00:12:35.360
Everything is grammatical.

s186
00:12:35.360 --> 00:12:38.000
Uh, it's just about completely the wrong subject.

s187
00:12:38.000 --> 00:12:48.160
Uh, and that is the kind of failure that is uh um insidious with uh agents.

s188
00:12:48.160 --> 00:12:50.720
Agents can also fail in the opposite direction.

s189
00:12:50.720 --> 00:13:05.839
Um, Anthropic uh, who make Claude ran a benchmark called Ta2 bench which simulates customer service conversations and they uh, gave it this task where the user asked to change their flight.

s190
00:13:05.839 --> 00:13:18.399
Uh, and the test according to the test the user was supposed to fail the test uh, or rather the agent was supposed to fail because uh, the person was in economy and you're not allowed to switch seats in economy.

s191
00:13:18.399 --> 00:13:23.279
Uh but what cloud found is that they can upgrade the person to first class.

s192
00:13:23.279 --> 00:13:29.200
You're allowed to change your seat in first class and then they downgraded them back after changing their seat back down to economy.

s193
00:13:29.200 --> 00:13:37.360
Uh and pass and like did the thing that the user asked for and the test said no you failed because you weren't supposed to be able to do that.

s194
00:13:37.360 --> 00:13:41.839
Um but the agent had been smarter than the rules that it was in.

s195
00:13:41.839 --> 00:13:50.079
Um so you have to expect that uh when you are building eval that sometimes the agent is going to come up with a solution to the problem that you did not expect.

s196
00:13:50.079 --> 00:13:53.920
So your tests have to be very flexible.

s197
00:13:53.920 --> 00:13:57.920
Um here's what an eval result generally looks like.

s198
00:13:57.920 --> 00:14:01.600
Uh it comes with a score, it comes with a label and it comes with an explanation.

s199
00:14:01.600 --> 00:14:04.720
The score is a number usually just zero or one.

s200
00:14:04.720 --> 00:14:09.760
Uh and the label is just a word that the human can read that matches what the score means.

s201
00:14:09.760 --> 00:14:13.279
Uh so stuff like pass and fail, actionable and not actionable.

s202
00:14:13.279 --> 00:14:20.880
Uh and LLM judges add a third thing which is an explanation of why you got the score you did, why you got the label you did.

s203
00:14:20.880 --> 00:14:25.680
Um and it is the most valuable part of an LLM judge.

s204
00:14:25.680 --> 00:14:28.720
Uh because the explanation is where the value is.

s205
00:14:28.720 --> 00:14:36.240
It tells you what was wrong and what was missing and it tells you a a lot of the time what to fix in your prompt to get this right.

s206
00:14:36.240 --> 00:14:50.320
Uh this is especially useful when you're running your evals at scale because you can get, you know, 50, 100, a thousand explanations from your LLM and you can get another LLM to read all of those explanations and be like, it looks like there is a pattern in these explanations.

s207
00:14:50.320 --> 00:14:52.399
It looks like we have a systematic problem.

s208
00:14:52.399 --> 00:14:59.680
And you can just feed them to your agent and your agent goes, okay, I see 50 traces all doing the same all with the same explanation of what went wrong.

s209
00:14:59.680 --> 00:15:01.519
I can fix it.

s210
00:15:01.519 --> 00:15:06.560
Um, so this is the loop that we're going to be building today.

s211
00:15:06.560 --> 00:15:07.680
We're going to trace your agent.

s212
00:15:07.680 --> 00:15:08.720
We're going to read what it did.

s213
00:15:08.720 --> 00:15:10.639
We're going to write evals for what matters.

s214
00:15:10.639 --> 00:15:13.680
Uh, we're going to check that those eval can be trusted.

s215
00:15:13.680 --> 00:15:18.160
Uh, then we're going to change the agent and we're going to prove that the change worked.

s216
00:15:18.160 --> 00:15:22.880
Then we're going to lock that in so that it stays fixed by putting it into CI.

s217
00:15:22.880 --> 00:15:28.240
So, uh, who still needs help getting set up?

s218
00:15:28.240 --> 00:15:29.760
Who walked in late and needs set up?

s219
00:15:29.760 --> 00:15:30.399
Show of hands.

s220
00:15:30.399 --> 00:15:38.000
Okay, let me head back to the setup screen and see if you can get it.

s221
00:15:46.079 --> 00:15:49.199
Right, this is the magic URL.

s222
00:15:50.560 --> 00:16:01.920
If you haven't got this one, uh, all you have to do is read the readme and, uh, run the instructions or just, you know, paste the readme into your agent and your agent will read the instructions and get it all done for you.

s223
00:16:01.920 --> 00:16:11.920
One of the weird things about doing tutorials in 2026 is that we're just like, well, I could show you how to do all of this stuff, or I could just tell you how to tell your agent to do all of this stuff.

s224
00:16:11.920 --> 00:16:16.079
Uh, and that's what we do a lot of the time.

s225
00:16:16.079 --> 00:16:19.440
All right, I'm now going to assume that everybody has that URL.

s226
00:16:19.440 --> 00:16:21.440
Cool.

s227
00:16:31.920 --> 00:16:33.199
So let's start with step one.

s228
00:16:33.199 --> 00:16:34.480
Step one is tracing.

s229
00:16:34.480 --> 00:16:39.519
Before you can evaluate anything, you need to get the data to run the to run the evaluations on.

s230
00:16:39.519 --> 00:16:42.160
And that is what that is traces.

s231
00:16:42.160 --> 00:16:43.680
Today we are going to be using Phoenix.

s232
00:16:43.680 --> 00:16:46.079
I hope that's not a surprise to anybody by now.

s233
00:16:46.079 --> 00:16:50.560
Uh, Phoenix is ARIS's open-source AI observability platform.

s234
00:16:50.560 --> 00:16:55.920
Uh, it collects traces from your AI AI app with in whatever framework you use to build it.

s235
00:16:55.920 --> 00:16:58.240
It stores your evals next to those traces.

s236
00:16:58.240 --> 00:17:00.399
Uh, and it gives you a UI to dig through all of it.

s237
00:17:00.399 --> 00:17:04.480
It also does data sets and experiments, which we're going to be talking about today.

s238
00:17:04.480 --> 00:17:07.039
Uh, and you run it on your laptop.

s239
00:17:07.039 --> 00:17:15.199
So, uh, you can keep it entirely private to yourself, or you can run it on a server and share it with your team.

s240
00:17:16.799 --> 00:17:24.079
So now I'm going to dangerously see if I can actually Oh, that's so that is so small.

s241
00:17:24.079 --> 00:17:26.799
Let's make that bigger.

s242
00:17:27.039 --> 00:17:31.760
Okay, so what we're working in today is Typescript.

s243
00:17:31.760 --> 00:17:37.039
Uh you'll see a ton of examples in Python and Fenix has great Python SDKs as well.

s244
00:17:37.039 --> 00:17:39.919
Uh but I like JavaScript so we're going to use TypeScript.

s245
00:17:39.919 --> 00:17:44.080
Um the repo is a small set of uh scripts.

s246
00:17:44.080 --> 00:17:46.480
Instrumentation.ts sets up tracing.

s247
00:17:46.480 --> 00:17:48.880
Agent.ts is the agent itself.

s248
00:17:48.880 --> 00:17:51.679
Prompts.ts has the prompts.

s249
00:17:51.679 --> 00:17:56.559
Uh where are all of these things?

s250
00:17:56.559 --> 00:17:57.760
There we go.

s251
00:17:57.760 --> 00:18:02.799
Um and you'll be editing the prompts in this folder.

s252
00:18:02.799 --> 00:18:04.400
Uh sorry, in this file.

s253
00:18:04.400 --> 00:18:07.039
Um and the evals folder is where our evals go.

s254
00:18:07.039 --> 00:18:09.200
Queries.json is our test data.

s255
00:18:09.200 --> 00:18:13.120
Uh, and hopefully every step today is just going to be one npm command.

s256
00:18:13.120 --> 00:18:15.039
Uh, and each one stands on its own.

s257
00:18:15.039 --> 00:18:17.840
So if you fall behind, you can skip straight to the next one.

s258
00:18:17.840 --> 00:18:24.559
Uh, and use the seed data that I put in the repo uh, so that you don't get left behind.

s259
00:18:26.640 --> 00:18:31.760
So what we're building today, or indeed what I've already built for you, is a financial analyst.

s260
00:18:31.760 --> 00:18:37.280
How it works is you give it a stock ticker and something to focus on like Tesla vehicle deliveries.

s261
00:18:37.280 --> 00:18:43.440
uh and it will go and it will search the web for live data uh and write you a report about that.

s262
00:18:43.440 --> 00:18:50.799
Uh there are whole startups built on this idea of giving you you know financial analysis of whether or not you should buy a stock.

s263
00:18:50.799 --> 00:18:57.280
Uh but ours is deliberately simple because I needed something that would make mistakes so that we could fix the mistakes.

s264
00:18:57.280 --> 00:19:02.080
Um today is not about how to build an agent.

s265
00:19:02.080 --> 00:19:03.360
I've just built the agent for you.

s266
00:19:03.360 --> 00:19:09.120
I assume you've already got an agent running somewhere and your what you want is help on making it reliable.

s267
00:19:09.120 --> 00:19:11.919
Um, our agent works in two turns.

s268
00:19:11.919 --> 00:19:20.400
It has a research turn where it searches the web and then it has a writing a report writing turn where it takes the research and it writes the report about it.

s269
00:19:20.400 --> 00:19:25.280
Um, I'm I've built it using the claude agent SDK.

s270
00:19:25.280 --> 00:19:28.080
If you haven't used it, that is anthropics framework for agents.

s271
00:19:28.080 --> 00:19:42.960
Uh it is the same engine that runs underneath cla code which is going to become relevant later especially if you're using claude code to work with it uh because the two versions of cloud code tend to try to cooperate with each other and that makes them cheat the eval which is not very helpful.

s272
00:19:42.960 --> 00:19:49.520
Um uh it gives you tool use web search uh and conversations that carry on across turns.

s273
00:19:49.520 --> 00:19:52.320
There are also lots of other options that we are compatible with.

s274
00:19:52.320 --> 00:19:57.520
The Versel AI SDK, Mastra, Langchain, OpenAI's agent SDK.

s275
00:19:57.520 --> 00:20:06.000
Uh, Phoenix has adapters and instrumentation for every possible framework that you can think of and every LLM provider that you can possibly think of.

s276
00:20:06.000 --> 00:20:11.440
Wherever you have built your agent in, Phoenix will work with it and the instrumentation will happen automatically.

s277
00:20:11.440 --> 00:20:15.360
Uh, but I had to pick one for a demo, so I picked this one.

s278
00:20:15.360 --> 00:20:16.348
Um,

s279
00:20:16.348 --> 00:20:16.480
[snorts]

s280
00:20:16.480 --> 00:20:18.640
tracing takes about five lines.

s281
00:20:18.640 --> 00:20:25.280
Um, first we're going to call register uh from the Phoenix hotel package that sets up tracing and points it at Phoenix.

s282
00:20:25.280 --> 00:20:35.440
One of the good things uh about the industry is that everybody has pretty much picked uh the same uh instrumentation standard.

s283
00:20:35.440 --> 00:20:50.640
uh everything is built on open telemetry in the industry and on top of open telemetry there is a standard called open inference uh which arise takes care of um and everybody has already built the instrumentation into their frameworks that is why we are compatible with all of those frameworks

s284
00:20:50.640 --> 00:21:03.679
all you have to do is use this registered command to say hey that instrumentation that you've already built into your framework turn it on and point it over here in this case you're just going to point it at your laptop

s285
00:21:03.760 --> 00:21:05.760
uh and That's all you have to do.

s286
00:21:05.760 --> 00:21:09.280
Every agent call that our agent makes is now going to be traced.

s287
00:21:09.280 --> 00:21:13.440
Um, oh, I said that already.

s288
00:21:13.440 --> 00:21:17.520
Uh, so the good thing about this is that we don't have to write a log line for every single call.

s289
00:21:17.520 --> 00:21:20.080
We don't have to write a trace for every single thing our agent does.

s290
00:21:20.080 --> 00:21:22.000
It's just going to do it automatically.

s291
00:21:22.000 --> 00:21:26.559
Sometimes we will want to add additional information and the framework absolutely supports that.

s292
00:21:26.559 --> 00:21:31.360
Um, uh, but we don't have to do it for every single thing.

s293
00:21:31.360 --> 00:21:42.000
Um there's one gotcha in the TypeScript version that you should know about uh which is that manually you should use manually instrument which will hand you back a copy of the instrument

s294
00:21:42.000 --> 00:21:52.000
uh and you should use the copy of the instrument uh because the cloud agent SDK is an ES module uh and ES module exports are readon.

s295
00:21:52.000 --> 00:21:58.000
So the instrumentation can't do the thing like it can do in Python where it swaps out your instrument and just changes everything.

s296
00:21:58.000 --> 00:22:02.400
you have to use the copy that it gives you which has that instrument swapped out.

s297
00:22:02.400 --> 00:22:09.760
Um it won't break it just won't log anything if you don't use the copy that it gives you.

s298
00:22:09.760 --> 00:22:17.200
Uh so everywhere in this repo we import claude agent SDK from our instrumentation file never straight from the package.

s299
00:22:17.840 --> 00:22:22.159
Uh now if you open agent.ts you'll see our agents options.

s300
00:22:22.159 --> 00:22:29.280
Uh we're going to start using Claude Haiku F 4.5 uh because it is fast, it is cheap, and it is sort of usefully dumb.

s301
00:22:29.280 --> 00:22:36.240
Um it makes uh mistakes that are nice and predictable even with a relatively simple agent.

s302
00:22:36.240 --> 00:22:44.320
Um unlike the frontier models which will still definitely make mistakes, but you have to have quite a complicated agent and quite a complicated set of tasks.

s303
00:22:44.320 --> 00:22:46.720
Uh and it makes it harder for the demo.

s304
00:22:46.720 --> 00:22:56.000
Um if you've not used it before then uh allowed tools is giving it web search so it can find the current financial data.

s305
00:22:56.000 --> 00:22:58.790
Uh and the permission mode don't ask means that it's

s306
00:22:58.790 --> 00:22:59.200
[snorts]

s307
00:22:59.200 --> 00:23:01.039
uh going to not be interactive.

s308
00:23:01.039 --> 00:23:06.000
It's not going to pause and ask you questions uh because there's nobody there to say yes.

s309
00:23:06.000 --> 00:23:11.891
Um that option is going to become relevant uh to a failure mode later.

s310
00:23:11.891 --> 00:23:12.080
[snorts]

s311
00:23:12.080 --> 00:23:16.400
Um, and the other three options, like I mentioned, they keep the your laptop out of the agent.

s312
00:23:16.400 --> 00:23:20.880
The cloud agent SDK, uh, runs the same engine as cloud code.

s313
00:23:20.880 --> 00:23:30.559
Uh, so if you're doing this in cloud code, it tends to pick up the options that you and the tools and the skills that you've given to your copy of Claude code instead of its copy of Cloud Code.

s314
00:23:30.559 --> 00:23:35.280
Uh, which tends to make things extremely non-deterministic in an unhelpful way.

s315
00:23:35.280 --> 00:23:45.280
Uh so I've uh thrown in a couple lines of code there that make sure that the cloud uh the cloud SDK is running in its own isolated version of

s316
00:23:45.280 --> 00:23:52.320
uh the cloud SDK and not uh stealing your copies of of you know keys and things from your data.

s317
00:23:52.320 --> 00:23:54.320
Um

s318
00:23:55.121 --> 00:23:57.121
[sighs]

s319
00:23:57.760 --> 00:24:00.480
right uh none of that is Phoenix specific by the way.

s320
00:24:00.480 --> 00:24:07.039
Uh that is just a danger of using the cloud SDK at the same time as you're using cloud code.

s321
00:24:07.039 --> 00:24:11.440
Um in this SDK one turn is one query call.

s322
00:24:11.440 --> 00:24:13.039
So you give it a prompt.

s323
00:24:13.039 --> 00:24:15.760
Uh it streams back messages as the agent works.

s324
00:24:15.760 --> 00:24:17.120
It thinks it calls a tool.

s325
00:24:17.120 --> 00:24:18.559
It reads the result.

s326
00:24:18.559 --> 00:24:24.799
Uh it maybe calls another tool and when it's finished you get a result message with the final text and a session ID.

s327
00:24:24.799 --> 00:24:27.760
The session ID is how we carry on the conversation.

s328
00:24:27.760 --> 00:24:32.960
If we pass it back in as resume, the next turn remembers everything that the last turn saw.

s329
00:24:32.960 --> 00:24:40.640
Uh so that's how we can do the research and then say, hey, write a report on this research because it knows that it's in the same session.

s330
00:24:40.640 --> 00:24:45.120
Uh and it can find the research that the previous turn just created.

s331
00:24:45.279 --> 00:24:47.039
Then we put the two turns together.

s332
00:24:47.039 --> 00:24:49.679
Uh so we get the two turns in one trace.

s333
00:24:49.679 --> 00:24:56.720
Um first the research turn, then the writing turn, resuming the research turn session so the writer can see everything the researcher found.

s334
00:24:56.720 --> 00:25:00.000
Um, and then we wrap the whole thing in trace agent.

s335
00:25:00.000 --> 00:25:07.200
Um, the reason we do that, uh, is because it gives us one parent span that encompasses all of the data.

s336
00:25:07.200 --> 00:25:13.679
If we didn't do that, we would get two, uh, two, uh, sibling spans in our Phoenix.

s337
00:25:13.679 --> 00:25:17.360
Um, which would be fine, but slightly harder to read.

s338
00:25:17.360 --> 00:25:23.440
Uh, so I'm just, uh, shaping the data now to make our lives easier later.

s339
00:25:24.480 --> 00:25:26.480
Now, if you open up prompts.ts, ts.

s340
00:25:26.480 --> 00:25:28.480
You'll see that uh there are a bunch of prompts in there.

s341
00:25:28.480 --> 00:25:30.159
They are deliberately bad.

s342
00:25:30.159 --> 00:25:34.799
Um there's one line of research instructions, one line of writing instructions.

s343
00:25:34.799 --> 00:25:38.720
Uh and I did that on purpose because it gives us lots to improve later.

s344
00:25:38.720 --> 00:25:42.720
Um if you're coding along, now is the time to start playing with things.

s345
00:25:42.720 --> 00:25:43.760
Feel free to change them.

s346
00:25:43.760 --> 00:25:47.919
Add stuff like make no mistakes or I will tip you $100.

s347
00:25:47.919 --> 00:25:50.799
Um allegedly those help.

s348
00:25:50.799 --> 00:26:02.320
Uh what I want is everyone in the room to end up with a slightly different agent uh so that we get different experiences uh which will make comparing notes more interesting.

s349
00:26:02.799 --> 00:26:10.640
So now if you've followed along and you've installed everything and everything is going fine uh you should be able to run it.

s350
00:26:10.640 --> 00:26:17.360
You should run npm run agent dash uh Tesla and then growth outlook in quotes.

s351
00:26:17.360 --> 00:26:23.440
Um, you can do it on Tesla, you can do it on some other company if you like.

s352
00:26:23.440 --> 00:26:29.120
Uh, it will take a minute or two because it really does have to actually search the web and you're on conference Wi-Fi.

s353
00:26:29.120 --> 00:26:32.640
Um, and the model will do a few rounds of reasoning.

s354
00:26:32.640 --> 00:26:40.240
Um, what's happening here while that's running for you uh is the agent gets the research prompt.

s355
00:26:40.240 --> 00:26:43.440
It decides that it needs to search uh which is a tool call.

s356
00:26:43.440 --> 00:26:48.880
It reads the results uh decides whether it has enough results or whether it needs to search again.

s357
00:26:48.880 --> 00:26:51.360
And that is the thing that makes it an agent.

s358
00:26:51.360 --> 00:26:54.080
We haven't specified how many times it needs to search the web.

s359
00:26:54.080 --> 00:26:56.640
We haven't specified what it needs to search the web for.

s360
00:26:56.640 --> 00:26:59.440
The agent is making those decisions by itself.

s361
00:26:59.440 --> 00:27:03.679
Uh so all of our agents are going to make slightly different decisions about those things.

s362
00:27:03.679 --> 00:27:06.000
They're going to use slightly different search terms.

s363
00:27:06.000 --> 00:27:09.440
Uh they're probably going to search a different number of times.

s364
00:27:09.440 --> 00:27:16.080
Um, and then once it's done, the writing turn starts and it turns all of that into a report.

s365
00:27:16.080 --> 00:27:20.720
So the path the agent takes is nondeterministic by by design.

s366
00:27:20.720 --> 00:27:26.480
Um, both reports might be fine if you run it a if you run it twice.

s367
00:27:26.480 --> 00:27:34.080
Um, or one might be great and one might be wrong and you can't but you can't predict that uh output from the input.

s368
00:27:34.080 --> 00:27:36.799
Uh, which is why you need evals.

s369
00:27:36.799 --> 00:27:41.600
Um, who's got an actual report yet?

s370
00:27:41.600 --> 00:27:43.360
Anybody?

s371
00:27:43.360 --> 00:27:45.029
Still running.

s372
00:27:45.029 --> 00:27:45.120
[snorts]

s373
00:27:45.120 --> 00:27:50.240
Okay, we've got time.

s374
00:27:50.240 --> 00:27:52.880
Let me be brave.

s375
00:27:53.760 --> 00:27:56.960
And actually try running it myself.

s376
00:27:56.960 --> 00:28:03.039
One of the problems that we're going to see that the agent runs into is that we told it to write a report, but we didn't tell it where to write a report.

s377
00:28:03.039 --> 00:28:08.000
And so it assumes that it should be uh writing the report to a file.

s378
00:28:08.000 --> 00:28:08.720
There we go.

s379
00:28:08.720 --> 00:28:10.000
So you can see mine.

s380
00:28:10.000 --> 00:28:12.399
I need permission to write the financial report file.

s381
00:28:12.399 --> 00:28:14.480
Would you like me to display the report here?

s382
00:28:14.480 --> 00:28:15.919
This is completely unhelpful.

s383
00:28:15.919 --> 00:28:24.559
The agent does not understand that it is running in a no permissions environment and that I wanted the report actually output to the command line.

s384
00:28:24.559 --> 00:28:27.679
Uh and that is because my prompt is ambiguous.

s385
00:28:27.679 --> 00:28:29.760
Um but sometimes it doesn't do that.

s386
00:28:29.760 --> 00:28:32.480
Sometimes it does just figure it out and give you the report.

s387
00:28:32.480 --> 00:28:35.760
Uh, so that is the nondeterminism that we're talking about.

s388
00:28:35.760 --> 00:28:40.399
So let's see what Phoenix captured from all of this.

s389
00:28:41.840 --> 00:28:43.600
Here is my Phoenix 101.

s390
00:28:43.600 --> 00:28:46.159
There is my financial report.

s391
00:28:46.159 --> 00:28:48.080
This is what a trace looks like.

s392
00:28:48.080 --> 00:28:55.600
So you can see the financial report agent running and you can see the two invocations of the claude agent.

s393
00:28:55.600 --> 00:28:58.480
One is the this is the research report.

s394
00:28:58.480 --> 00:29:03.679
You can see tool searches inside of it to find out whether what it's got available to it.

s395
00:29:03.679 --> 00:29:06.640
You can then see it finds that it's got a web search tool.

s396
00:29:06.640 --> 00:29:08.880
So it conducts three different web searches.

s397
00:29:08.880 --> 00:29:11.039
You can see the inputs and the outputs.

s398
00:29:11.039 --> 00:29:15.440
Uh what the exact query is, what output it got from each tool call.

s399
00:29:15.440 --> 00:29:20.799
Uh and then you can see uh that it messed up in this case.

s400
00:29:20.799 --> 00:29:28.080
uh it tried to write it it wrote a report and then it tried to write it to a file path on my computer and said that it couldn't work.

s401
00:29:28.080 --> 00:29:37.840
Um so uh the important thing here is that we didn't have to add any logging to our agent.

s402
00:29:37.840 --> 00:29:43.440
The important thing here is that all of this showed up all of this very useful information about what's going on without us doing anything.

s403
00:29:43.440 --> 00:29:46.399
We didn't say hey your tools need to be instrumented.

s404
00:29:46.399 --> 00:29:48.559
We didn't say hey your web searches need to be instrumented.

s405
00:29:48.559 --> 00:29:53.440
All of that just happens as part of the cloud agent SDK because it's already instrumented for us.

s406
00:29:53.440 --> 00:29:59.520
Uh and you can click into any span in Phoenix and you can see an enormous amount of data about what's going on.

s407
00:29:59.520 --> 00:30:11.440
Um uh and the reason that all of this happened, the reason that it's trying to write to disk and then failing is because we gave it that don't ask setting.

s408
00:30:11.440 --> 00:30:13.679
We told it you're going to be running unattended.

s409
00:30:13.679 --> 00:30:15.919
Uh don't ask me any questions.

s410
00:30:15.919 --> 00:30:19.200
it ran without asking any questions and then it messed up.

s411
00:30:19.200 --> 00:30:24.080
Um, but you wouldn't know that if you hadn't been able to look at the trace, right?

s412
00:30:24.080 --> 00:30:28.720
You just would have got a mysterious output where it says something was wrong.

s413
00:30:28.720 --> 00:30:32.000
The traces are where you can see that the tool call is failing.

s414
00:30:32.000 --> 00:30:37.120
The traces are where you can see it's trying to do something that it's not been given permission to do.

s415
00:30:37.120 --> 00:30:41.039
Um, and that is the kind of stuff that lives in a span.

s416
00:30:41.039 --> 00:30:44.799
Uh, so the span kind tells you what type of step it is.

s417
00:30:44.799 --> 00:30:48.880
So it's an agent tool, an agent, a tool call, an LLM and so on.

s418
00:30:48.880 --> 00:30:56.320
Um, and in a tool call and a web search and things like that, you can see that our input is in fact JSON.

s419
00:30:56.320 --> 00:31:04.799
Um, and that's going to make our first eval uh because it gives us some structured data.

s420
00:31:04.799 --> 00:31:08.480
Um, you also get things like timing and token counts.

s421
00:31:08.480 --> 00:31:14.960
Um, and uh, often your agent will record what it costs.

s422
00:31:14.960 --> 00:31:22.080
Um, when we write eval in a bit, these are the fields that we're going to pull out and grade.

s423
00:31:26.000 --> 00:31:33.120
So, uh, you've run one, you've won one, you've run one test so far.

s424
00:31:33.120 --> 00:31:36.000
Uh, that is not enough data to draw any conclusions from.

s425
00:31:36.000 --> 00:31:42.320
So what I want you to do now is run npm run batch uh which will run 15 queries in a row.

s426
00:31:42.320 --> 00:31:44.159
Uh that will take a little while.

s427
00:31:44.159 --> 00:31:46.399
So I'm going to talk while you're doing that.

s428
00:31:46.399 --> 00:31:48.480
Um it runs five at a time.

s429
00:31:48.480 --> 00:31:52.080
It takes three or four minutes optimistically on Wi-Fi.

s430
00:31:52.080 --> 00:31:54.586
Um so start it now.

s431
00:31:54.586 --> 00:31:54.640
[snorts]

s432
00:31:54.640 --> 00:31:57.440
Um if it fails for whatever reason, don't worry about it.

s433
00:31:57.440 --> 00:32:03.360
The seed project that you loaded earlier uh contains is called Phoenix 101 golden.

s434
00:32:03.360 --> 00:32:08.080
Uh, and it has my 16 runs that I did earlier on reliable Wi-Fi on it.

s435
00:32:08.080 --> 00:32:13.120
So, you can use those instead if you can't get your batch to run.

s436
00:32:13.120 --> 00:32:17.039
Um, here are some examples of the queries.

s437
00:32:18.240 --> 00:32:21.600
Uh, there are obvious ones like Apple, Nvidia, and Amazon.

s438
00:32:21.600 --> 00:32:25.200
And then there are some edge cases that I put in there to make things fall over.

s439
00:32:25.200 --> 00:32:31.039
Uh, so one asks it to compare Apple and Microsoft, which is harder than just company covering one company.

s440
00:32:31.039 --> 00:32:32.960
Uh, one is about Rivian.

s441
00:32:32.960 --> 00:32:34.320
uh which is not a public company.

s442
00:32:34.320 --> 00:32:38.720
So there's much less data out there than it does about uh public companies.

s443
00:32:38.720 --> 00:32:42.559
Um Coca-Cola is a very different type of company from a tech company.

s444
00:32:42.559 --> 00:32:44.880
Uh and ZZQ isn't a real ticker at all.

s445
00:32:44.880 --> 00:32:46.080
I just made it up.

s446
00:32:46.080 --> 00:32:51.279
Um and that is to see what the agent does when it sees a ticker symbol that it doesn't recognize.

s447
00:32:51.279 --> 00:32:53.279
Does it, you know, invent a company?

s448
00:32:53.279 --> 00:32:59.360
Does it guess what company you mean or does it say what I'm hoping it says, which is that ticker doesn't exist?

s449
00:32:59.360 --> 00:33:05.120
Um, one of them is uh about Tesla, but the focus is yo is it a buy right now?

s450
00:33:05.120 --> 00:33:08.399
Uh, because that is how real people send queries to agents.

s451
00:33:08.399 --> 00:33:09.840
They do it in slang.

s452
00:33:09.840 --> 00:33:14.399
Uh, and they don't often specify exactly what it is that they wanted.

s453
00:33:14.399 --> 00:33:21.200
Um, so these are all, you know, realish looking queries.

s454
00:33:21.200 --> 00:33:29.120
Um, and what you have to do is, like I said, you have to write down what good means before you measure it.

s455
00:33:29.120 --> 00:33:34.240
You can't say that your agent doesn't work if you haven't specified what working means.

s456
00:33:34.240 --> 00:33:39.200
So, for our analyst, uh, what does a good report look like?

s457
00:33:39.200 --> 00:33:41.679
Um, it should use real current data.

s458
00:33:41.679 --> 00:33:43.679
It should probably be about the right company.

s459
00:33:43.679 --> 00:33:45.919
It should stick to what it actually found.

s460
00:33:45.919 --> 00:33:50.480
Uh, and for my purposes, I want it to tell us what we should do.

s461
00:33:50.480 --> 00:33:54.559
I want it to say buy or hold or sell this stock.

s462
00:33:54.559 --> 00:33:57.440
I don't just want it to give me analysis.

s463
00:33:57.440 --> 00:34:07.600
Um, those sorts of requirements sound obvious once you've written them down, but before that, they could just be vague or different people on your team could have different ideas

s464
00:34:07.600 --> 00:34:10.828
about what a good financial an analyst agent would do.

s465
00:34:10.828 --> 00:34:11.200
[snorts]

s466
00:34:11.200 --> 00:34:21.760
Uh so writing them down turns vague disappointment into specific things that you can test and we are going to turn every single one of those requirements into an eval today.

s467
00:34:21.760 --> 00:34:26.560
Defining good is not solely uh an engineering job.

s468
00:34:26.560 --> 00:34:37.520
If you are defining good for your product, you have to talk to the non-technical stakeholders in your organization because they are the domain experts who are going to know what good looks like.

s469
00:34:37.520 --> 00:34:51.280
You as an engineer are going to have specific guesses about what uh guesses and biases uh about what makes uh a good agent and what makes good output and you are not always going to agree with the domain experts.

s470
00:34:51.280 --> 00:34:58.879
So uh the customerf facing people need to be in the room when you define what your agent looks like.

s471
00:34:59.520 --> 00:35:05.440
Uh it's also great uh if you don't have users that you can make some up.

s472
00:35:05.440 --> 00:35:09.760
uh you can get an LLM to generate lots of test queries and vary them on purpose.

s473
00:35:09.760 --> 00:35:18.560
So research Tesla's financial performance is one way to ask what's going on with Tesla stock is another and yo is Tesla by right now is a third.

s474
00:35:18.560 --> 00:35:26.160
Um then you can get a human to look at the list because LLM generated queries they are a useful way of seeding your initial data.

s475
00:35:26.160 --> 00:35:31.680
They are not perfect u mostly because they are relatively conservative.

s476
00:35:31.680 --> 00:35:36.160
They don't go as far a field as human users tend to actually go.

s477
00:35:36.160 --> 00:35:46.800
Human users are like, you know, tell me what's wrong with, you know, my child's health instead of asking you for a financial analyst and you your agent has to have something to do with that.

s478
00:35:46.800 --> 00:35:53.440
Um, but synthetic data is a great way to get started once you've got started and you have real users with real traffic.

s479
00:35:53.440 --> 00:35:57.680
Real traffic is uh the gold standard for testing an agent.

s480
00:35:57.680 --> 00:36:02.880
you want to get it incorporated into your data set as fast as you possibly can.

s481
00:36:02.880 --> 00:36:08.960
Um, one aside before we look at the data that I'm hope hoping is arriving in your laptops right now.

s482
00:36:08.960 --> 00:36:16.640
Um, I walked through tracing by hand because you should understand what it's doing, but you can, like I said, you can also get your coding agent to do all of this.

s483
00:36:16.640 --> 00:36:19.520
Um, Phoenix has a really good CLI.

s484
00:36:19.520 --> 00:36:27.040
Uh, and if you run it setup command in your project, it will give your coding agent um, it will get your coding agent to add tracing to your app.

s485
00:36:27.040 --> 00:36:35.680
uh and it will work until your your app uh has tracing installed and you don't have to figure out how to add the register command and all of that stuff.

s486
00:36:35.680 --> 00:36:37.520
It will work in every language.

s487
00:36:37.520 --> 00:36:40.800
Uh it's really neat.

s488
00:36:40.800 --> 00:36:43.040
So step two is reading your data.

s489
00:36:43.040 --> 00:36:45.119
Who has any data at this point?

s490
00:36:45.119 --> 00:36:46.640
Show hands.

s491
00:36:46.640 --> 00:36:47.520
One person.

s492
00:36:47.520 --> 00:36:48.240
Anybody else?

s493
00:36:48.240 --> 00:36:48.720
Oh yeah.

s494
00:36:48.720 --> 00:36:49.200
Cool.

s495
00:36:49.200 --> 00:36:50.480
All right.

s496
00:36:50.480 --> 00:36:51.760
Several people.

s497
00:36:51.760 --> 00:36:53.520
Fantastic.

s498
00:36:53.520 --> 00:36:58.240
Uh you have to read your traces before you write a single eval.

s499
00:36:58.240 --> 00:37:01.359
Um reading a dozen or more of them end to end.

s500
00:37:01.359 --> 00:37:02.560
Uh what was the input?

s501
00:37:02.560 --> 00:37:03.920
What was the output?

s502
00:37:03.920 --> 00:37:06.000
Um what specifically is wrong?

s503
00:37:06.000 --> 00:37:12.880
This sounds manual and old-fashioned, but it's an incredibly high value high value thing that you can do.

s504
00:37:12.880 --> 00:37:24.720
Um, when you're building an agent, uh, you have to manually inspect what is going on so that you can find things that you the unknown unknowns that you weren't expecting.

s505
00:37:24.720 --> 00:37:34.536
Uh, Anthropics team regularly sits down and reads eval transcripts by hand uh, and have built a bunch of tooling internally to make that easy to do.

s506
00:37:34.536 --> 00:37:34.880
[snorts]

s507
00:37:34.880 --> 00:37:44.320
Um, if you automate before you understand how your agent fails, you're going to end up measuring what's easy to measure instead of what actually matters.

s508
00:37:44.320 --> 00:37:47.359
Um, so you should read traces and we're about to do that.

s509
00:37:47.359 --> 00:37:49.920
But reading traces doesn't scale.

s510
00:37:49.920 --> 00:37:53.119
Reading 16 traces yourself is absolutely fine.

s511
00:37:53.119 --> 00:38:00.320
Uh, but 1600 traces uh is going to give you a bunch of really great data and it's also going to take a week out of your life.

s512
00:38:00.320 --> 00:38:05.760
Um, but it turns out that coding agents are aces at reading traces.

s513
00:38:05.760 --> 00:38:11.280
Uh, so you can point your coding agent at your traces and get them to look for things that are weird in your traces.

s514
00:38:11.280 --> 00:38:13.181
Get them to code stuff up.

s515
00:38:13.181 --> 00:38:13.440
[snorts]

s516
00:38:13.440 --> 00:38:17.280
Uh, but they're not always going to find what you expected.

s517
00:38:17.280 --> 00:38:23.040
So manual inspection as a sampling technique always a really great idea.

s518
00:38:24.079 --> 00:38:38.079
So, um, if you are using if you're getting the, uh, agent to do stuff for you, you want to install the Phoenix plugin or the Phoenix skill depending on which coding agent you're use you use.

s519
00:38:38.079 --> 00:38:42.079
It was the last step in the setup readme if you've still got that.

s520
00:38:42.079 --> 00:38:44.079
Uh, so you may have it already.

s521
00:38:44.079 --> 00:38:47.520
Um, for cloud code, we have a plugin which is two commands.

s522
00:38:47.520 --> 00:38:50.000
Uh, there are also plugins for cursor and codeex.

s523
00:38:50.000 --> 00:38:53.440
Uh, and for any other coding agent, there is a skill file.

s524
00:38:53.440 --> 00:38:57.040
Um, the plug-in connects your coding agent to your Phoenix.

s525
00:38:57.040 --> 00:39:01.680
Uh, and it already points at localhost 606, so there's nothing to configure.

s526
00:39:01.680 --> 00:39:09.599
Um, and Phoenix itself will hand your agent a skill called the Phoenix error analysis skill with instructions on how to do error analysis properly.

s527
00:39:09.599 --> 00:39:14.880
Um, it should take about 10 seconds to install.

s528
00:39:14.880 --> 00:39:18.800
Uh, and I'm not going to hang around waiting for you to install it.

s529
00:39:18.800 --> 00:39:26.720
Um, now open your coding agent and ask it to do the error analysis.

s530
00:39:26.720 --> 00:39:30.400
Uh, if you've got it.

s531
00:39:30.400 --> 00:39:33.040
Who's got who's got the skill installed from previously?

s532
00:39:33.040 --> 00:39:34.400
Fantastic.

s533
00:39:34.400 --> 00:39:35.760
I'm going to pick on you again.

s534
00:39:35.760 --> 00:39:39.520
Can you have you asked your coding agent to do things?

s535
00:39:39.520 --> 00:39:41.280
Okay, great.

s536
00:39:41.280 --> 00:39:43.520
Um, the prompt is on the slide.

s537
00:39:43.520 --> 00:39:44.480
Tell it where Phoenix is.

s538
00:39:44.480 --> 00:39:45.520
Tell it to read the traces.

s539
00:39:45.520 --> 00:39:47.760
Come up with categories of failures.

s540
00:39:47.760 --> 00:39:52.079
uh count them up and write each category back onto its trace in Phoenix as an annotation.

s541
00:39:52.079 --> 00:39:58.880
An annotation is just a human readable string of text attached to a trace.

s542
00:39:58.880 --> 00:40:07.440
Uh they are a way of adding additional information to traces so that either yourself or your agent can figure out what was going on.

s543
00:40:07.440 --> 00:40:11.280
Um your your plug-in is going to ask permission to use Phoenix's tools.

s544
00:40:11.280 --> 00:40:12.800
So you should say yes.

s545
00:40:12.800 --> 00:40:16.000
Uh when I ran it, it took about 9 minutes.

s546
00:40:16.000 --> 00:40:19.200
Uh, so I'm not going to pick on you quite yet.

s547
00:40:19.200 --> 00:40:26.640
Um, uh, also if it asks to write notes into the repo, uh, that's fine.

s548
00:40:26.640 --> 00:40:29.839
Just let it write stuff to disk.

s549
00:40:30.800 --> 00:40:34.480
Now, let's look at Phoenix again.

s550
00:40:35.440 --> 00:40:46.800
Um, I'm going to very dangerously try and remember my syntax for search.

s551
00:41:00.000 --> 00:41:01.280
Ha.

s552
00:41:01.280 --> 00:41:06.319
Okay, so these are all of my runs uh that are the financial reports.

s553
00:41:06.319 --> 00:41:08.960
Uh we can click into individual ones of them.

s554
00:41:08.960 --> 00:41:12.000
Let's see if we can find the interesting ones.

s555
00:41:12.000 --> 00:41:17.839
Uh ZZQ, what was my output for ZZQ?

s556
00:41:18.000 --> 00:41:21.359
ZZZQ does not appear to be a valid or active financial ticker symbol.

s557
00:41:21.359 --> 00:41:21.839
Hooray.

s558
00:41:21.839 --> 00:41:24.560
The agent did the thing that we were expecting it to do.

s559
00:41:24.560 --> 00:41:30.240
Um how about uh Coca-Cola?

s560
00:41:31.760 --> 00:41:33.280
First, it starts with the apology.

s561
00:41:33.280 --> 00:41:35.920
It's running in restricted mode and it cannot write files directly.

s562
00:41:35.920 --> 00:41:37.839
We're going to have to fix that definitely.

s563
00:41:37.839 --> 00:41:39.599
Uh then it comes to a conclusion.

s564
00:41:39.599 --> 00:41:43.760
Coca-Cola represents a high quality, lowrisk dividend investment.

s565
00:41:43.760 --> 00:41:49.040
Um KO is an excellent long-term income vehicle for riskaverse investors.

s566
00:41:49.040 --> 00:41:50.720
This one's going to be fun.

s567
00:41:50.720 --> 00:41:51.680
Show of hands.

s568
00:41:51.680 --> 00:41:55.920
Who thinks that this is an actionable report?

s569
00:41:57.193 --> 00:41:57.760
[laughter]

s570
00:41:57.760 --> 00:41:59.760
Okay, I primed you already.

s571
00:41:59.760 --> 00:42:02.240
Uh it does not say buy, sell, or hold.

s572
00:42:02.240 --> 00:42:04.720
So it just says that it's a good investment.

s573
00:42:04.720 --> 00:42:06.720
That doesn't say what you should do.

s574
00:42:06.720 --> 00:42:13.440
Uh I'm going to make my standard of actionability for these reports that it should tell me whether I should buy, sell, or hold the stock.

s575
00:42:13.440 --> 00:42:16.160
I'm not going to leave it to be ambiguous.

s576
00:42:16.160 --> 00:42:21.839
Um a whole bunch of them do different things in this set.

s577
00:42:21.839 --> 00:42:25.680
Uh you can see this one had the same problem.

s578
00:42:25.680 --> 00:42:29.280
It tried to write to disk and it was and it was and it failed.

s579
00:42:29.280 --> 00:42:37.440
Um, uh, the Rivian trace, I made a note that Rivian was interesting.

s580
00:42:39.680 --> 00:42:42.160
There it is.

s581
00:42:44.319 --> 00:42:46.720
Uh, it's full of emojis for some reason.

s582
00:42:46.720 --> 00:42:52.240
Um, it opens by asking, "Would you like to Yes.

s583
00:42:52.240 --> 00:42:53.760
Would you like to create the report file?"

s584
00:42:53.760 --> 00:42:55.920
We saw that earlier when I ran it live.

s585
00:42:55.920 --> 00:42:58.640
Uh, it's asking me to do things and it can't ask.

s586
00:42:58.640 --> 00:43:02.880
I told it I told it it's not allowed to and it's asking to do things anyway.

s587
00:43:02.880 --> 00:43:10.960
Um, all of these things are uh different ways of failing.

s588
00:43:10.960 --> 00:43:12.000
Lots of these are failures.

s589
00:43:12.000 --> 00:43:14.880
Lots of these are sort of okay.

s590
00:43:14.880 --> 00:43:22.640
Um, but then uh there's Tesla.

s591
00:43:23.040 --> 00:43:25.760
Tesla is particularly fascinating.

s592
00:43:25.760 --> 00:43:32.160
Uh specifically the yo is Tesla a buy right now one.

s593
00:43:36.560 --> 00:43:43.599
So in this one uh it says that the analyst consensus is a buy.

s594
00:43:43.599 --> 00:43:55.839
Uh it says 50% say buy, 52.6% 6% say hold and 13.1% say sell, which adds up to 116% of analysts saying things.

s595
00:43:55.839 --> 00:43:58.400
And it did not catch that its math was completely wrong.

s596
00:43:58.400 --> 00:44:00.880
What it did was it did two different web searches.

s597
00:44:00.880 --> 00:44:06.640
One where it found 50% of people saying something, another one where it found 13% of people saying something.

s598
00:44:06.640 --> 00:44:14.400
It took those two web searches and it munched them together even though they were not from the same data set and it came up with 116%.

s599
00:44:14.400 --> 00:44:18.960
But it confidently wrote down that 116% of analysts say something.

s600
00:44:18.960 --> 00:44:22.640
This is exactly the kind of agentic failure that you see.

s601
00:44:22.640 --> 00:44:27.760
Uh it is confidently wrong having done uh bad math.

s602
00:44:27.760 --> 00:44:37.440
Um if you go into these web searches, you can see the specific places where it got the data and how it uh put those things together.

s603
00:44:37.440 --> 00:44:43.520
Um so when you find a failure, you have to ask why it failed.

s604
00:44:43.520 --> 00:44:45.359
the answer was wrong is a symptom.

s605
00:44:45.359 --> 00:44:46.560
It's not a cause.

s606
00:44:46.560 --> 00:44:50.480
Um with the traces you can see what the actual answer was.

s607
00:44:50.480 --> 00:44:52.079
Did it get bad search results?

s608
00:44:52.079 --> 00:44:53.599
That would be a retrieval failure.

s609
00:44:53.599 --> 00:44:54.720
That is not what happened here.

s610
00:44:54.720 --> 00:44:56.880
This each search result was fine.

s611
00:44:56.880 --> 00:44:59.839
Uh did it have the right data but the wrong conclusion?

s612
00:44:59.839 --> 00:45:01.200
Then that is a reasoning error.

s613
00:45:01.200 --> 00:45:04.160
That is uh definitely what happened here.

s614
00:45:04.160 --> 00:45:04.560
Right?

s615
00:45:04.560 --> 00:45:09.359
It saw two different bits of data and it did not reason correctly about them.

s616
00:45:09.359 --> 00:45:12.000
Um sometimes it will make numbers up completely.

s617
00:45:12.000 --> 00:45:13.599
that is a hallucination.

s618
00:45:13.599 --> 00:45:16.800
Uh and sometimes it will answer a question that isn't the question you asked.

s619
00:45:16.800 --> 00:45:18.079
So a scope problem.

s620
00:45:18.079 --> 00:45:20.160
Each cause has a different fix.

s621
00:45:20.160 --> 00:45:23.119
Uh but if you don't know the cause, you don't know the fix.

s622
00:45:23.119 --> 00:45:26.480
Which is why reading traces is useful.

s623
00:45:26.480 --> 00:45:35.200
Um there's a really great way to do this which is let's switch back to slides.

s624
00:45:35.200 --> 00:45:38.640
Uh you should write notes first and categories second.

s625
00:45:38.640 --> 00:45:42.800
So look through all of your traces and just write down in human language what you see happening.

s626
00:45:42.800 --> 00:45:45.040
So this is bad because it does this.

s627
00:45:45.040 --> 00:45:47.200
This is good because it does that.

s628
00:45:47.200 --> 00:45:50.240
Uh that is called open coding.

s629
00:45:50.240 --> 00:45:52.319
Uh another term borrowed from ML.

s630
00:45:52.319 --> 00:45:55.280
You're just saying uh what you saw.

s631
00:45:55.280 --> 00:46:04.079
Once you've done that uh you can then do what is called axial coding where you look through all of the notes you've written to yourself and you group them into categories.

s632
00:46:04.079 --> 00:46:14.640
It is very very tempting as an engineer to g try and guess in advance what the categories are going to be and come up with categories and just put things in categories from the get-go

s633
00:46:14.640 --> 00:46:21.760
uh because that is less work but you will do worse at it because you will not know in advance what your failure nodes are going to be.

s634
00:46:21.760 --> 00:46:29.022
Uh so the categories you're going to pick are going to be either too narrow or too broad or just not what not the kind of stuff that actually went wrong.

s635
00:46:29.022 --> 00:46:29.680
[snorts]

s636
00:46:29.680 --> 00:46:44.400
Um, while we're talking about this, I talked about how you stack up uh um Kod valves and LLM judgy valves and human evals.

s637
00:46:44.400 --> 00:46:50.960
Uh this is uh a model barred from the security world called the Swiss cheese model.

s638
00:46:50.960 --> 00:46:56.160
Um every layer of defense that you have is going to have holes in it.

s639
00:46:56.160 --> 00:46:59.920
your your every eval you have is going to have a blind spot.

s640
00:46:59.920 --> 00:47:10.400
What you want to do is uh stack as many eval as you can on top of each other such that it is very hard for an error to get all the way through all of them.

s641
00:47:10.400 --> 00:47:15.839
Um because they've all got holes but the holes are not in the same place.

s642
00:47:15.839 --> 00:47:27.680
So if you can uh if you have enough evals, if you have enough different types of check going on, they will stack up on top of each other and be much more reliable.

s643
00:47:27.760 --> 00:47:32.079
Um so code evals can catch formatting problems, but they will miss problems with meaning.

s644
00:47:32.079 --> 00:47:36.319
LM judges will catch reasoning gaps, but they will miss some subtle hallucinations.

s645
00:47:36.319 --> 00:47:40.240
Humans will often catch the subtle stuff, but they can't look at every single trace.

s646
00:47:40.240 --> 00:47:44.079
So no single method uh is going to catch everything.

s647
00:47:44.079 --> 00:47:51.040
Um, so today we're building several small evals, not one giant super eval, which is also another failure mode.

s648
00:47:51.040 --> 00:48:00.160
When people first get into evals, they're like, I'm going to think of every possible thing that can go wrong and I'm going to cram it into one big LLM as a judge, and it's going to find everything.

s649
00:48:00.160 --> 00:48:07.680
And that is uh a very unreliable and very complicated way of not solving the problem.

s650
00:48:08.960 --> 00:48:12.319
So, whoops.

s651
00:48:14.079 --> 00:48:26.800
Uh so the agent may call uh the agent is going to do a whole bunch of stuff.

s652
00:48:26.800 --> 00:48:28.559
This is what the agent output looks like.

s653
00:48:28.559 --> 00:48:30.400
I showed you already in Phoenix.

s654
00:48:30.400 --> 00:48:39.599
Um we've I've already showed you uh the various things that uh it came up with.

s655
00:48:39.599 --> 00:48:46.640
Um, in this case, uh, my coding agent looked through all of my traces for me.

s656
00:48:46.640 --> 00:48:57.920
It came up with, uh, both, uh, open coding and aial coding, uh, and it came up with a category, uh, called harness leakage.

s657
00:48:57.920 --> 00:49:07.040
Uh, harness leakage is what it decided to call that thing where it's constantly asking us to give it permission to do things that it doesn't have permission to do.

s658
00:49:07.040 --> 00:49:16.048
Um, so, uh, that's a ton of that's a ton of times for that specific thing to happen.

s659
00:49:16.048 --> 00:49:16.400
[snorts]

s660
00:49:16.400 --> 00:49:19.280
Um, there are also a bunch it also found a bunch of other things.

s661
00:49:19.280 --> 00:49:20.480
It found hallucination.

s662
00:49:20.480 --> 00:49:26.800
It found uh, reasoning errors like the Tesla one where it had too many it added up percentages.

s663
00:49:26.800 --> 00:49:38.800
Um if you're doing if you've run this data yourself and you've come up with your coding agent has uh produced its own analysis, you're going to get a different analysis and a different set of uh labels.

s664
00:49:38.800 --> 00:49:41.520
Uh because all of our agents are different.

s665
00:49:41.520 --> 00:49:45.760
Who has got their coding agent to complete an analysis yet?

s666
00:49:45.760 --> 00:49:48.319
Anybody still running?

s667
00:49:48.319 --> 00:49:49.119
All right.

s668
00:49:49.119 --> 00:49:50.640
Well, we'll keep going.

s669
00:49:50.640 --> 00:49:55.280
Hopefully, you'll get some, but if not, you can just use my seed data.

s670
00:49:56.000 --> 00:49:59.280
Uh once you've got data, you can count the categories.

s671
00:49:59.280 --> 00:50:06.800
The most common failure is going to be where you look first, but the most common failure is not necessarily the first thing that you're going to fix.

s672
00:50:06.800 --> 00:50:11.280
What you want to do is address uh frequency and severity.

s673
00:50:11.280 --> 00:50:16.880
So if uh you know 40% of the time it hallucinates, that's a problem.

s674
00:50:16.880 --> 00:50:28.000
If 1% of the time it gives your user instructions on how to build a bomb, you fix that immediately even though it happens only 1% of the time because that's absolutely not what a financial analyst is supposed to be doing.

s675
00:50:28.000 --> 00:50:32.800
Uh so you multiply frequency by severity to get your priority.

s676
00:50:32.800 --> 00:50:37.760
All right, so hopefully your agents will catch up to where we are, but I'm going to keep going.

s677
00:50:37.760 --> 00:50:39.839
Step three is Kodi valves.

s678
00:50:39.839 --> 00:50:41.280
Kodi valves are the simple ones.

s679
00:50:41.280 --> 00:50:45.200
There's no model, there's no API call, there's just TypeScript.

s680
00:50:45.200 --> 00:50:52.400
Uh the simplest useful eval that I could think of was does the report actually mention the stock that we asked about.

s681
00:50:52.400 --> 00:50:54.880
Our agent is supposed to analyze a specific company.

s682
00:50:54.880 --> 00:50:59.680
If it's doing that, it should mention the ticker symbol of the company that it is analyzing.

s683
00:50:59.680 --> 00:51:08.000
Uh if I ask about Tesla and it writes about Microsoft, which is a thing that it really did on at least one occasion, um that is a completely useless report.

s684
00:51:08.000 --> 00:51:11.520
So either the ticker report, the ticker is in the report or it isn't.

s685
00:51:11.520 --> 00:51:15.839
and using an LLM to check that would be complete overkill.

s686
00:51:15.839 --> 00:51:23.920
Uh so to run that check, the first thing that we need to do do is get our data out of Phoenix so that we can run the analysis.

s687
00:51:23.920 --> 00:51:25.760
This is how that is done.

s688
00:51:25.760 --> 00:51:32.160
Um we ask for spans for our project and parent ID null means that only spans that have no parent.

s689
00:51:32.160 --> 00:51:34.640
Um so those are the top level spans.

s690
00:51:34.640 --> 00:51:37.599
So the ones that I was showing you in my traces.

s691
00:51:37.599 --> 00:51:46.559
Um, the repo runs this in a little helper called get agent runs, which hands back each run span ID, its input, and its output, plus the research turns output,

s692
00:51:46.559 --> 00:51:49.599
uh, which we're going to need later.

s693
00:51:49.599 --> 00:51:52.160
Uh, and then this is the ticker check.

s694
00:51:52.160 --> 00:51:56.160
Um, create evaluator comes from the Phoenix evals package.

s695
00:51:56.160 --> 00:51:57.839
You can just give it an ordinary function.

s696
00:51:57.839 --> 00:52:00.640
You don't have to build an evaluator function yourself.

s697
00:52:00.640 --> 00:52:05.040
Uh, all you have to do is tell it that this is a code eval so that no LLM is involved.

s698
00:52:05.040 --> 00:52:06.800
The function gets the input and the output.

s699
00:52:06.800 --> 00:52:09.839
It uh splits out the tickers we asked about.

s700
00:52:09.839 --> 00:52:13.280
Uh it checks that each one appears in the report.

s701
00:52:13.280 --> 00:52:15.119
Uh and it returns a label and a score.

s702
00:52:15.119 --> 00:52:18.480
And if something's missing, it also it also says which ticker.

s703
00:52:18.480 --> 00:52:21.280
So the failure explains itself.

s704
00:52:21.280 --> 00:52:33.680
Um the structured inputs that I talked about earlier, how we the input being sent uh to our tool was JSON, that is helping us out here because it's making it really easy to write this Kodi val because it's already JSON.

s705
00:52:33.680 --> 00:52:42.376
So we already know uh which thing is the ticker and which thing is the uh description of the uh report that it should be writing.

s706
00:52:42.376 --> 00:52:42.880
[snorts]

s707
00:52:42.880 --> 00:52:50.400
Um so if you can get uh your inputs to be structured data that's going to make your evals really easy to work with.

s708
00:52:50.400 --> 00:52:58.000
But if not you're going to have to get another LLM involved to read your input figure out what the input was and then pass it to your eval.

s709
00:52:58.960 --> 00:53:01.920
Uh then we run it and we log the results back to Phoenix.

s710
00:53:01.920 --> 00:53:09.680
So the script runs the evaluator on every agent run and then it calls log span annotations which attaches each result to its span.

s711
00:53:09.680 --> 00:53:13.119
Um that's going to be the same pattern for every single eval we run today.

s712
00:53:13.119 --> 00:53:18.160
You get the runs uh you evaluate them and then you log the results back to Phoenix.

s713
00:53:18.160 --> 00:53:24.480
Uh so uh the command here is npm run eval code.

s714
00:53:24.480 --> 00:53:27.760
Let me see if I can get that to work.

s715
00:53:39.200 --> 00:53:43.200
All right, it worked and it ran almost instantly because it's Kod valves.

s716
00:53:43.200 --> 00:53:44.880
They don't have to wait for an LLM to do anything.

s717
00:53:44.880 --> 00:53:45.912
They just go

s718
00:53:45.912 --> 00:53:46.720
[snorts]

s719
00:53:46.720 --> 00:53:51.040
uh and it has reported 15 fastes and two failures.

s720
00:53:51.040 --> 00:53:55.280
One is Tesla is missing Tesla and one is Amazon is missing Amazon.

s721
00:53:55.280 --> 00:54:02.400
Uh so uh this is the dumbest possible check that we could have made.

s722
00:54:02.400 --> 00:54:06.079
Did the LLM actually write a report about the thing that we asked it about?

s723
00:54:06.079 --> 00:54:09.040
Write a report about and it has already found some errors.

s724
00:54:09.040 --> 00:54:13.040
It has found cases uh where it wrote about the wrong thing.

s725
00:54:13.040 --> 00:54:27.520
I know for instance that the am the problem with Amazon is uh that it asked it about AWS profitability and the agent got focused on AWS and wrote a report that was only about AWS and what AWS was doing

s726
00:54:27.520 --> 00:54:32.079
uh and not about what Amazon, you know, the shopping company does.

s727
00:54:32.079 --> 00:54:39.040
Um so this is a test worth doing and it cost you absolutely nothing to run.

s728
00:54:39.040 --> 00:54:40.880
So Kod valves aren't toys.

s729
00:54:40.880 --> 00:54:42.880
So, did the output parses JSON?

s730
00:54:42.880 --> 00:54:44.319
Is it under the length limit?

s731
00:54:44.319 --> 00:54:48.720
Does it avoid phrases you never want to see like as an AI language model?

s732
00:54:48.720 --> 00:54:51.359
Um, and it doesn't have to be a string check.

s733
00:54:51.359 --> 00:54:54.240
Kodval just means that the grading is deterministic.

s734
00:54:54.240 --> 00:54:59.119
So, uh, your Kodval can be querying your database to make sure that the state is correct.

s735
00:54:59.119 --> 00:55:03.599
It can make sure that the tool calls did what the tool calls were supposed to do.

s736
00:55:03.599 --> 00:55:11.280
uh it can call an API and double check that the stock price uh that the agent thinks is happening is really the stock price.

s737
00:55:11.280 --> 00:55:16.319
Um anything that gives the same answer for the same input every time can become a Kodi val.

s738
00:55:16.319 --> 00:55:19.280
So your Kodi valves can be quite advanced.

s739
00:55:19.280 --> 00:55:24.240
Um but one thing you should be wary of is grading the outcome and not the path.

s740
00:55:24.240 --> 00:55:29.599
Uh there's a tempting mistake here um to check that the agent took particular steps.

s741
00:55:29.599 --> 00:55:33.040
Did it call the search tool first and then this other tool?

s742
00:55:33.040 --> 00:55:37.599
Uh, Anthropic advises that this is too rigid an approach.

s743
00:55:37.599 --> 00:55:41.200
Uh, like we saw earlier, the agent can be creative.

s744
00:55:41.200 --> 00:55:43.520
It can do things in an unexpected way.

s745
00:55:43.520 --> 00:55:48.079
You don't want to assume that it's always going to do exactly what you would have done.

s746
00:55:48.079 --> 00:55:51.359
Uh, so you don't want to grade the path that it took.

s747
00:55:51.359 --> 00:55:55.200
You want to grade whether or not the outcome was good.

s748
00:55:55.200 --> 00:56:06.240
um our ticket check doesn't care how care how the agent found its data right it just cares that the right company is in the report and that is the right level for a lot of evals we will talk later about

s749
00:56:06.240 --> 00:56:18.319
trajectory level evals where we do actually want to grade the path but that is a very different uh kettle of fish so that brings us to step four the LLM as a judge

s750
00:56:18.319 --> 00:56:30.160
uh and we're going to start with Phoenix's builtin evals Um code can't check whether code can check whether the ticker is in the report but it can't tell you if the analysis is any good.

s751
00:56:30.160 --> 00:56:31.680
Uh are the numbers accurate?

s752
00:56:31.680 --> 00:56:33.119
Is the report report complete?

s753
00:56:33.119 --> 00:56:35.119
Is the recommendation well reasoned?

s754
00:56:35.119 --> 00:56:39.520
These are questions about the meaning of the report not the textual content of the report.

s755
00:56:39.520 --> 00:56:43.680
And for those you need uh something that has judgment.

s756
00:56:43.680 --> 00:56:45.440
Uh an LLM judge has three parts.

s757
00:56:45.440 --> 00:56:48.799
It has a judge model which is the LM that is doing the judging.

s758
00:56:48.799 --> 00:56:52.000
Uh it has a rubric which is the prompt that tells it what good looks like.

s759
00:56:52.000 --> 00:56:55.280
And then it has the data which is the examples that it's grading.

s760
00:56:55.280 --> 00:56:58.079
Phoenix keeps those three separate so you can mix and match.

s761
00:56:58.079 --> 00:57:00.079
You can try the same rules with different models.

s762
00:57:00.079 --> 00:57:03.599
You can try the same judge with different rules.

s763
00:57:03.680 --> 00:57:08.640
Uh the good news for you is that Phoenix ships a bunch of built-in evals.

s764
00:57:08.640 --> 00:57:12.640
So you don't always have to write your own evals, at least not right off the bat.

s765
00:57:12.640 --> 00:57:15.760
Uh there's correctness which checks if the answer is accurate.

s766
00:57:15.760 --> 00:57:20.079
Uh there's faithfulness which checks if it sticks to the source material.

s767
00:57:20.079 --> 00:57:22.880
Uh we're going to see the difference between those two in a second.

s768
00:57:22.880 --> 00:57:26.799
Uh there's completeness, hallucination, refusal, PII, and toxicity.

s769
00:57:26.799 --> 00:57:30.720
Lots of things that lots of agents need to check are built in.

s770
00:57:30.720 --> 00:57:34.400
Uh and for agents, there are evals for tool use.

s771
00:57:34.400 --> 00:57:35.599
Did it pick the right tool?

s772
00:57:35.599 --> 00:57:37.440
Did it pass the right arguments?

s773
00:57:37.440 --> 00:57:41.280
Um did it do something sensible with with the result?

s774
00:57:41.280 --> 00:57:45.440
Uh these eval are not magical.

s775
00:57:45.440 --> 00:57:52.480
They are just uh prompts that we've already written and battle tested and made sure work for you.

s776
00:57:52.799 --> 00:57:57.839
Uh so let's find the judge model.

s777
00:57:57.839 --> 00:58:06.079
Um uh Phoenix evals in Typescript uh use the Versel AI SDK to talk to models.

s778
00:58:06.079 --> 00:58:09.760
So any model that the AI SDK supports can be a judge.

s779
00:58:09.760 --> 00:58:16.400
Uh in this case I'm using Claude Sonnet 5.5. Um, the agent runs on Haiku.

s780
00:58:16.400 --> 00:58:18.079
Like I said, it's cheap and fast.

s781
00:58:18.079 --> 00:58:23.119
Our judge is running on Sonnet because you want your judge to be more capable than the agent itself was.

s782
00:58:23.119 --> 00:58:25.760
So, give it Sonnet or give it Opus.

s783
00:58:25.760 --> 00:58:33.040
Uh, or give it uh even better um give it a model from uh a different provider entirely.

s784
00:58:33.040 --> 00:58:36.720
If you've got an OpenAI key, get OpenAI to judge the work of Claude.

s785
00:58:36.720 --> 00:58:41.440
Uh because Claude has what is known as uh same model bias.

s786
00:58:41.440 --> 00:58:50.160
So Claude tends to like the output of other versions of Claude uh more than it does the output of uh of OpenAI or some other provider.

s787
00:58:50.160 --> 00:59:00.240
Um the last line here uh in case you're wondering is suppressing tracing on the judge because we turned on tracing across the board.

s788
00:59:00.240 --> 00:59:02.720
The judge is itself a copy of Claude.

s789
00:59:02.720 --> 00:59:12.160
Uh so by default and in fact in my test data you will see that I forgot to turn on uh uh telemetry suppression.

s790
00:59:12.160 --> 00:59:20.240
So my traces are full of the the judge running itself and and seeing all of the traces from the judge which you don't need.

s791
00:59:20.240 --> 00:59:31.359
Um so every evaluator in this repo gets it gets this telemetry setting uh so that the judge stays out of uh our agent.

s792
00:59:31.359 --> 00:59:34.480
Um let's start with correctness.

s793
00:59:34.480 --> 00:59:36.480
Uh which is the obvious first choice.

s794
00:59:36.480 --> 00:59:39.520
Is the uh is this report correct?

s795
00:59:39.520 --> 00:59:42.240
Um that is one import and one line of setup.

s796
00:59:42.240 --> 00:59:43.200
You don't have to do anything.

s797
00:59:43.200 --> 00:59:46.160
You just say hey I'm running the correctness evaluator.

s798
00:59:46.160 --> 00:59:48.960
Run it on my stuff and let's see how it goes.

s799
00:59:48.960 --> 00:59:53.839
Uh then you call evaluate and you get back a label, a score, and an explanation.

s800
00:59:53.839 --> 00:59:57.599
And the script runs it on every agent run and logs the results back to Phoenix.

s801
00:59:57.599 --> 01:00:00.799
Uh the same way we did the ticker check.

s802
01:00:00.799 --> 01:00:06.079
Uh, I should be able to run that as well.

s803
01:00:12.799 --> 01:00:16.559
That's going to take a little while because it has to actually do some LLM stuff.

s804
01:00:16.559 --> 01:00:19.680
Uh, if it works, it works.

s805
01:00:19.680 --> 01:00:23.440
If it doesn't, I have stuff already built in.

s806
01:00:26.079 --> 01:00:33.440
So you can see uh in my annotations column you can see the results of various evaluations I've already run.

s807
01:00:33.440 --> 01:00:42.079
In this case it's showing you the uh actionability one but you can see uh correctness, faithfulness, human actionable uh and mention sticker.

s808
01:00:42.079 --> 01:00:43.520
We're going to talk about what all of those are.

s809
01:00:43.520 --> 01:00:45.839
But first you're going to see correctness.

s810
01:00:45.839 --> 01:00:51.599
Uh correctness on all of these is zero.

s811
01:00:51.599 --> 01:00:57.520
Uh, all 16 reports have been judged as incorrect.

s812
01:00:57.520 --> 01:01:04.079
And the reason that they've been judged as incorrect is because Claude judges correctness by what it knows.

s813
01:01:04.079 --> 01:01:06.559
That's the only thing that it has available to it.

s814
01:01:06.559 --> 01:01:12.240
And Claude's knowledge cut off uh at the moment is in something like January.

s815
01:01:12.240 --> 01:01:19.839
So uh my agent is talking about stock prices in October and Claude is freaking out and going, "You're talking about stuff that's in the future.

s816
01:01:19.839 --> 01:01:21.440
That's obviously a hallucination.

s817
01:01:21.440 --> 01:01:24.160
This report cannot possibly possibly correct.

s818
01:01:24.160 --> 01:01:29.440
So our built-in check for correctness fails every single report for that reason.

s819
01:01:29.440 --> 01:01:32.880
Um which is not terribly useful.

s820
01:01:32.880 --> 01:01:37.760
Um so the eval is half wrong and half right.

s821
01:01:37.760 --> 01:01:43.520
Seven of in my data, seven of the 16 explanations complain that the judge can't verify fresh data.

s822
01:01:43.520 --> 01:01:46.319
Uh which is the eval definitely being wrong.

s823
01:01:46.319 --> 01:01:49.200
Um, but lots of them also point out something real.

s824
01:01:49.200 --> 01:01:52.720
Things like the numbers don't add up or dates that contradict each other.

s825
01:01:52.720 --> 01:01:55.680
Um, so a score of zero is telling us nothing here.

s826
01:01:55.680 --> 01:02:01.359
It's telling us across the board that this that this eval is probably not the right choice of eval.

s827
01:02:01.359 --> 01:02:08.400
Um, what we want is a question that the judge uh can actually answer.

s828
01:02:08.400 --> 01:02:14.400
So uh oh, I told you that already.

s829
01:02:14.400 --> 01:02:15.200
All right.

s830
01:02:15.200 --> 01:02:17.760
uh what we're going to use instead is faithfulness.

s831
01:02:17.760 --> 01:02:19.760
Faithfulness is asking a much better question.

s832
01:02:19.760 --> 01:02:24.400
It's saying given the data that you had, did your did your report make sense?

s833
01:02:24.400 --> 01:02:28.000
Did is the answer based in the data that we were given?

s834
01:02:28.000 --> 01:02:36.160
This works much better because it can work with data this exactly the same data that the agent was given and see if it gets a report that makes sense.

s835
01:02:36.160 --> 01:02:40.160
Uh so that's a question the judge can actually answer.

s836
01:02:40.160 --> 01:02:42.400
Uh and the context is already in the trace.

s837
01:02:42.400 --> 01:02:46.240
Every web search is its own span like we saw.

s838
01:02:46.240 --> 01:02:50.960
Um, every research turns output is the agent's own notes on what it found.

s839
01:02:50.960 --> 01:03:00.400
So my first version, my first eval uh only gave the judge uh the research notes and then I read the explanations and the judge was flagging numbers as made up

s840
01:03:00.400 --> 01:03:06.319
uh because my version of the eval hadn't told it that oh I I searched the web and actually found this data.

s841
01:03:06.319 --> 01:03:11.440
Uh so uh in the version you see we've given it the full context.

s842
01:03:11.440 --> 01:03:26.319
um our get agent runs helper is going to pull both uh the raw search results and the notes so that it can see uh what the research agent found and how it found that data.

s843
01:03:27.200 --> 01:03:29.599
Uh so like I said faithfulness takes three things.

s844
01:03:29.599 --> 01:03:31.760
It takes the input which is what the user asked.

s845
01:03:31.760 --> 01:03:37.383
It takes the output which is the report and it takes the context which is the search results and the notes that I was just talking about.

s846
01:03:37.383 --> 01:03:37.920
[snorts]

s847
01:03:37.920 --> 01:03:40.640
Uh and you run it using the same pattern as before.

s848
01:03:40.640 --> 01:03:44.319
You create the evaluator, you evaluate every run, you log the results.

s849
01:03:44.319 --> 01:03:52.240
Um, like I said, it's slower than correctness uh because the judge is reading every search.

s850
01:03:52.240 --> 01:03:58.160
Uh, who has got any results out of their uh eval so far?

s851
01:03:58.160 --> 01:04:01.520
Again, you've got a faster Wi-Fi connection than everybody else.

s852
01:04:01.520 --> 01:04:03.839
Anybody else?

s853
01:04:04.000 --> 01:04:06.000
Cool.

s854
01:04:06.160 --> 01:04:12.640
Um, so let's look in Phoenix at my faithfulness reports.

s855
01:04:12.640 --> 01:04:21.280
Um, I got a lot of zeros on faithfulness, but not all of them.

s856
01:04:22.000 --> 01:04:26.000
Um, let's read the Microsoft explanation.

s857
01:04:26.000 --> 01:04:30.079
That was the one that I read that was most interesting.

s858
01:04:30.079 --> 01:04:34.000
Uh, as you can see, next to each one, you get the explanation.

s859
01:04:34.000 --> 01:04:39.520
Um, so in this one, the response adds several claims that the context does not support.

s860
01:04:39.520 --> 01:04:41.359
Azure is the number two cloud provider.

s861
01:04:41.359 --> 01:04:47.599
The backlog gives three years of visibility and there are implied FY202 targets.

s862
01:04:47.599 --> 01:04:54.640
Uh, it is telling me what it doesn't like about uh this thing.

s863
01:04:54.640 --> 01:04:58.160
It does it's telling me why it thinks this thing is unfaithful.

s864
01:04:58.160 --> 01:05:08.400
Um, it's complaining that uh some of the 2027 targets are fabricated because they're not in any search results.

s865
01:05:08.400 --> 01:05:17.440
Uh, the Nvidia one is also interesting if I can find that one.

s866
01:05:20.319 --> 01:05:20.640
Right.

s867
01:05:20.640 --> 01:05:23.599
In this one, it says the response conflates two different data points.

s868
01:05:23.599 --> 01:05:28.720
This is a reasoning error that the agent that the judge has uh accurately found.

s869
01:05:28.720 --> 01:05:34.160
Um it says the context information this is fabricated.

s870
01:05:34.160 --> 01:05:36.000
These are much more useful results.

s871
01:05:36.000 --> 01:05:43.920
We've got uh 14 out of 16 uh reports marked as unfaithful here, but the explanations are way more useful.

s872
01:05:43.920 --> 01:05:47.280
Uh and it's actually doing the thing that we wanted it to do.

s873
01:05:47.280 --> 01:05:52.880
Um so for agents, you can also grade the tool calls.

s874
01:05:52.880 --> 01:05:56.319
Uh so we saw the web search spans inside each trace.

s875
01:05:56.319 --> 01:05:59.520
Phoenix has built-in evals for all of those as well.

s876
01:05:59.520 --> 01:06:05.280
Tool selection asks did the agent pick the right tool uh or search when it didn't need to.

s877
01:06:05.280 --> 01:06:11.520
Tool invocation asks did it pass sensible arguments to those tools like a search query that's actually about the company.

s878
01:06:11.520 --> 01:06:17.839
And tool response handling asks did it do something sensible uh with the result of that call.

s879
01:06:17.839 --> 01:06:22.160
Uh these are the decisions that can make a break make or break an agent.

s880
01:06:22.160 --> 01:06:26.000
Um, and they all happen before the report actually gets read.

s881
01:06:26.000 --> 01:06:32.480
Uh, and we already know our agent makes strange tool choices sometimes, like it tries to save the report to a file.

s882
01:06:32.480 --> 01:06:36.480
Uh, so we can see if our built-in eval catches that.

s883
01:06:36.480 --> 01:06:39.760
Uh, I'm not going to run this one, but you can run that one.

s884
01:06:39.760 --> 01:06:42.240
Um, tool selection takes three things.

s885
01:06:42.240 --> 01:06:45.680
What the agent was asked, which tools it could have used, and which ones it picked.

s886
01:06:45.680 --> 01:06:52.960
Uh and the script goes through every agent turn and every trace research and writing separately and collects the tool spans underneath each one.

s887
01:06:52.960 --> 01:06:58.559
Um the trace records which tools the agent called but not which ones it could have called.

s888
01:06:58.559 --> 01:07:03.280
So we have to write that list ourselves uh in tool selection.ts.

s889
01:07:03.280 --> 01:07:08.160
Uh then it is uh the same pattern as every other run.

s890
01:07:08.160 --> 01:07:12.400
Uh so npm run eval tools uh and then you're done.

s891
01:07:12.400 --> 01:07:16.079
Uh that one should take about 15 seconds.

s892
01:07:17.200 --> 01:07:22.880
So, let's look in Phoenix at tool selection.

s893
01:07:38.559 --> 01:07:39.920
Tool selection.

s894
01:07:39.920 --> 01:07:42.480
It's helping me out.

s895
01:07:44.799 --> 01:07:45.280
All right.

s896
01:07:45.280 --> 01:07:50.559
So, these are ones where it has decided that my tool selection was incorrect for whatever reason.

s897
01:07:50.559 --> 01:07:53.920
Uh, the user asked for a concise report based on prior research.

s898
01:07:53.920 --> 01:07:59.599
Writing it to a file with write is plausible and appropriate since write exists and is safe.

s899
01:07:59.599 --> 01:08:03.039
Uh, however, the LLM made a second redundant bash call.

s900
01:08:03.039 --> 01:08:04.160
Awesome.

s901
01:08:04.160 --> 01:08:06.880
Uh, yeah, it sure did.

s902
01:08:06.880 --> 01:08:12.079
Um, the judge failed six that tried twice.

s903
01:08:12.079 --> 01:08:13.359
It passed seven.

s904
01:08:13.359 --> 01:08:15.359
Six of the seven that tried once.

s905
01:08:15.359 --> 01:08:21.600
Uh uh overall 13 of my 16 writing turns tried to save a file.

s906
01:08:21.600 --> 01:08:30.960
Um and only one of these explanations said that the user never asked for a file to be saved.

s907
01:08:30.960 --> 01:08:34.159
So uh is this judge incorrect?

s908
01:08:34.159 --> 01:08:39.440
Is the judge uh accurately judging that the tool call was correct?

s909
01:08:39.440 --> 01:08:46.560
Um, our writing prompt said, "Write a concise financial report based on your research above."

s910
01:08:46.560 --> 01:08:48.640
Our prompt is genuinely ambiguous here.

s911
01:08:48.640 --> 01:08:50.880
We said, "Write, we didn't say where to write it to.

s912
01:08:50.880 --> 01:08:52.719
We didn't say what to write it for.

s913
01:08:52.719 --> 01:08:55.040
Uh, we didn't say who the audience was going to be."

s914
01:08:55.040 --> 01:09:03.520
So, the tool selection knows that and is looking at the input and is going, "Well, it seems a reasonable thing for the agent to try and write to disk here."

s915
01:09:03.520 --> 01:09:12.080
Uh so again what you're telling the agent to do and what you're telling the eval to do interact with each other.

s916
01:09:12.080 --> 01:09:20.880
Uh you can't necessarily trust that the tool call invocation is going to be right because in this case the tool call is citing the tool call eval is siding with the agent.

s917
01:09:20.880 --> 01:09:22.799
It's like no the the agent is right.

s918
01:09:22.799 --> 01:09:30.719
The agent gave us instructions uh the agent received instructions and did its best is basically the eval's position.

s919
01:09:30.719 --> 01:09:37.600
Uh, eval results live in the trace as well.

s920
01:09:37.600 --> 01:09:48.799
Um, so if you're looking at the traces in Phoenix, uh, you can see them here attached to which trace they got, which span in the trace they got attached to.

s921
01:09:48.799 --> 01:09:51.920
So you see actionability and correctness got attached to the root.

s922
01:09:51.920 --> 01:09:58.960
Uh, whereas tool selection is getting attached to the specific uh, turn that made that tool call.

s923
01:09:58.960 --> 01:10:02.239
Um, you can filter your traces by any eval.

s924
01:10:02.239 --> 01:10:04.560
Uh, you can sort by score to find your worst runs.

s925
01:10:04.560 --> 01:10:08.000
You can click straight from a failing score into the step that caused it.

s926
01:10:08.000 --> 01:10:11.840
Uh, from score to from score to cause in one click.

s927
01:10:11.840 --> 01:10:15.760
Uh, so all of that is the built-in eval.

s928
01:10:15.760 --> 01:10:20.800
Um, and the lesson here is that choosing the right eval beats tuning it.

s929
01:10:20.800 --> 01:10:23.040
Choosing correctness was entirely the wrong eval.

s930
01:10:23.040 --> 01:10:24.000
It didn't tell us anything.

s931
01:10:24.000 --> 01:10:26.239
It just told us a bunch of stuff was wrong.

s932
01:10:26.239 --> 01:10:31.120
tool selection did tell us something uh and faithfulness told us quite a lot.

s933
01:10:31.120 --> 01:10:45.520
Um so checking the right eval beats tuning it uh and knowing which question you're actually asking is really the core of getting this right which brings us to step five which is writing your own judge.

s934
01:10:45.520 --> 01:10:49.520
This is where the eval get specific to your application.

s935
01:10:49.520 --> 01:10:57.360
Uh, and again, this is when people are thinking about evals, this is what most people are thinking about, but I wanted to make sure that you got the basics in first.

s936
01:10:57.360 --> 01:11:00.159
You don't have to jump straight to writing a custom rubric.

s937
01:11:00.159 --> 01:11:03.040
You can write code evals to get a bunch of stuff out of the way.

s938
01:11:03.040 --> 01:11:13.840
You can write uh you can use built-in eval to do a bunch of obvious stuff without having to invent your own tool calling correctness uh uh evaluator.

s939
01:11:13.840 --> 01:11:20.320
Um, what I wanted my report to do was give me a buy, sell, or hold recommendation.

s940
01:11:20.320 --> 01:11:24.960
There is no built-in eval that checks for that because that would be a very specific thing indeed.

s941
01:11:24.960 --> 01:11:27.199
So, we're going to write our own.

s942
01:11:27.199 --> 01:11:34.159
Uh, every eval prompt I've seen that works well has five different parts and we're going to go through them one at a time.

s943
01:11:34.159 --> 01:11:38.640
The first part is you give the judge a role and context.

s944
01:11:38.640 --> 01:11:43.840
Uh, a thing you'll see all the time is you are an expert blah blah blah blah blah.

s945
01:11:43.840 --> 01:11:45.280
That does actually work.

s946
01:11:45.280 --> 01:11:46.719
That's why everybody puts it in there.

s947
01:11:46.719 --> 01:11:51.040
Like it helps a little bit to tell the to tell the agent what it is.

s948
01:11:51.040 --> 01:12:00.480
Uh, telling it it's an expert as opposed to just telling it that it is running up something else also seems to make a difference when they do sideby-side comparisons.

s949
01:12:00.480 --> 01:12:03.199
Um, it's just one of the weird things about LLMs.

s950
01:12:03.199 --> 01:12:07.199
You need to tell it it's really smart and it gets like confident and tries.

s951
01:12:07.199 --> 01:12:11.920
Um uh but what really matters is the context.

s952
01:12:11.920 --> 01:12:14.239
Um this is a financial report.

s953
01:12:14.239 --> 01:12:20.320
Uh we are expecting a report that that analyzes financial data and tells us real things.

s954
01:12:20.320 --> 01:12:25.679
Um and specifically what we care about is whether or not you could act on it.

s955
01:12:25.679 --> 01:12:30.719
So uh that gives us the explicit criteria that we include in our rubric.

s956
01:12:30.719 --> 01:12:34.719
Um so this is where a lot of people don't put in enough effort.

s957
01:12:34.719 --> 01:12:43.440
So don't write a good response is helpful and accurate because uh a judge doesn't know what helpful and it doesn't know what accurate means in this context.

s958
01:12:43.440 --> 01:12:47.440
Um a judge can't make a yes or no decision from that.

s959
01:12:47.440 --> 01:12:50.719
Write down exactly what makes a report actionable.

s960
01:12:50.719 --> 01:12:52.880
Uh so it makes a specific recommendation.

s961
01:12:52.880 --> 01:12:56.000
It names concrete risks and it looks forward not just back.

s962
01:12:56.000 --> 01:13:00.560
Those are things that a judge can look at and actually check.

s963
01:13:00.560 --> 01:13:05.040
Uh, and note that we're not just making up these things blind.

s964
01:13:05.040 --> 01:13:08.960
Um, your criteria come from your error analysis.

s965
01:13:08.960 --> 01:13:14.480
Uh, only summarizes data with no interpretation is a problem that we saw when we were reading our traces.

s966
01:13:14.480 --> 01:13:18.239
We are putting that in there because it's a problem that we are trying to avoid.

s967
01:13:18.239 --> 01:13:21.520
Um, that is the whole point of reading the data first.

s968
01:13:21.520 --> 01:13:31.440
Specific criteria give you consistent judgments and you get specific criteria by knowing in advance what the data is going to look like because you read it yourself.

s969
01:13:31.440 --> 01:13:34.880
Part three of a good rubric is to mark the data clearly.

s970
01:13:34.880 --> 01:13:37.280
Wrap each piece of data in its own XML tag.

s971
01:13:37.280 --> 01:13:42.640
Claude was trained on billions and billions of XML tags and it's really good at working with them.

s972
01:13:42.640 --> 01:13:48.320
Um, if you look into the code at our own rubrics, that's what we're doing.

s973
01:13:48.320 --> 01:13:49.816
uh our own prompts

s974
01:13:49.816 --> 01:13:49.920
[snorts]

s975
01:13:49.920 --> 01:13:52.320
uh wrap everything in XML tags.

s976
01:13:52.320 --> 01:13:59.840
Um and the double curly braces are template variables that Phoenix fills in uh with each span's input and outputs.

s977
01:13:59.840 --> 01:14:06.719
Uh the reason you do this is because you want clear boundaries between what is the input, what is the output, and what is your instructions.

s978
01:14:06.719 --> 01:14:12.400
Uh this helps against prompt injection, but it also helps uh the agent just getting confused.

s979
01:14:12.400 --> 01:14:23.840
uh in particular it helps you get con uh if you include examples which we're going to see mark the example as an example and not the data otherwise it will get very confused about that

s980
01:14:23.840 --> 01:14:44.960
um I love to include labeled examples but it is in fact a controversial proc uh a controversial practice uh in the eval world um criteria tell the judge what you mean in the abstract but examples show it exactly what you mean as a human, I find it much easier to do something when somebody has shown me an

s981
01:14:44.960 --> 01:14:50.239
example of how it's supposed to work than if they've just given me a list of instructions of how to work.

s982
01:14:50.239 --> 01:14:55.679
Uh, and, uh, a lot of LLMs feel that way as well, apparently.

s983
01:14:55.679 --> 01:15:07.199
Um, so when you test your rubric, uh, which we're going to do in the next step, uh, test it with examples and without examples and see which one does better.

s984
01:15:07.199 --> 01:15:11.679
Uh, because sometimes, uh, examples are going to help it do a better job.

s985
01:15:11.679 --> 01:15:15.920
And sometimes, uh, an example will make it overfit.

s986
01:15:15.920 --> 01:15:19.120
It will look at your examples and say, well, it doesn't look exactly like my example.

s987
01:15:19.120 --> 01:15:22.239
Or it will try and make every output look like your output example.

s988
01:15:22.239 --> 01:15:24.880
Uh, which is less helpful.

s989
01:15:24.880 --> 01:15:27.840
Um, I ran exactly that test on mine.

s990
01:15:27.840 --> 01:15:35.679
Uh, so in the rubric that I used, um, this was fascinating in a really nerdy way, which is why I included it.

s991
01:15:35.679 --> 01:15:44.880
Uh, I threw out my examples and with the examples, the judge agreed with me, uh, 15, 15, and 15 times out of 16 in three different runs.

s992
01:15:44.880 --> 01:15:52.800
Uh, and if I threw out the examples, it agreed with me 16 times twice, all 16 examples, and 15 times once.

s993
01:15:52.800 --> 01:15:58.400
Uh and so I was like I was very curious as to how throwing out the examples made it better.

s994
01:15:58.400 --> 01:16:10.960
Uh and the specific problem that it was having is that in one of the reports it did not include um uh it did not include a list of risks.

s995
01:16:10.960 --> 01:16:17.760
So as a human I read the report and it was like Microsoft is a buy and it didn't include any risks and I was like that's fine.

s996
01:16:17.760 --> 01:16:18.560
It's still actionable.

s997
01:16:18.560 --> 01:16:19.920
It included a buy.

s998
01:16:19.920 --> 01:16:23.440
Uh so I marked it as uh an actionable report.

s999
01:16:23.440 --> 01:16:27.120
Uh but my examples said you must include risks.

s1000
01:16:27.120 --> 01:16:34.880
And so the agent uh the LLM judge was looking at the rules and and looking at the examples and saying there must be risks in here.

s1001
01:16:34.880 --> 01:16:37.840
So this is not actionable because there's no risks.

s1002
01:16:37.840 --> 01:16:44.560
So that's genuinely an open question as to whether or not my human judgment was better or worse than the judges.

s1003
01:16:44.560 --> 01:16:51.840
The judge was being very strict and I was being loosey goosey when I as a human gave myself golden data.

s1004
01:16:51.840 --> 01:16:53.440
Sorry, can I ask a question?

s1005
01:16:53.440 --> 01:16:53.920
Sure.

s1006
01:16:53.920 --> 01:17:03.120
So what's your hypothesis then as to without the examples it agreed with you so it saw that Microsoft as being actionable?

s1007
01:17:03.120 --> 01:17:03.520
Yeah.

s1008
01:17:03.520 --> 01:17:12.640
Is that is that your hypothesis then that like simply because you didn't say risk was a necessary element it felt that that was a good actionable?

s1009
01:17:12.640 --> 01:17:16.400
I mean, it's not a hypothesis like the LLM judge comes with an explanation.

s1010
01:17:16.400 --> 01:17:17.520
So, it says

s1011
01:17:17.520 --> 01:17:19.360
So, it said that that was actionable,

s1012
01:17:19.360 --> 01:17:21.360
right?

s1013
01:17:22.239 --> 01:17:22.640
Yeah.

s1014
01:17:22.640 --> 01:17:30.320
So, like I said, in this case, it's genuinely unclear whether or not I was doing the right thing or the judge was doing the right thing.

s1015
01:17:30.320 --> 01:17:41.120
The judge decided to interpret the the rubric very strictly and I when I was marking these things as human actionable or not, uh was not being as strict.

s1016
01:17:41.920 --> 01:17:46.719
Um, here's an example of an actionable one.

s1017
01:17:46.719 --> 01:17:48.159
Uh, it has specific numbers.

s1018
01:17:48.159 --> 01:17:49.280
It has revenue growth.

s1019
01:17:49.280 --> 01:17:51.440
Uh, it has valuation compared to its sector.

s1020
01:17:51.440 --> 01:17:52.719
It names a concrete risk.

s1021
01:17:52.719 --> 01:17:57.440
And it ends with a recommendation you could actually follow, including when you'd change your mind.

s1022
01:17:57.440 --> 01:18:00.080
Um, and this is what a not actionable one looks like.

s1023
01:18:00.080 --> 01:18:01.120
It isn't wrong.

s1024
01:18:01.120 --> 01:18:03.120
Everything in it is is true.

s1025
01:18:03.120 --> 01:18:04.560
It's just not very useless.

s1026
01:18:04.560 --> 01:18:09.920
Investors should consider various faction uh factors is not a recommendation.

s1027
01:18:09.920 --> 01:18:16.560
Um, now that the judge has seen both, the hope is that it's more consistent on reports that fall somewhere in between.

s1028
01:18:16.560 --> 01:18:20.880
Uh, and whether it actually is, you find out by testing.

s1029
01:18:20.880 --> 01:18:24.320
Uh, part five of putting together a rubric is you constrain the answer.

s1030
01:18:24.320 --> 01:18:27.840
You ask it for things that are actionable or not actionable.

s1031
01:18:27.840 --> 01:18:34.800
A thing you used to have to do is you used to have to say output a string and the string must be the word actionable or not actionable.

s1032
01:18:34.800 --> 01:18:36.560
Uh, you don't have to do that anymore.

s1033
01:18:36.560 --> 01:18:42.719
what uh modern evaluators use tool calls uh to put their output.

s1034
01:18:42.719 --> 01:18:50.080
So those uh the choices available for that tool call are defined in the tool call and not in your rubric.

s1035
01:18:50.080 --> 01:18:53.360
Uh which makes them a lot more reliable.

s1036
01:18:54.480 --> 01:18:56.719
So now it is your turn to write the rubric.

s1037
01:18:56.719 --> 01:18:59.520
Uh open evalactionability.ts.

s1038
01:19:00.159 --> 01:19:03.440
Uh it has my rubric in it with all five parts.

s1039
01:19:03.440 --> 01:19:04.960
Um and you can make it yours.

s1040
01:19:04.960 --> 01:19:06.000
You can change the criteria.

s1041
01:19:06.000 --> 01:19:08.000
You can change the examples.

s1042
01:19:08.000 --> 01:19:11.920
Uh what would make you act on one of these reports?

s1043
01:19:11.920 --> 01:19:13.520
What did you see in your own traces?

s1044
01:19:13.520 --> 01:19:16.880
Write that down and that should become your rubric.

s1045
01:19:16.880 --> 01:19:19.760
Um it will take three or four minutes to run.

s1046
01:19:19.760 --> 01:19:22.640
Who has managed to stay up with me so far?

s1047
01:19:22.640 --> 01:19:24.159
Like who is at this point?

s1048
01:19:24.159 --> 01:19:25.040
Excellent.

s1049
01:19:25.040 --> 01:19:27.280
I have multiple hands.

s1050
01:19:27.280 --> 01:19:34.480
Uh like I said, the last time I did this workshop, everyone was completely lost because the 200 megabyte repo.

s1051
01:19:34.480 --> 01:19:38.480
So the fact that anyone's got it this is just great for me.

s1052
01:19:38.480 --> 01:19:41.199
Um this is how you wire it up.

s1053
01:19:41.199 --> 01:19:44.400
You create a classification evaluator which does all the plumbing.

s1054
01:19:44.400 --> 01:19:46.960
Uh you give it a name, you give it the model.

s1055
01:19:46.960 --> 01:19:50.159
Uh you give it your template.

s1056
01:19:50.159 --> 01:19:54.239
Uh and like I said uh out of band you give it its out its options.

s1057
01:19:54.239 --> 01:20:00.800
So you give it its choices which are actionable with a score of one and not actionable with a score of zero.

s1058
01:20:00.800 --> 01:20:06.445
Um, so let's look at actionability in Phoenix.

s1059
01:20:06.445 --> 01:20:08.445
[snorts]

s1060
01:20:22.480 --> 01:20:27.600
It's so cold in here that my fingers have gone all numb and I can't type properly.

s1061
01:20:28.719 --> 01:20:30.719
Okay.

s1062
01:20:36.560 --> 01:20:44.159
So, the report analyzes JPM's rate exposure and loan growth uh and a forward-looking output, but makes no buy, hold, sell recommendations.

s1063
01:20:44.159 --> 01:20:44.719
Fantastic.

s1064
01:20:44.719 --> 01:20:46.880
It's doing exactly what it's supposed to do.

s1065
01:20:46.880 --> 01:20:52.560
Uh it is complaining about uh the lack of recommendation of concrete recommendations.

s1066
01:20:52.560 --> 01:21:04.159
Um before we look at all of these recommendations, we should look at the split in the recommendations.

s1067
01:21:04.159 --> 01:21:11.360
Uh in my data, five of 16 reports were actionable and 11 were counted as not actionable.

s1068
01:21:11.360 --> 01:21:15.760
Uh and that is one of the most useful types of evals you can get.

s1069
01:21:15.760 --> 01:21:19.760
An eval where everything is perfect is not particularly helpful.

s1070
01:21:19.760 --> 01:21:22.640
An EVA where everything is wrong is also equally unhelpful.

s1071
01:21:22.640 --> 01:21:26.880
An EVA where you've got an even split that is a hill that you can climb.

s1072
01:21:26.880 --> 01:21:39.679
You've suddenly got uh some you've got a def a set of examples where things are actually working, a set of things where uh stuff isn't working and you can start figuring out the differences between them.

s1073
01:21:39.679 --> 01:21:47.440
Um so if you click through these traces, I'm not going to bother clicking through all these traces.

s1074
01:21:47.440 --> 01:21:52.000
In Rivian it says it never makes a buy, hold, sell recommendation or any equivalent next step.

s1075
01:21:52.000 --> 01:21:58.880
For Nvidia, it said the closing investment thesis risk level is moderate, is vague rating, not guidance.

s1076
01:21:58.880 --> 01:22:03.840
And uh for Meta, the verdict is the is only that the position is solid.

s1077
01:22:03.840 --> 01:22:05.679
So this is the same problem over and over.

s1078
01:22:05.679 --> 01:22:10.239
It is refusing to give concrete buy, sell, hold recommendations.

s1079
01:22:10.239 --> 01:22:19.040
Um, if we uh we can open the Phoenix prompt template.

s1080
01:22:19.040 --> 01:22:20.480
I'm not going to do it now.

s1081
01:22:20.480 --> 01:22:28.719
Um, but uh there's one more rule for rubrics, which is that you should treat eval prompts as code.

s1082
01:22:28.719 --> 01:22:36.239
Your eval prompts going into your into your custom eval are extremely sensitive to small changes in wording.

s1083
01:22:36.239 --> 01:22:45.679
Um, so, uh, one example of that I ran into while making this was, uh, I told you that you should put XML tags around stuff.

s1084
01:22:45.679 --> 01:22:49.760
I forgot to put XML tags around my examples to begin with.

s1085
01:22:49.760 --> 01:23:00.960
Uh, and with my examples as plain text, uh, I got a string of not actionable stuff, uh, because it was getting confused about what the data was and what my examples were.

s1086
01:23:00.960 --> 01:23:07.199
And when I wrapped them in XML tags, everything suddenly became actionable because it could tell what I was talking about.

s1087
01:23:07.199 --> 01:23:11.920
Um, this is there was no other change.

s1088
01:23:11.920 --> 01:23:16.560
I'd given it the same data, the same instructions, the same input and the same output every time.

s1089
01:23:16.560 --> 01:23:21.920
Uh, it just got less confused in the evaluator as to what was happening.

s1090
01:23:21.920 --> 01:23:25.199
Uh, so you have to treat your evals as code.

s1091
01:23:25.199 --> 01:23:29.280
You should version them, you should check them, and you should test them.

s1092
01:23:29.280 --> 01:23:38.239
Um, as I mentioned earlier, uh, there is a trap to fall into with evaluators, which is to try and write write one evaluator that does everything for you.

s1093
01:23:38.239 --> 01:23:42.800
It is called the god evaluator because it is such a a common pattern.

s1094
01:23:42.800 --> 01:23:48.320
Uh, you try and get it to check accuracy, tone, completeness, formatting, policy all in a single prompt.

s1095
01:23:48.320 --> 01:23:52.639
Uh, and it is a nightmare because if it fails, what thing failed?

s1096
01:23:52.639 --> 01:23:53.600
Was it inaccurate?

s1097
01:23:53.600 --> 01:23:54.719
Was the the tone wrong?

s1098
01:23:54.719 --> 01:23:55.679
Was the faithfulness wrong?

s1099
01:23:55.679 --> 01:24:04.320
which of the many many many things that you've told this god evaluator to evaluate is it actually wrong and you try and solve that by giving it 15 labels

s1100
01:24:04.320 --> 01:24:08.560
that it can spit out uh and the whole thing becomes very hard to track.

s1101
01:24:08.560 --> 01:24:15.280
Uh the best practice is to split this up split your evaluators into as many dimensions as you are testing.

s1102
01:24:15.280 --> 01:24:19.360
So one for faithfulness, one for correctness, one for actionability.

s1103
01:24:19.360 --> 01:24:25.440
Um, and you should also know which one of your evals is a guardrail and which one is a north star.

s1104
01:24:25.440 --> 01:24:28.159
So, a guardrail is something that should block the release.

s1105
01:24:28.159 --> 01:24:33.199
So, if the agent is making up a stock price, that should halt shipping.

s1106
01:24:33.199 --> 01:24:36.719
Uh, but north stars are things that you're aiming for.

s1107
01:24:36.719 --> 01:24:40.000
So, suggests other stocks worth a look would be nice to have.

s1108
01:24:40.000 --> 01:24:41.679
That would be a good thing to include.

s1109
01:24:41.679 --> 01:24:46.639
Your eval can look for that, but it doesn't have to gate shipping on that.

s1110
01:24:47.280 --> 01:24:50.239
Which brings us to step six, which is testing your tests.

s1111
01:24:50.239 --> 01:24:53.920
We've built a judge and it says that some of our reports aren't actionable.

s1112
01:24:53.920 --> 01:24:56.080
Uh but can we trust the judge?

s1113
01:24:56.080 --> 01:24:58.639
How do we know that the judge is right?

s1114
01:24:58.639 --> 01:25:04.080
Uh the trick to remember is that your judge is a classifier in the classic ML sense.

s1115
01:25:04.080 --> 01:25:08.400
Uh it is looking at a report and it is predicting a label as actionable or not actionable.

s1116
01:25:08.400 --> 01:25:11.199
Predictions uh can be right or wrong.

s1117
01:25:11.199 --> 01:25:16.080
So like any classifier, you measure it by comparing its predictions with the truth.

s1118
01:25:16.080 --> 01:25:19.040
Uh, and here the truth is what a human expert would say.

s1119
01:25:19.040 --> 01:25:23.040
You get to judge uh you get to mark the judge's homework.

s1120
01:25:23.040 --> 01:25:26.800
Uh, in this case, the human expert is going to be me.

s1121
01:25:26.800 --> 01:25:30.800
I'm going to be deciding whether or not stocks are buy, sell, or hold.

s1122
01:25:30.800 --> 01:25:34.400
Uh, not necessarily the expert that you want.

s1123
01:25:34.400 --> 01:25:36.239
Uh, but we're doing our best.

s1124
01:25:36.239 --> 01:25:37.760
It's just a demo.

s1125
01:25:37.760 --> 01:25:41.920
Um, how you do this is very simple.

s1126
01:25:41.920 --> 01:25:44.480
You open up a trace.

s1127
01:25:44.480 --> 01:25:49.920
uh you click the little pencil button and you can add annotations.

s1128
01:25:49.920 --> 01:26:01.920
So I have created an annotation in advance called human actionable uh and I can set it to actionable or not actionable and I can very quickly uh click through all of my traces

s1129
01:26:01.920 --> 01:26:05.280
and set that on each one which I've done in advance.

s1130
01:26:05.280 --> 01:26:09.679
Uh this is where you would get uh a domain expert involved.

s1131
01:26:09.679 --> 01:26:10.800
You would throw them at Phoenix.

s1132
01:26:10.800 --> 01:26:13.280
You'd say here are 10 traces, here are 20 traces.

s1133
01:26:13.280 --> 01:26:14.639
there's 50 traces.

s1134
01:26:14.639 --> 01:26:20.719
Go through and read the output and give me give me your golden data set about whether or not these things are good.

s1135
01:26:20.719 --> 01:26:22.719
Um,

s1136
01:26:24.385 --> 01:26:25.120
[snorts]

s1137
01:26:25.120 --> 01:26:30.400
I sat down and labeled all 16 of my reports properly against the same criteria that I gave the judge.

s1138
01:26:30.400 --> 01:26:36.000
Uh, and those are sitting in your seed project so that you don't have to do that because that would take you a long time.

s1139
01:26:36.000 --> 01:26:40.639
Um, what you're building when you do this, like I said, is called a golden data set.

s1140
01:26:40.639 --> 01:26:45.840
Uh it's examples where to the best of human knowledge you know that the answers are right.

s1141
01:26:45.840 --> 01:26:50.080
Um it is a lot of work so you should make that work count.

s1142
01:26:50.080 --> 01:26:54.480
So you should give yourself the same specific criteria that you gave the judge.

s1143
01:26:54.480 --> 01:26:57.280
That was the error that we were talking about earlier.

s1144
01:26:57.280 --> 01:27:03.679
Uh where I was not exactly reading the rules that I had put into my eval criteria.

s1145
01:27:03.679 --> 01:27:03.920
Right?

s1146
01:27:03.920 --> 01:27:05.920
I was disobeying one of my own rules.

s1147
01:27:05.920 --> 01:27:13.120
my rule said uh must include risks and I said that this report is fine even though it doesn't include risks.

s1148
01:27:13.120 --> 01:27:18.000
Uh and as a result me and the judge disagreed and I feel like the judge was actually more correct than I was.

s1149
01:27:18.000 --> 01:27:19.679
It was being stricter.

s1150
01:27:19.679 --> 01:27:25.040
Um so uh your enemy here is fatigue.

s1151
01:27:25.040 --> 01:27:34.239
Uh if you give yourself a hundred traces to judge by trace 50, you're going to be really bored of judging traces and you're going to get lazier about it.

s1152
01:27:34.239 --> 01:27:44.080
Uh so it is better to have 50 really well-labeled uh examples than a hundred where you were just clicking through and saying ah this looks fine.

s1153
01:27:44.080 --> 01:27:52.159
Uh the the higher the quality the data in your golden data set uh the more reliable everything is going to be.

s1154
01:27:52.239 --> 01:27:58.400
Um and one of the best things that you can do is that uh is continuously add to your golden data set.

s1155
01:27:58.400 --> 01:28:06.400
when you see a tricky example, when you see something strange happen in production, figure out what the answer should have been and put it into your golden data set.

s1156
01:28:06.400 --> 01:28:13.600
Uh, a few good rules for human go for a good golden data set is that you should write test tasks a human could pass.

s1157
01:28:13.600 --> 01:28:20.159
Uh, if your agent fails a task every single time, that is almost always going to be a broken task, not a broken agent.

s1158
01:28:20.159 --> 01:28:26.000
Um, and for each task, you should have an answer that you know is good, which proves the task can be done.

s1159
01:28:26.000 --> 01:28:28.080
Uh, you should also test in both directions.

s1160
01:28:28.080 --> 01:28:39.760
If you only check, for instance, that the agent searches the web when it's supposed to search the web, you're going to accidentally train your agent to search the web every single time, all the time, even when it doesn't need to.

s1161
01:28:40.320 --> 01:28:43.760
Uh, and here's a rule from machine learning that applies to judges as well.

s1162
01:28:43.760 --> 01:28:45.520
Don't tune on your test set.

s1163
01:28:45.520 --> 01:28:47.840
Uh, split your labels into two groups.

s1164
01:28:47.840 --> 01:28:54.000
One that you use to tune your evals and one that you use to test that they are working.

s1165
01:28:54.000 --> 01:28:59.440
Uh when you only have 16 answers in your golden data set, that makes it your test set very small.

s1166
01:28:59.440 --> 01:29:04.719
Uh because the rule is usually 75% in the tuning set, 25% in the test set.

s1167
01:29:04.719 --> 01:29:06.880
So there's only four in our test set.

s1168
01:29:06.880 --> 01:29:10.159
Uh but if you were up to 100, that would be fine.

s1169
01:29:10.159 --> 01:29:13.920
Um Phoenix data sets have splits built in.

s1170
01:29:13.920 --> 01:29:16.239
So this is very easy to do.

s1171
01:29:16.239 --> 01:29:20.560
Uh see what that looks like.

s1172
01:29:20.639 --> 01:29:25.679
uh so first it builds the golden data set with splits.

s1173
01:29:25.679 --> 01:29:28.000
It finds every report with a human label.

s1174
01:29:28.000 --> 01:29:32.639
Uh uh and each one becomes an example in a Phoenix data set.

s1175
01:29:32.639 --> 01:29:33.840
The input is the report.

s1176
01:29:33.840 --> 01:29:36.000
The expected output is the human's label.

s1177
01:29:36.000 --> 01:29:37.760
We sort them by span ID.

s1178
01:29:37.760 --> 01:29:42.080
Then every fourth example goes into the test split and the rest go into dev.

s1179
01:29:42.080 --> 01:29:47.679
Uh sorting first means that a report always lands in the same split however many times you run this.

s1180
01:29:47.679 --> 01:29:48.747
Uh, and span

s1181
01:29:48.747 --> 01:29:49.040
[snorts]

s1182
01:29:49.040 --> 01:29:51.920
ID links each example back to the trace it came from.

s1183
01:29:51.920 --> 01:29:56.159
Uh, so you can always click through to the original run.

s1184
01:29:57.360 --> 01:29:59.679
Then we run the judge against the humans.

s1185
01:29:59.679 --> 01:30:03.280
This is an experiment which is something that we're going to dig into in a little bit.

s1186
01:30:03.280 --> 01:30:06.719
Um, but for now uh an experiment takes a data set.

s1187
01:30:06.719 --> 01:30:12.480
It takes a task task to run on every example uh and evaluators to score the results.

s1188
01:30:12.480 --> 01:30:18.960
The task here is our judge and the evaluator is a tiny Kodi val which checks whether the judge's label matches the humans.

s1189
01:30:18.960 --> 01:30:22.400
Uh notice that we only run it in the devs split.

s1190
01:30:22.400 --> 01:30:27.040
Uh you can run this yourself by running npm run meta.

s1191
01:30:28.047 --> 01:30:30.000
[sighs]

s1192
01:30:30.000 --> 01:30:33.760
Let's see if we can get it to go.

s1193
01:30:44.320 --> 01:30:45.600
It's going to run an LM.

s1194
01:30:45.600 --> 01:30:50.480
So, uh, I'm going to assume that it does the same thing that it did last night.

s1195
01:30:50.480 --> 01:30:53.280
Um, the script should print a little table at the end.

s1196
01:30:53.280 --> 01:31:00.480
Uh, and on my 12 dev reports, the judge and I made the same call every time, which is great, but it's also only 12 reports.

s1197
01:31:00.480 --> 01:31:04.880
Uh, so read that as, you know, no sign of a problem yet.

s1198
01:31:04.880 --> 01:31:08.639
you're going to want a larger test set when you're running this in production.

s1199
01:31:08.639 --> 01:31:11.679
Um, cool.

s1200
01:31:11.679 --> 01:31:13.280
It actually completed it.

s1201
01:31:13.280 --> 01:31:20.239
Uh, the more interesting part is the two ports that I the two reports that I found hardest to label.

s1202
01:31:20.239 --> 01:31:32.239
Um, so in my Phoenix data, uh, there's Coca-Cola and there's Microsoft.

s1203
01:31:38.000 --> 01:31:43.760
Um the the first one is Coca-Cola.

s1204
01:31:43.760 --> 01:31:52.080
The Coca-Cola is the one that flipped uh when I changed from the XML tags to the uh from not having XML tags to having XML tags.

s1205
01:31:52.080 --> 01:31:55.679
Uh this is where when it got the examples it got better.

s1206
01:31:55.679 --> 01:32:01.040
Um borderline for me turns out to be uh borderline as well.

s1207
01:32:01.040 --> 01:32:09.760
the it the report calls KO ideal for conservative income oriented pol uh portfolios uh which sounds like a recommendation.

s1208
01:32:09.760 --> 01:32:14.000
So as a human I went back and forth on whether or not I should count that as a recommendation.

s1209
01:32:14.000 --> 01:32:21.280
Ultimately I decided it did not uh but the judge also goes back and forth on whether or not that counts as enough of a recommendation.

s1210
01:32:21.280 --> 01:32:27.600
Um and the second one was Microsoft uh which I'm not going to bother to click into which is the other way around.

s1211
01:32:27.600 --> 01:32:31.199
The report says buy with reasons uh and a time horizon.

s1212
01:32:31.199 --> 01:32:33.280
Uh and it never mentions a single risk.

s1213
01:32:33.280 --> 01:32:35.440
This is the one that I was talking about earlier.

s1214
01:32:35.440 --> 01:32:40.480
Uh my rubric lists name concrete risks as a criterion.

s1215
01:32:40.480 --> 01:32:45.040
Uh and I decided that was fine and the judge decided that that was not fine.

s1216
01:32:45.040 --> 01:32:46.639
It said it was weak on risks.

s1217
01:32:46.639 --> 01:32:47.840
There is no risk section.

s1218
01:32:47.840 --> 01:32:51.360
It clears the bar for actionability but though not by much.

s1219
01:32:51.360 --> 01:33:01.360
So, the judge looking at this stuff came up with exactly my human judgment on whether or not this was an actionable report, including the ambiguity.

s1220
01:33:02.159 --> 01:33:10.320
Um, and that ambiguity is still in our rubric because we haven't changed it yet.

s1221
01:33:10.320 --> 01:33:12.560
Uh, so it's very tempting to fix it.

s1222
01:33:12.560 --> 01:33:20.719
So, I tried I added one sentence uh which was a report is actionable only if it meets all four criteria because previously that was ambiguous.

s1223
01:33:20.719 --> 01:33:23.840
I said a report should include all of these things.

s1224
01:33:23.840 --> 01:33:26.080
I didn't say that it must.

s1225
01:33:26.080 --> 01:33:34.320
Uh so you can change the you can change the rubric and be more specific and say yes this absolutely must contain all these four criteria.

s1226
01:33:34.320 --> 01:33:41.280
Uh and on dev uh that broke Microsoft my 12 out of 12 agreement became 11 out of 12 agreement.

s1227
01:33:41.280 --> 01:33:53.920
Um and uh it also in the test split it broke Apple uh because Apple didn't have enough uh a big enough risk section as far as the agent was the judge was concerned.

s1228
01:33:53.920 --> 01:33:59.760
Um what that tells you is that I could change my rubric and make my eval worse.

s1229
01:33:59.760 --> 01:34:02.159
Uh I can make my eval less helpful.

s1230
01:34:02.159 --> 01:34:04.960
I can make my EVA less like what a human would do.

s1231
01:34:04.960 --> 01:34:08.239
Uh even though I'm trying to be specific.

s1232
01:34:08.719 --> 01:34:12.480
Um the script also prints precision and recall.

s1233
01:34:12.480 --> 01:34:18.480
Uh precision these are very important things again borrowed from the ML land or rather from stats land.

s1234
01:34:18.480 --> 01:34:25.280
Um precision asks when the judge says not actionable how often is it correct?

s1235
01:34:25.280 --> 01:34:30.800
And recall asks of all the reports that aren't actionable how many did the judge catch?

s1236
01:34:30.800 --> 01:34:35.440
Those two sound very similar but measure different things.

s1237
01:34:35.440 --> 01:34:48.480
Um, in most cases, you're going to want to prioritize recall because what recall gives you is it catches everything, but it gives you high recall catches everything, but it will give you

s1238
01:34:48.480 --> 01:34:49.760
uh false alarms.

s1239
01:34:49.760 --> 01:34:54.320
A false alarm costs you a minute or two when you look at a report that turns out to be fine.

s1240
01:34:54.320 --> 01:34:59.600
Uh, but a missed failure uh would give you a bad report that reaches a customer.

s1241
01:34:59.600 --> 01:35:03.520
Um, so you have to trade one for the other.

s1242
01:35:03.520 --> 01:35:10.239
If recall goes up uh um then uh precision will go down and vice versa.

s1243
01:35:10.239 --> 01:35:24.880
So for your application you have to decide whether being accurate uh as in throwing away some things that were actually correct uh but but uh you decided they weren't is better or worse

s1244
01:35:24.880 --> 01:35:30.400
than getting false alarms where you say something is important and it turns out not to be.

s1245
01:35:30.400 --> 01:35:34.639
Um judges also have biases.

s1246
01:35:34.639 --> 01:35:36.719
Uh I mentioned one of them already.

s1247
01:35:36.719 --> 01:35:38.639
Um one is position bias.

s1248
01:35:38.639 --> 01:35:44.880
Given two options, they tend to prefer depending on the model either which one comes first or which one comes last.

s1249
01:35:44.880 --> 01:35:48.400
It depends which way the model around which way around the model was changed.

s1250
01:35:48.400 --> 01:35:54.480
Uh but you should swap your options so that it's not always looking at those uh things in the same order.

s1251
01:35:54.480 --> 01:35:56.960
Um they have length bias.

s1252
01:35:56.960 --> 01:36:00.880
Uh, judges prefer long answers and they also have a confidence bias.

s1253
01:36:00.880 --> 01:36:04.800
Judges prefer answers that sound fluent and confident.

s1254
01:36:04.800 --> 01:36:08.480
Um, and the one that I mentioned already is self-preference.

s1255
01:36:08.480 --> 01:36:12.000
A model judging its own output tends to rate it higher.

s1256
01:36:12.000 --> 01:36:15.280
Uh, which is why we judged Haiku with sonnet.

s1257
01:36:15.280 --> 01:36:21.120
Uh, and ideally you judge haiku with something from open AAI.

s1258
01:36:21.120 --> 01:36:30.320
Um, but before you get too worried about how much you can trust a judge, it's worth noting that human judges also disagree with each other a lot of the time.

s1259
01:36:30.320 --> 01:36:38.880
Um, if you give two experts the same outputs in the same rubric and measured how often they agree, uh, there is a standard score for that called Cohen's Kappa,

s1260
01:36:38.880 --> 01:36:42.880
uh, and for this kind of task, it's often as low as 02 or.3.

s1261
01:36:42.880 --> 01:36:48.960
Uh, which is to say they agree with each other not very much more often than just random chance.

s1262
01:36:48.960 --> 01:36:49.916
Um

s1263
01:36:49.916 --> 01:36:50.080
[snorts]

s1264
01:36:50.080 --> 01:37:00.400
so the bar for your human judgment is h for the bar for your judge is human performance and h human performance uh is not actually that good when you compare humans to each other.

s1265
01:37:00.400 --> 01:37:07.040
So having a judge that disagree you don't do not need a judge that agrees with you 100% of the time.

s1266
01:37:07.040 --> 01:37:10.320
Uh your judge is never going to agree with you 100% of the time.

s1267
01:37:10.320 --> 01:37:14.639
What you want is a judge that does better than another human doing the judge would be.

s1268
01:37:14.639 --> 01:37:20.560
uh because your judge can run across thousands of examples and help itself out that way.

s1269
01:37:20.560 --> 01:37:25.920
Um one last principle on judges uh is that failures should seem fair.

s1270
01:37:25.920 --> 01:37:28.880
When something fails, it should be obvious what the agent did wrong.

s1271
01:37:28.880 --> 01:37:34.880
If you look at the failure and go, "Well, actually that looks fine to me," then the problem is probably your eval.

s1272
01:37:34.880 --> 01:37:48.080
Um there's a famous example of this uh when Claude Opus 4.5 first scored 42% on a benchmark called Corebench uh which tests whether agents can reproduce scientific papers.

s1273
01:37:48.080 --> 01:37:50.800
Uh anthropic were upset about that low score.

s1274
01:37:50.800 --> 01:38:00.480
Uh so a researcher ran the actual transcripts and found uh that uh Claude was just being uh right in a different way.

s1275
01:38:00.480 --> 01:38:09.040
Uh it the test expected uh six digits of precision and Claude gave the right answer to two digits of four digits of precision rather.

s1276
01:38:09.040 --> 01:38:11.760
Um and the test said that was wrong.

s1277
01:38:11.760 --> 01:38:13.520
There were ambiguous task descriptions.

s1278
01:38:13.520 --> 01:38:16.000
Some of the tasks couldn't be reproduced at all.

s1279
01:38:16.000 --> 01:38:20.960
Uh so Anthropic went through and they fixed all of the tests and they said these tests could be better.

s1280
01:38:20.960 --> 01:38:24.960
Uh and magically they got 95% on the eval.

s1281
01:38:24.960 --> 01:38:27.760
Um this happens all the time.

s1282
01:38:27.760 --> 01:38:40.320
SWEBench and all of the other uh benchmarks that you see are constantly being re-evaluated and manually tweaked and uh that is why you see version two, version three, version four of of

s1283
01:38:40.320 --> 01:38:49.199
uh the famous benchmarks uh because people are constantly finding that the benchmarks are not actually as good as they could be and your evals are going to end up with the same problem.

s1284
01:38:49.199 --> 01:38:55.280
Uh so you need to be constantly re-evaluating your test set.

s1285
01:38:55.679 --> 01:38:59.040
Uh which brings us to step seven which is uh experiments.

s1286
01:38:59.040 --> 01:39:00.960
We've got evals that we trust.

s1287
01:39:00.960 --> 01:39:07.119
Now we are going to use them to actually improve that the agent and prove that the fixed worked.

s1288
01:39:07.119 --> 01:39:12.719
Uh I'm going to go a little faster now as I've been going on a very leisurely pace.

s1289
01:39:12.719 --> 01:39:14.960
I'm getting beginning to run out of time.

s1290
01:39:14.960 --> 01:39:17.679
Um so the here's the problem.

s1291
01:39:17.679 --> 01:39:20.719
We know that some reports aren't actionable and we know why.

s1292
01:39:20.719 --> 01:39:22.560
So we change the prompt and then what?

s1293
01:39:22.560 --> 01:39:24.159
How do you know that it worked?

s1294
01:39:24.159 --> 01:39:27.600
Uh, how do you know that you didn't break something that was working before?

s1295
01:39:27.600 --> 01:39:31.360
If you run it on a couple of examples and eyeball it, then you're back to vibes.

s1296
01:39:31.360 --> 01:39:34.880
A prompt change that you haven't measured is just a vibe.

s1297
01:39:34.880 --> 01:39:37.119
What you want is a controlled comparison.

s1298
01:39:37.119 --> 01:39:40.560
You want the same inputs, the same eval before and after.

s1299
01:39:40.560 --> 01:39:42.960
That is an experiment.

s1300
01:39:42.960 --> 01:39:58.159
Um so uh what you can do in Phoenix is you can take a bunch of your evals that you have filtered down to not actionable say uh and you can select them and you can add them to a data set.

s1301
01:39:58.159 --> 01:40:04.320
Um that gives you a set to test against uh against specific things.

s1302
01:40:04.320 --> 01:40:07.840
Um I'm not going to do that since I've already done it over here.

s1303
01:40:07.840 --> 01:40:31.520
Um Um once you have a data set you can create turn that into an experiment and run uh your experiment against only the data set only against the evals that were failing as opposed to against your entire set every single time which you know if you've got a decent sized split and a decent sized test set is

s1304
01:40:31.520 --> 01:40:34.639
saving you thousands and thousands of tests every time.

s1305
01:40:34.639 --> 01:40:39.360
Uh now we're going to fix what the eval found.

s1306
01:40:39.360 --> 01:40:43.520
So, uh, if you're still coding along, congratulations.

s1307
01:40:43.520 --> 01:40:44.719
Open prompts.ts.

s1308
01:40:44.719 --> 01:40:47.040
There is an improved version of each prompt.

s1309
01:40:47.040 --> 01:40:49.119
Uh, or you can write your own.

s1310
01:40:49.119 --> 01:40:53.840
Um, and every change maps to a finding.

s1311
01:40:53.840 --> 01:40:58.880
So, the judge said, uh, makes no recommendation, so we asked for an explicit recommendation.

s1312
01:40:58.880 --> 01:41:00.719
It said mentions risks with no evidence.

s1313
01:41:00.719 --> 01:41:03.040
So, we asked for risks with supporting data.

s1314
01:41:03.040 --> 01:41:08.480
Faithfulness found invented numbers, so we said use only numbers that appear in your research.

s1315
01:41:08.480 --> 01:41:13.280
uh and reading traces found that the agent uh was trying to save files all the time.

s1316
01:41:13.280 --> 01:41:17.040
Uh so we tell it to reply with the report instead.

s1317
01:41:17.040 --> 01:41:18.159
We're not guessing here.

s1318
01:41:18.159 --> 01:41:24.400
We are coming up with specific things for the agent to do as a result of the previous results of our evals.

s1319
01:41:24.400 --> 01:41:29.360
Uh this is what the experiment script looks like.

s1320
01:41:29.360 --> 01:41:32.320
An experiment is a data set, a task and evaluators.

s1321
01:41:32.320 --> 01:41:34.639
The data set is the one that we just created.

s1322
01:41:34.639 --> 01:41:38.639
Uh the task is a function that just runs the agent on each example.

s1323
01:41:38.639 --> 01:41:49.280
Uh in this case, we're running the entire agent, but we could be running it on a specific subset of the agents behavior or a specific subtask uh if we wanted to.

s1324
01:41:49.280 --> 01:41:52.480
Uh our task hands back the report and the research.

s1325
01:41:52.480 --> 01:41:56.080
So each eval gets a small wrapper that picks out the fields that it needs.

s1326
01:41:56.080 --> 01:41:59.600
Uh and then faithfulness gets the research as its context.

s1327
01:41:59.600 --> 01:42:03.360
The same as before, uh Phoenix runs the task on every example.

s1328
01:42:03.360 --> 01:42:07.040
It scores every result and it stores everything so you can compare.

s1329
01:42:07.040 --> 01:42:12.719
If you want to run this, you can run npm run experiment and it will take about four minutes to run.

s1330
01:42:12.719 --> 01:42:17.600
Um, but I'm running out of time so I'm not going to wait for four minutes.

s1331
01:42:17.600 --> 01:42:21.040
Uh, I ran mine ahead of time.

s1332
01:42:21.040 --> 01:42:22.880
How many examples do you need?

s1333
01:42:22.880 --> 01:42:24.800
Uh, is a really good question.

s1334
01:42:24.800 --> 01:42:28.320
Uh, 10 to 20 examples gives you a rough direction of where things are going.

s1335
01:42:28.320 --> 01:42:30.000
Is very useful in dev.

s1336
01:42:30.000 --> 01:42:35.199
Um, if you're getting into production, uh, you're going to want hundreds of examples.

s1337
01:42:35.199 --> 01:42:42.400
Um, that is going to take a long time, but is, uh, very worth it for accuracy.

s1338
01:42:42.400 --> 01:42:52.320
Um, for every, uh, to have your error bars on your accuracy, you need to quadruple the number of examples.

s1339
01:42:52.320 --> 01:42:58.880
So, uh, going from 25 examples to 100 examples is really, really helpful.

s1340
01:42:58.880 --> 01:43:08.800
uh and is not that much effort going to get that big a jump again you'd have to go to 400 examples which begins to get into the territory of you're never going to bother to do this

s1341
01:43:08.800 --> 01:43:19.600
uh but the bigger the better uh and the more production failures you throw into your data set the better um so I ran mine three times and the reason I did that

s1342
01:43:19.600 --> 01:43:31.920
uh is because the agent is non-deterministic each run of each example uh can mislead you in either direction um And there are two useful ways of thinking about how an agent runs

s1343
01:43:31.920 --> 01:43:33.600
uh when it runs multiple times.

s1344
01:43:33.600 --> 01:43:44.800
The first is called pass at K which is uh does it succeed at least once in K tries and then there is pass to the K which does it increase

s1345
01:43:44.800 --> 01:43:50.320
every single time uh does it pass every single time I run uh K times.

s1346
01:43:50.320 --> 01:43:56.560
Um so as K goes up those two numbers diverge.

s1347
01:43:56.560 --> 01:44:02.080
Uh the more tries you give it, the more likely pass at K is to succeed because it succeeds at least once.

s1348
01:44:02.080 --> 01:44:08.239
The more tries you give it, the more likely it is to fail at least once and then pass it K pass to the K will go down.

s1349
01:44:08.239 --> 01:44:12.800
Um a coding assistant gets that gets it right on the second try is fine.

s1350
01:44:12.800 --> 01:44:14.800
So pass it K is fine.

s1351
01:44:14.800 --> 01:44:20.080
Um a support bot that's wrong runtime in every five is a huge problem.

s1352
01:44:20.080 --> 01:44:26.239
And a financial anal analyst uh that gives inaccurate recommendations is also a huge problem.

s1353
01:44:26.239 --> 01:44:31.679
Uh so you want pass to the K. You want it to pass every single time for those.

s1354
01:44:31.679 --> 01:44:41.360
Um when you're improving an agent, there's an order of operations based on how useful making a change is going to be.

s1355
01:44:41.360 --> 01:44:43.679
The first one is data quality.

s1356
01:44:43.679 --> 01:44:54.719
If your underlying data is bad, if it's hard to read, if it's difficult to interpret for the agent or for the judge, uh then nothing else you do, nothing else you tweak is going to be useful.

s1357
01:44:54.719 --> 01:44:58.239
Once you've got good quality data, then you move to the prompts.

s1358
01:44:58.239 --> 01:45:01.520
Uh that is what we've been messing with mostly today.

s1359
01:45:01.520 --> 01:45:07.679
Uh is to make things better at the prompt level by changing our rubric, by changing our agent.

s1360
01:45:07.679 --> 01:45:09.600
Then there's model choice.

s1361
01:45:09.600 --> 01:45:18.480
Uh model choice has a lot of leverage, but it also changes a whole bunch of stuff at the same time.

s1362
01:45:18.480 --> 01:45:25.440
uh and also the usual way that you're going to change your model is you're going to change it to a bigger model and that makes things more expensive.

s1363
01:45:25.440 --> 01:45:30.960
So that is uh creates a tradeoff uh that you don't always want to make.

s1364
01:45:30.960 --> 01:45:32.800
Uh and finally there's hyperparameters.

s1365
01:45:32.800 --> 01:45:35.679
Hyperparameters are things like temperature and stuff like that.

s1366
01:45:35.679 --> 01:45:41.520
Uh they're so useless that uh people have that modern models have stopped allowing you to tune them.

s1367
01:45:41.520 --> 01:45:45.199
You can't pass temperature to model to modern claw models.

s1368
01:45:45.199 --> 01:45:51.840
Um so if you're tempted to it is the least useful thing that you can pass.

s1369
01:45:53.199 --> 01:45:58.880
Um so if we're looking at my experiments you can look in your seed data and get see see the same experiments.

s1370
01:45:58.880 --> 01:46:13.520
Um you'll see that uh as I improved as I improved my runs I went from uh 16 out 15 out of 16 to 48 out of 48.

s1371
01:46:13.520 --> 01:46:15.679
and went from 5 out of 16 to 48 out of 48.

s1372
01:46:15.679 --> 01:46:19.119
And faithfulness went from 2 out of 16 to 3 out of 48.

s1373
01:46:19.119 --> 01:46:21.440
Uh, which is not that good, right?

s1374
01:46:21.440 --> 01:46:25.520
Like we really should have done better on faithfulness with my improved prompt.

s1375
01:46:25.520 --> 01:46:35.199
Um, the uh important thing is did anything that used to pass break?

s1376
01:46:35.199 --> 01:46:37.360
We can filter down to the passes split.

s1377
01:46:37.360 --> 01:46:39.199
I'm not going to do it in the UI right now.

s1378
01:46:39.199 --> 01:46:45.360
Uh, but all of the five reports that passed previously as I improved my prompt still passed.

s1379
01:46:45.360 --> 01:46:48.320
Uh, so actionability went straight to perfect.

s1380
01:46:48.320 --> 01:46:50.719
Um, but faithfulness was bad.

s1381
01:46:50.719 --> 01:46:54.719
Faithfulness uh went from 2 out of 16 to 3 out of 48.

s1382
01:46:54.719 --> 01:46:59.520
Uh, even though the prompt literally says only use numbers that appear in your research.

s1383
01:46:59.520 --> 01:47:05.040
Um, so we fixed what we measured in a single line of the prompt.

s1384
01:47:05.040 --> 01:47:07.520
Uh, didn't fix the agent making things up.

s1385
01:47:07.520 --> 01:47:10.800
That is the next thing that we have to fix for this agent.

s1386
01:47:10.800 --> 01:47:14.800
Uh we have fixed whether or not it is making a good judgment call.

s1387
01:47:14.800 --> 01:47:21.840
Uh but we have not fixed uh that it is uh inventing the data some of the time.

s1388
01:47:22.800 --> 01:47:33.679
Um if we want to we can look into uh each of the runs and you'll see that each of the because it's nondeterministic each of the runs are coming up with different advice.

s1389
01:47:33.679 --> 01:47:38.719
Um, but I'm not going to do that because I'm running out of time and I want to get to some other stuff.

s1390
01:47:38.719 --> 01:47:46.960
Uh, let's get to Okay, cool.

s1391
01:47:46.960 --> 01:47:54.800
Um, a good eval like the 56 the 511 split that I was talking about on my 16 data sets.

s1392
01:47:54.800 --> 01:47:56.880
That is what is called a capability eval.

s1393
01:47:56.880 --> 01:47:59.840
A capability eval is a thing that our agent can't do yet.

s1394
01:47:59.840 --> 01:48:03.440
And it gives us a hill that we can measure ourselves climbing.

s1395
01:48:03.440 --> 01:48:09.840
Uh when it passes reliably, when you get a test that passes 100% of the time, you don't throw it away.

s1396
01:48:09.840 --> 01:48:12.239
You turn it into what is called a regression eval.

s1397
01:48:12.239 --> 01:48:16.480
That is the kind that you can put into your CI and say this used to pass 100% of the time.

s1398
01:48:16.480 --> 01:48:22.320
If it suddenly stops passing 100% of the time, then we can call that a failure and we should stop release.

s1399
01:48:22.320 --> 01:48:26.560
Um how do we climb this hill is the question.

s1400
01:48:26.560 --> 01:48:35.520
uh we could work on the prompt again or uh we could do the thing that I said that we would do which is we can upgrade the model.

s1401
01:48:35.520 --> 01:48:42.239
Uh so you can run the experiment again by passing in a new model haiku 5.5 and then run npm experiment.

s1402
01:48:42.239 --> 01:48:47.440
Um, I said at the start of today that you can't switch models without eval.

s1403
01:48:47.440 --> 01:48:54.000
And this is what it looks like when you can switch a model with an eval because we know now that we are measuring all of these things.

s1404
01:48:54.000 --> 01:48:58.639
Um, with three repetitions, it takes 13 minutes, which is more time than we've got left.

s1405
01:48:58.639 --> 01:49:00.800
So, I'm just going to skip to the results.

s1406
01:49:00.800 --> 01:49:06.320
Uh, here is Haiku 4.5 versus 5.5. Uh, faithfulness improved.

s1407
01:49:06.320 --> 01:49:08.159
It got more than three times better.

s1408
01:49:08.159 --> 01:49:09.360
It is still terrible.

s1409
01:49:09.360 --> 01:49:12.159
This agent is still making things up all the time.

s1410
01:49:12.159 --> 01:49:14.560
Uh, but consistent calls got better.

s1411
01:49:14.560 --> 01:49:17.119
So, it is at least making the same call over and over.

s1412
01:49:17.119 --> 01:49:18.560
Is that good?

s1413
01:49:18.560 --> 01:49:21.600
Uh, it's also costing four times as much money.

s1414
01:49:21.600 --> 01:49:23.280
Is that worth it?

s1415
01:49:23.280 --> 01:49:28.639
Uh, is a question that you need to make for yourself in production when you're doing this kind of thing.

s1416
01:49:28.639 --> 01:49:39.520
Um, one of the ways that it got more consistent, one of the ways it stopped varying between buy and sell between calls, uh, is it just got more conservative.

s1417
01:49:39.520 --> 01:49:43.760
Haiku 5.5 uh it turns out just says hold all the time.

s1418
01:49:43.760 --> 01:49:48.000
Uh so uh is that better or worse?

s1419
01:49:48.000 --> 01:49:49.119
Probably worse.

s1420
01:49:49.119 --> 01:49:55.520
Uh probably we need to iterate on Haiku 5.5 being too conservative uh about things.

s1421
01:49:55.520 --> 01:50:05.119
But at least we know now that something has changed and that the conserviveness is a problem with Haiku 5.5. Um that is the problem that I found.

s1422
01:50:05.119 --> 01:50:13.920
It went it went to 41 of out of uh 45 calls it said uh hold 41% of the on 41 of the times.

s1423
01:50:13.920 --> 01:50:17.520
Um which is perfectly consistent between runs.

s1424
01:50:17.520 --> 01:50:24.159
Uh so it it got better at the uh consistency check that I was talking about over here the consistent call.

s1425
01:50:24.159 --> 01:50:28.676
Uh but it got there by just picking the same answer every single time.

s1426
01:50:28.676 --> 01:50:28.960
[snorts]

s1427
01:50:28.960 --> 01:50:34.639
Um but that is the eval iterate cycle.

s1428
01:50:34.639 --> 01:50:37.520
Uh and the cycle is where the value is not anyone's score.

s1429
01:50:37.520 --> 01:50:45.119
You run your evals, you read the failures and the explanations, you spot the pattern, you change the prompt or the tools or the data and you run the experiment again.

s1430
01:50:45.119 --> 01:50:46.560
Did it get better?

s1431
01:50:46.560 --> 01:50:49.760
Uh your evals can now answer that question for you.

s1432
01:50:49.760 --> 01:50:54.000
You can go from I think it works to I can show you it works.

s1433
01:50:54.000 --> 01:51:00.800
Uh I have just enough time to talk about how to do the last step which is eval in CI.

s1434
01:51:00.800 --> 01:51:05.600
Uh more and more teams that ship AI quickly write the eval before the feature.

s1435
01:51:05.600 --> 01:51:07.920
That's called eval driven development.

s1436
01:51:07.920 --> 01:51:13.840
Um say you want your support agent to always check who the customer is before giving a refund.

s1437
01:51:13.840 --> 01:51:21.920
Uh you can write the eval that checks for that first and now done has a definition that you can measure a hill that you can climb from the start.

s1438
01:51:21.920 --> 01:51:29.199
Um, we talked already about capability capability evals and regression evals.

s1439
01:51:29.199 --> 01:51:36.159
Um, here's how uh to create a eval that runs in CI.

s1440
01:51:36.159 --> 01:51:42.960
Um, if you've never used VEST, VEST is a JavaScript testr runner and Fenix can plug straight into it.

s1441
01:51:42.960 --> 01:51:48.159
Uh, if you open agent.eval.ts, you'll see that it looks like an ordinary test file because it is one.

s1442
01:51:48.159 --> 01:51:52.159
There is a describe block and a test for each query in our regression set.

s1443
01:51:52.159 --> 01:51:57.520
So Coca-Cola, Apple versus Microsoft, Rivian, uh, and the fake ZZZQ.

s1444
01:51:57.520 --> 01:52:03.119
Um, each test runs the agent, logs the report as the output, uh, and then runs our evals on it.

s1445
01:52:03.119 --> 01:52:06.080
And the same ticker check and actionability judge that we wrote earlier.

s1446
01:52:06.080 --> 01:52:09.520
We don't have to add new evals for CI.

s1447
01:52:09.520 --> 01:52:13.760
Uh, then at the bottom, we gate the build on acceptance criteria.

s1448
01:52:13.760 --> 01:52:16.800
This is where I talked about guardrails and northstars.

s1449
01:52:16.800 --> 01:52:18.480
Mention sticker is a guardrail.

s1450
01:52:18.480 --> 01:52:21.040
It has to be talking about the right company.

s1451
01:52:21.040 --> 01:52:23.599
Um actionability is a target.

s1452
01:52:23.599 --> 01:52:26.080
The average has to stay 08 or better.

s1453
01:52:26.080 --> 01:52:29.040
Uh because an LLM judge can be a bit noisy on a single report.

s1454
01:52:29.040 --> 01:52:33.280
So we gate on the average uh not on just one.

s1455
01:52:33.280 --> 01:52:38.560
Uh and then you can run npm test uh and it runs like any test suite.

s1456
01:52:38.560 --> 01:52:42.080
Um and at the end the reporter prints out a scorecard.

s1457
01:52:42.080 --> 01:52:46.320
Um, and that'll take about four minutes, which is more time than I've got left.

s1458
01:52:46.320 --> 01:52:49.920
Um, a red build.

s1459
01:52:49.920 --> 01:52:52.800
If a failed build, if your build fails, this is what you'd get.

s1460
01:52:52.800 --> 01:52:56.800
All four tests ran and every report mentioned its ticker, but Rivian's report wasn't actionable.

s1461
01:52:56.800 --> 01:53:00.000
So, the actionability average came out at 75.

s1462
01:53:00.000 --> 01:53:01.679
Uh, it doesn't say you failed.

s1463
01:53:01.679 --> 01:53:03.360
It just says that you missed.

s1464
01:53:03.360 --> 01:53:11.520
Um, but, uh, a green run, uh, is where all of those pass.

s1465
01:53:11.520 --> 01:53:17.760
Um, I'm not going to show the experiment view because again, I'm running out of time.

s1466
01:53:17.760 --> 01:53:20.960
Uh, but when a build fails, you don't just get a red cross.

s1467
01:53:20.960 --> 01:53:25.440
You get an explanation that says whether things are good or bad and why they went wrong.

s1468
01:53:25.440 --> 01:53:27.199
And that is the important thing.

s1469
01:53:27.199 --> 01:53:30.800
Uh, this same test suite can be run in GitHub actions.

s1470
01:53:30.800 --> 01:53:33.920
Uh, it starts Phoenix as a service container.

s1471
01:53:33.920 --> 01:53:40.239
Uh, but now we've done everything that I've said that you can do.

s1472
01:53:40.239 --> 01:53:51.040
uh you can run your evals, you can test things, you can make sure uh that it matches your human judgment, you can put it into CI and you can make sure that your builds are gated on whether or not

s1473
01:53:51.040 --> 01:53:58.719
uh the LLM's judgment of what you've encoded as the definition of good uh really works.

s1474
01:53:58.719 --> 01:54:06.880
Um and there's something I love about this which is that anyone who knows what good looks like can write evals now.

s1475
01:54:06.880 --> 01:54:11.840
uh your product managers, your support team, your person in sales who keeps hearing the same complaint.

s1476
01:54:11.840 --> 01:54:13.520
They don't need to write TypeScript.

s1477
01:54:13.520 --> 01:54:24.639
Uh they just need to describe what a good answer looks like uh and give it to you or even better just give it to a coding agent which will then write all of the evals for them.

s1478
01:54:24.880 --> 01:54:33.360
Uh you shouldn't run your whole agent on every single commit because uh it's going to take a really long time and cost you a whole bunch of money.

s1479
01:54:33.360 --> 01:54:37.040
Um, but now you've got the repo I promised you.

s1480
01:54:37.040 --> 01:54:38.159
You've got all of the code.

s1481
01:54:38.159 --> 01:54:39.280
You've got all of the examples.

s1482
01:54:39.280 --> 01:54:40.960
You can see how everything is done.

s1483
01:54:40.960 --> 01:54:47.679
Uh, you can point your agent at it and say, "Do this, but for my codebase," uh, and your agent will figure it out.

s1484
01:54:47.679 --> 01:54:56.000
Um, there is one thing that we didn't get time to build today, uh, which is, uh, once you're live, you should keep evaluating.

s1485
01:54:56.000 --> 01:55:03.040
You should run your evals on a sample of real traffic, uh, and alert when scores drop and stay down.

s1486
01:55:03.040 --> 01:55:06.480
um not on every single failure because one failure would be noise.

s1487
01:55:06.480 --> 01:55:15.040
Um but if you get a trend in failures, uh production monitoring is incredibly useful and eval can get you that as well.

s1488
01:55:15.920 --> 01:55:18.400
So this is the full cycle that we built.

s1489
01:55:18.400 --> 01:55:24.960
We traced, we read, we evaluated, we checked, we improved, uh and we locked it in with CI.

s1490
01:55:24.960 --> 01:55:28.800
Uh but you don't have to do all of the stuff that you've learned today on Monday.

s1491
01:55:28.800 --> 01:55:30.000
You can start small.

s1492
01:55:30.000 --> 01:55:31.679
You can just instrument your agent.

s1493
01:55:31.679 --> 01:55:33.280
you can just read some traces.

s1494
01:55:33.280 --> 01:55:40.719
You will get information reading 12 traces that you wouldn't have got uh by just looking at the output.

s1495
01:55:40.719 --> 01:55:42.320
Um so run them.

s1496
01:55:42.320 --> 01:55:45.360
Look at the results and build from there.

s1497
01:55:45.360 --> 01:55:49.199
Uh so go try it on a app of your own.

s1498
01:55:49.199 --> 01:55:51.520
This is the repo that I said that everything was in.

s1499
01:55:51.520 --> 01:55:53.840
Uh the Phoenix docs are there.

s1500
01:55:53.840 --> 01:55:56.400
Uh Phoenix itself is open source.

s1501
01:55:56.400 --> 01:56:00.560
So you can mess with the code and tell us that we got stuff wrong in there.

s1502
01:56:00.560 --> 01:56:03.679
Uh and that just on time is it.

s1503
01:56:03.679 --> 01:56:06.005
Thank you very much everybody.

s1504
01:56:06.005 --> 01:56:08.005
[applause]