How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

AI Engineer · 116 min · 1504 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12My name is Lauri Voss.
00:13I'm head of developer relations at Arise AI.
00:16Uh I used to do JavaScript at npm, Inc. way back in the day, so you may remember me from then.
00:21Uh but these days I think about AI applications and how to make them run and how to make them actually work.
00:28Um, I had this slide up for the 15 minutes before.
00:31I hope you've seen it already.
00:32If not, now is the time to get started.
00:36Um, this is a moderately hands-on workshop.
00:40There's I've already built an agent for you.
00:43It's sitting there in that repo.
00:45Uh, you will get to play with the prompts and run actual evals.
00:48Um, but it's got checkpoints all the way through.
00:52So if you get lost, you can just switch to the right checkpoint and uh uh jump to the part where it's already working.
00:59Um you're going to need node 22 or later.
01:02All of this all of these examples are going to be in Typescript.
01:05Um because you know it's me, I'm going to use JavaScript if I can.
01:10Uh and uh you're going to be installing and running Phoenix on your own laptop.
01:16uh on conference Wi-Fi that can be a challenge, but I tried it this morning and it came through pretty quickly.
01:22So, uh UVX is magic.
01:24Um you're not going to need to run any Python in this thing.
01:27All of the Python is going to be run for you on your behalf by JavaScript, which is nice.
01:33Um I think that's it for the setup.
01:36Um so, what we're going to do today is um uh oh yeah, sorry, one more bit of setup.
01:44Once Phoenix is installed, uh you should open up your browser on localhost 606 so that you can see what it looks like.
01:51Um it's going to be empty right now, but we are going to run the commands that will fill it with data.
01:58Um you will also need an anthropic key.
02:03All of my examples are using Claude.
02:05Uh you can do all of this stuff using any LLM provider in the world, but I had to pick an example, so I picked the one that I use every day.
02:13So, this one is using Claude.
02:15Um, who needs help with the setup?
02:20Show of hands.
02:21Just shout at me.
02:22It's fine.
02:25All right.
02:25So, the first 10.
02:26What's up?
02:31Just getting started.
02:33Okay, great.
02:35Yeah.
02:36So, the first 10 minutes are theory, which will give you some time to catch up on the setup uh while I'm talking.
02:47So, let's get started with the theory.
02:49Um, here's the situation I'm assuming most of you are in.
02:53You have already built an application that has an LLM in it.
02:56Uh, maybe a chatbot, maybe an agent.
02:58You tried it a bunch of times and it looks pretty good.
03:01Uh, then you shipped it and somebody used it in a way that you weren't expecting and it broke.
03:06Um, or you changed the prompt to fix one thing.
03:10uh and you have this nagging feeling that you broke something else.
03:13Uh and there's no way of knowing.
03:15That is the problem that today is about.
03:17That is the problem that evals are here to solve.
03:19How do you know if your agent is any good?
03:23How do you know if your change made things better or worse?
03:26Uh and how do you stop it getting worse when you make changes uh without having to read every output yourself?
03:36My promise by the end of today is that you're going to have a working agent traced end to end with code evals with LLM judge evals and with a custom eval that you wrote yourself.
03:46Uh and you'll have checked whether your judge can be trusted.
03:51You'll have run an experiment to prove that your changes worked.
03:54Uh and you'll have have all of it running as a test suite with npm tests so you can put it into npmci.
04:01Um, that is the bit that I'm most excited about.
04:04So, it's going to be the last thing that we do.
04:06I see some people have walked in late.
04:08So, let me just go back to the setup slide very quickly so that they're not completely lost.
04:17Right.
04:19Remember this repo because you're going to want this repo.
04:32How is the Wi-Fi working for people?
04:34Is it Are you getting places?
04:36Excellent.
04:37Okay.
04:38I did this at World's Fair and uh every and I accidentally had a repo that was full of product images that looked really great in the demo, but it was 400 megabytes of product images and it just complete failure.
04:50Nobody was able to get even the repo down.
04:54So, I've learned my lesson and today has no images.
04:57Uh, so let's cover the basics.
05:01What is an eval?
05:02The shortest version, the one that I use whenever I'm talking to somebody who's been an engineer for a while is that uh there are these things called traces and they are basically logs and there's these things called evals and they're basically tests and evals are using are running tests on your traces.
05:20Um the difference between an eval and a unit test is that uh evals are for outputs that are never exactly the same twice, which is to say the non-deterministic
05:30outputs that you get from LLMs.
05:32Um traces record what your AI did at an incredible level of detail.
05:37So every agent step, every tool call, every LLM call, the inputs and outputs at every single step.
05:43A trace is made up of what we call spans.
05:46Each span is one step in the execution.
05:48You can think of it as just a line in the log.
05:50Um, and a so a whole agent turn is a span that has other spans nested inside of it.
05:56Uh, and each one records its input, its output, how long it took, how many tokens it used, uh, and sometimes how much it cost.
06:04So traces tell you what happened and evals tell you whether it was any good and that is the whole mental model for today.
06:10This is very far from being rocket science.
06:15Looks good to me is not a test suite.
06:17Um, but it's how most AI features get shipped.
06:21You build it, you try it a few times, does it look right?
06:24Yeah, that looks good.
06:25You ship it.
06:26Uh, and it probably does look good on the three things that you tried.
06:30Um, but three tries is not a test suite.
06:33And the usual fix is unit tests.
06:35Um, but unit tests don't work here because the same prompt is going to give a different answer text every single time you run your application.
06:43Uh, so two completely different answers can both be right.
06:46a thousand different answers can both can all be right.
06:49Uh just differently right.
06:51So there's no expected string to assert against.
06:54Uh so humans tend to fall back on human judgment.
06:58They look at the strings being output and they're like well this looks fine.
07:01[snorts]
07:01Um but that doesn't scale.
07:03It doesn't catch regressions and it certainly doesn't run in CI where there are no humans to watch things.
07:09Uh what that looks like in practice is you change your system prompts to fix the tone and the tone gets better.
07:16Uh but now the bot is making up product features.
07:18Uh and you don't find out until a customer does.
07:21Every prompt change can affect every kind of input your users send.
07:25Uh and you can't test all of those by hand.
07:27So you end up playing a game of whack-a-ole where you fix one thing and something else breaks and you fix that and something else breaks.
07:33uh evals fix that by giving you a number that you can track and can compare and act on.
07:42This matters more every month because uh every couple of weeks somebody releases a new model and you're like, "Oh, I want to switch to the new model."
07:50Uh but you have no way of knowing which things the new model is going to be better at uh and which things the new model is going to be worse at unless you have a set of evals that you can run things through.
08:00Uh, and that's exactly what we're going to do today.
08:03We're going to start with uh Claude 4.5 and then we are going to upgrade it to Claude uh Haiku 5.5 um uh which was conveniently released like a week ago making this demo extremely easy to put together.
08:17I was like I want to upgrade the model for this demo.
08:20So I'm going to make that the demo.
08:22Uh so with eval you run the suite you can compare the numbers and by the end of the afternoon you know uh whether you can upgrade your model.
08:31Um and there are a few kinds of evals.
08:33The first kind is code evals.
08:34Code evals are fast and free and a little bit brittle.
08:38They are very very similar to unit tests.
08:41They are just ordinary functions where you basically assert a string.
08:46There are a bunch of techniques you can use to make them more flexible than a straight unit test that asserts a basic string.
08:54Uh but fundamentally that's what they're doing.
08:56So the things that you can test with a Kodi valal are stuff like did the output parse as JSON?
09:01Is it under 500 words?
09:03Did it mention the thing that I asked about?
09:05The advantage of them is that they run in milliseconds and they cost nothing to run.
09:09Um, but there are lots of valid answers that a string match can't tell you about.
09:17Um, and for that you need the second kind of eval which is the LLM is a judge eval.
09:22Uh, which is what most people think about when they're thinking about evals.
09:26In this one you use a second LLM to grade the output of the first LLM.
09:31[snorts]
09:31Um, you grade it against what is called a rubric.
09:34There are a bunch of terms in uh eval land that are borrowed from ML uh that I find obscure the meaning of what's going on.
09:43One of them is traces and one of them is evals and the other one is rubric.
09:46I'm like rubric just means rules.
09:48Why do we have a special word for it?
09:50Anyway, uh L&M judges understand meaning not just strings.
09:56So stuff like is this accurate?
09:58Does it stick to the source documents?
10:00Is the tone right for a customer?
10:02Um, the catch is that because the LLM judge isn't itself an LLM, it has all of the flaws that LLMs have.
10:10So, it can be unpredictable.
10:11It is never the same twice.
10:13Uh, and it can make mistakes.
10:15Uh, they also cost money because you're using an LLM.
10:20Uh, and they can be confidently wrong.
10:22So, you have to check them.
10:23And that is one of the things that we're going to do today.
10:26Then there's a third kind of eval uh, which is the human eval.
10:30somebody who knows the domain, who can read the output and grade it as an expert.
10:35Uh, humans are the gold standard for quality.
10:39Um, they're also slow and expensive.
10:42Uh, and they get bored and start skimming.
10:44So, uh, you want to use all three at the same time.
10:48These are not competing approaches.
10:50They are complimentary approaches.
10:51And we're going to talk about a little bit about how you use them in a complimentary way later.
10:56So which eval do you use when?
10:59Use code when the answer is deterministic.
11:02So things like formats and lengths and required fields and band phrases.
11:07Uh but use an LLM judge when you need understanding.
11:10Is it accurate?
11:11Is it grounded?
11:12Is it useful?
11:13Uh and use humans for the kinds of failures you haven't seen before.
11:17Stuff that you haven't anticipated.
11:19And to check that your LLM judges are judging correctly.
11:24Um, all of that applies to any LLM app, but agents in particular make all of this stuff harder because agents are more complicated.
11:31A single simp a single LLM call is relatively simple.
11:35You give it an input, you get an output.
11:37Uh, but an agent tends to chain a whole bunch of steps together.
11:41So, one of the things that can go wrong is very early on in the chain, the agent makes a mistake and it cascades through all of the rest of the steps in your agent and each one gets progressively more off the rails.
11:53Um, and one of the biggest problems is being confidently wrong.
11:59Confidently wrong is worse than obviously wrong.
12:01Uh, so I talked about the cascading failure.
12:04Um, one example of that is say you've written a financial analysis agent, which is the kind of agent that I'm going to show you today.
12:12Um, and you ask it for a report on Tesla.
12:16It searches for Tesla and instead of getting results about the car company, it gets results about the 18th century inventor.
12:21It writes an beautiful, perfectly formatted report about an 18th century inventor.
12:26Uh, and none of your checks fail because you're like, it's well formatted.
12:32It's in a great tone.
12:33Everything is grammatical.
12:35Uh, it's just about completely the wrong subject.
12:38Uh, and that is the kind of failure that is uh um insidious with uh agents.
12:48Agents can also fail in the opposite direction.
12:50Um, Anthropic uh, who make Claude ran a benchmark called Ta2 bench which simulates customer service conversations and they uh, gave it this task where the user asked to change their flight.
13:05Uh, and the test according to the test the user was supposed to fail the test uh, or rather the agent was supposed to fail because uh, the person was in economy and you're not allowed to switch seats in economy.
13:18Uh but what cloud found is that they can upgrade the person to first class.
13:23You're allowed to change your seat in first class and then they downgraded them back after changing their seat back down to economy.
13:29Uh and pass and like did the thing that the user asked for and the test said no you failed because you weren't supposed to be able to do that.
13:37Um but the agent had been smarter than the rules that it was in.
13:41Um so you have to expect that uh when you are building eval that sometimes the agent is going to come up with a solution to the problem that you did not expect.
13:50So your tests have to be very flexible.
13:53Um here's what an eval result generally looks like.
13:57Uh it comes with a score, it comes with a label and it comes with an explanation.
14:01The score is a number usually just zero or one.
14:04Uh and the label is just a word that the human can read that matches what the score means.
14:09Uh so stuff like pass and fail, actionable and not actionable.
14:13Uh and LLM judges add a third thing which is an explanation of why you got the score you did, why you got the label you did.
14:20Um and it is the most valuable part of an LLM judge.
14:25Uh because the explanation is where the value is.
14:28It tells you what was wrong and what was missing and it tells you a a lot of the time what to fix in your prompt to get this right.
14:36Uh this is especially useful when you're running your evals at scale because you can get, you know, 50, 100, a thousand explanations from your LLM and you can get another LLM to read all of those explanations and be like, it looks like there is a pattern in these explanations.
14:50It looks like we have a systematic problem.
14:52And you can just feed them to your agent and your agent goes, okay, I see 50 traces all doing the same all with the same explanation of what went wrong.
14:59I can fix it.
15:01Um, so this is the loop that we're going to be building today.
15:06We're going to trace your agent.
15:07We're going to read what it did.
15:08We're going to write evals for what matters.
15:10Uh, we're going to check that those eval can be trusted.
15:13Uh, then we're going to change the agent and we're going to prove that the change worked.
15:18Then we're going to lock that in so that it stays fixed by putting it into CI.
15:22So, uh, who still needs help getting set up?
15:28Who walked in late and needs set up?
15:29Show of hands.
15:30Okay, let me head back to the setup screen and see if you can get it.
15:46Right, this is the magic URL.
15:50If you haven't got this one, uh, all you have to do is read the readme and, uh, run the instructions or just, you know, paste the readme into your agent and your agent will read the instructions and get it all done for you.
16:01One of the weird things about doing tutorials in 2026 is that we're just like, well, I could show you how to do all of this stuff, or I could just tell you how to tell your agent to do all of this stuff.
16:11Uh, and that's what we do a lot of the time.
16:16All right, I'm now going to assume that everybody has that URL.
16:19Cool.
16:31So let's start with step one.
16:33Step one is tracing.
16:34Before you can evaluate anything, you need to get the data to run the to run the evaluations on.
16:39And that is what that is traces.
16:42Today we are going to be using Phoenix.
16:43I hope that's not a surprise to anybody by now.
16:46Uh, Phoenix is ARIS's open-source AI observability platform.
16:50Uh, it collects traces from your AI AI app with in whatever framework you use to build it.
16:55It stores your evals next to those traces.
16:58Uh, and it gives you a UI to dig through all of it.
17:00It also does data sets and experiments, which we're going to be talking about today.
17:04Uh, and you run it on your laptop.
17:07So, uh, you can keep it entirely private to yourself, or you can run it on a server and share it with your team.
17:16So now I'm going to dangerously see if I can actually Oh, that's so that is so small.
17:24Let's make that bigger.
17:27Okay, so what we're working in today is Typescript.
17:31Uh you'll see a ton of examples in Python and Fenix has great Python SDKs as well.
17:37Uh but I like JavaScript so we're going to use TypeScript.
17:39Um the repo is a small set of uh scripts.
17:44Instrumentation.ts sets up tracing.
17:46Agent.ts is the agent itself.
17:48Prompts.ts has the prompts.
17:51Uh where are all of these things?
17:56There we go.
17:57Um and you'll be editing the prompts in this folder.
18:02Uh sorry, in this file.
18:04Um and the evals folder is where our evals go.
18:07Queries.json is our test data.
18:09Uh, and hopefully every step today is just going to be one npm command.
18:13Uh, and each one stands on its own.
18:15So if you fall behind, you can skip straight to the next one.
18:17Uh, and use the seed data that I put in the repo uh, so that you don't get left behind.
18:26So what we're building today, or indeed what I've already built for you, is a financial analyst.
18:31How it works is you give it a stock ticker and something to focus on like Tesla vehicle deliveries.
18:37uh and it will go and it will search the web for live data uh and write you a report about that.
18:43Uh there are whole startups built on this idea of giving you you know financial analysis of whether or not you should buy a stock.
18:50Uh but ours is deliberately simple because I needed something that would make mistakes so that we could fix the mistakes.
18:57Um today is not about how to build an agent.
19:02I've just built the agent for you.
19:03I assume you've already got an agent running somewhere and your what you want is help on making it reliable.
19:09Um, our agent works in two turns.
19:11It has a research turn where it searches the web and then it has a writing a report writing turn where it takes the research and it writes the report about it.
19:20Um, I'm I've built it using the claude agent SDK.
19:25If you haven't used it, that is anthropics framework for agents.
19:28Uh it is the same engine that runs underneath cla code which is going to become relevant later especially if you're using claude code to work with it uh because the two versions of cloud code tend to try to cooperate with each other and that makes them cheat the eval which is not very helpful.
19:42Um uh it gives you tool use web search uh and conversations that carry on across turns.
19:49There are also lots of other options that we are compatible with.
19:52The Versel AI SDK, Mastra, Langchain, OpenAI's agent SDK.
19:57Uh, Phoenix has adapters and instrumentation for every possible framework that you can think of and every LLM provider that you can possibly think of.
20:06Wherever you have built your agent in, Phoenix will work with it and the instrumentation will happen automatically.
20:11Uh, but I had to pick one for a demo, so I picked this one.
20:15Um,
20:16[snorts]
20:16tracing takes about five lines.
20:18Um, first we're going to call register uh from the Phoenix hotel package that sets up tracing and points it at Phoenix.
20:25One of the good things uh about the industry is that everybody has pretty much picked uh the same uh instrumentation standard.
20:35uh everything is built on open telemetry in the industry and on top of open telemetry there is a standard called open inference uh which arise takes care of um and everybody has already built the instrumentation into their frameworks that is why we are compatible with all of those frameworks
20:50all you have to do is use this registered command to say hey that instrumentation that you've already built into your framework turn it on and point it over here in this case you're just going to point it at your laptop
21:03uh and That's all you have to do.
21:05Every agent call that our agent makes is now going to be traced.
21:09Um, oh, I said that already.
21:13Uh, so the good thing about this is that we don't have to write a log line for every single call.
21:17We don't have to write a trace for every single thing our agent does.
21:20It's just going to do it automatically.
21:22Sometimes we will want to add additional information and the framework absolutely supports that.
21:26Um, uh, but we don't have to do it for every single thing.
21:31Um there's one gotcha in the TypeScript version that you should know about uh which is that manually you should use manually instrument which will hand you back a copy of the instrument
21:42uh and you should use the copy of the instrument uh because the cloud agent SDK is an ES module uh and ES module exports are readon.
21:52So the instrumentation can't do the thing like it can do in Python where it swaps out your instrument and just changes everything.
21:58you have to use the copy that it gives you which has that instrument swapped out.
22:02Um it won't break it just won't log anything if you don't use the copy that it gives you.
22:09Uh so everywhere in this repo we import claude agent SDK from our instrumentation file never straight from the package.
22:17Uh now if you open agent.ts you'll see our agents options.
22:22Uh we're going to start using Claude Haiku F 4.5 uh because it is fast, it is cheap, and it is sort of usefully dumb.
22:29Um it makes uh mistakes that are nice and predictable even with a relatively simple agent.
22:36Um unlike the frontier models which will still definitely make mistakes, but you have to have quite a complicated agent and quite a complicated set of tasks.
22:44Uh and it makes it harder for the demo.
22:46Um if you've not used it before then uh allowed tools is giving it web search so it can find the current financial data.
22:56Uh and the permission mode don't ask means that it's
22:58[snorts]
22:59uh going to not be interactive.
23:01It's not going to pause and ask you questions uh because there's nobody there to say yes.
23:06Um that option is going to become relevant uh to a failure mode later.
23:11[snorts]
23:12Um, and the other three options, like I mentioned, they keep the your laptop out of the agent.
23:16The cloud agent SDK, uh, runs the same engine as cloud code.
23:20Uh, so if you're doing this in cloud code, it tends to pick up the options that you and the tools and the skills that you've given to your copy of Claude code instead of its copy of Cloud Code.
23:30Uh, which tends to make things extremely non-deterministic in an unhelpful way.
23:35Uh so I've uh thrown in a couple lines of code there that make sure that the cloud uh the cloud SDK is running in its own isolated version of
23:45uh the cloud SDK and not uh stealing your copies of of you know keys and things from your data.
23:52Um
23:55[sighs]
23:57right uh none of that is Phoenix specific by the way.
24:00Uh that is just a danger of using the cloud SDK at the same time as you're using cloud code.
24:07Um in this SDK one turn is one query call.
24:11So you give it a prompt.
24:13Uh it streams back messages as the agent works.
24:15It thinks it calls a tool.
24:17It reads the result.
24:18Uh it maybe calls another tool and when it's finished you get a result message with the final text and a session ID.
24:24The session ID is how we carry on the conversation.
24:27If we pass it back in as resume, the next turn remembers everything that the last turn saw.
24:32Uh so that's how we can do the research and then say, hey, write a report on this research because it knows that it's in the same session.
24:40Uh and it can find the research that the previous turn just created.
24:45Then we put the two turns together.
24:47Uh so we get the two turns in one trace.
24:49Um first the research turn, then the writing turn, resuming the research turn session so the writer can see everything the researcher found.
24:56Um, and then we wrap the whole thing in trace agent.
25:00Um, the reason we do that, uh, is because it gives us one parent span that encompasses all of the data.
25:07If we didn't do that, we would get two, uh, two, uh, sibling spans in our Phoenix.
25:13Um, which would be fine, but slightly harder to read.
25:17Uh, so I'm just, uh, shaping the data now to make our lives easier later.
25:24Now, if you open up prompts.ts, ts.
25:26You'll see that uh there are a bunch of prompts in there.
25:28They are deliberately bad.
25:30Um there's one line of research instructions, one line of writing instructions.
25:34Uh and I did that on purpose because it gives us lots to improve later.
25:38Um if you're coding along, now is the time to start playing with things.
25:42Feel free to change them.
25:43Add stuff like make no mistakes or I will tip you $100.
25:47Um allegedly those help.
25:50Uh what I want is everyone in the room to end up with a slightly different agent uh so that we get different experiences uh which will make comparing notes more interesting.
26:02So now if you've followed along and you've installed everything and everything is going fine uh you should be able to run it.
26:10You should run npm run agent dash uh Tesla and then growth outlook in quotes.
26:17Um, you can do it on Tesla, you can do it on some other company if you like.
26:23Uh, it will take a minute or two because it really does have to actually search the web and you're on conference Wi-Fi.
26:29Um, and the model will do a few rounds of reasoning.
26:32Um, what's happening here while that's running for you uh is the agent gets the research prompt.
26:40It decides that it needs to search uh which is a tool call.
26:43It reads the results uh decides whether it has enough results or whether it needs to search again.
26:48And that is the thing that makes it an agent.
26:51We haven't specified how many times it needs to search the web.
26:54We haven't specified what it needs to search the web for.
26:56The agent is making those decisions by itself.
26:59Uh so all of our agents are going to make slightly different decisions about those things.
27:03They're going to use slightly different search terms.
27:06Uh they're probably going to search a different number of times.
27:09Um, and then once it's done, the writing turn starts and it turns all of that into a report.
27:16So the path the agent takes is nondeterministic by by design.
27:20Um, both reports might be fine if you run it a if you run it twice.
27:26Um, or one might be great and one might be wrong and you can't but you can't predict that uh output from the input.
27:34Uh, which is why you need evals.
27:36Um, who's got an actual report yet?
27:41Anybody?
27:43Still running.
27:45[snorts]
27:45Okay, we've got time.
27:50Let me be brave.
27:53And actually try running it myself.
27:56One of the problems that we're going to see that the agent runs into is that we told it to write a report, but we didn't tell it where to write a report.
28:03And so it assumes that it should be uh writing the report to a file.
28:08There we go.
28:08So you can see mine.
28:10I need permission to write the financial report file.
28:12Would you like me to display the report here?
28:14This is completely unhelpful.
28:15The agent does not understand that it is running in a no permissions environment and that I wanted the report actually output to the command line.
28:24Uh and that is because my prompt is ambiguous.
28:27Um but sometimes it doesn't do that.
28:29Sometimes it does just figure it out and give you the report.
28:32Uh, so that is the nondeterminism that we're talking about.
28:35So let's see what Phoenix captured from all of this.
28:41Here is my Phoenix 101.
28:43There is my financial report.
28:46This is what a trace looks like.
28:48So you can see the financial report agent running and you can see the two invocations of the claude agent.
28:55One is the this is the research report.
28:58You can see tool searches inside of it to find out whether what it's got available to it.
29:03You can then see it finds that it's got a web search tool.
29:06So it conducts three different web searches.
29:08You can see the inputs and the outputs.
29:11Uh what the exact query is, what output it got from each tool call.
29:15Uh and then you can see uh that it messed up in this case.
29:20uh it tried to write it it wrote a report and then it tried to write it to a file path on my computer and said that it couldn't work.
29:28Um so uh the important thing here is that we didn't have to add any logging to our agent.
29:37The important thing here is that all of this showed up all of this very useful information about what's going on without us doing anything.
29:43We didn't say hey your tools need to be instrumented.
29:46We didn't say hey your web searches need to be instrumented.
29:48All of that just happens as part of the cloud agent SDK because it's already instrumented for us.
29:53Uh and you can click into any span in Phoenix and you can see an enormous amount of data about what's going on.
29:59Um uh and the reason that all of this happened, the reason that it's trying to write to disk and then failing is because we gave it that don't ask setting.
30:11We told it you're going to be running unattended.
30:13Uh don't ask me any questions.
30:15it ran without asking any questions and then it messed up.
30:19Um, but you wouldn't know that if you hadn't been able to look at the trace, right?
30:24You just would have got a mysterious output where it says something was wrong.
30:28The traces are where you can see that the tool call is failing.
30:32The traces are where you can see it's trying to do something that it's not been given permission to do.
30:37Um, and that is the kind of stuff that lives in a span.
30:41Uh, so the span kind tells you what type of step it is.
30:44So it's an agent tool, an agent, a tool call, an LLM and so on.
30:48Um, and in a tool call and a web search and things like that, you can see that our input is in fact JSON.
30:56Um, and that's going to make our first eval uh because it gives us some structured data.
31:04Um, you also get things like timing and token counts.
31:08Um, and uh, often your agent will record what it costs.
31:14Um, when we write eval in a bit, these are the fields that we're going to pull out and grade.
31:26So, uh, you've run one, you've won one, you've run one test so far.
31:33Uh, that is not enough data to draw any conclusions from.
31:36So what I want you to do now is run npm run batch uh which will run 15 queries in a row.
31:42Uh that will take a little while.
31:44So I'm going to talk while you're doing that.
31:46Um it runs five at a time.
31:48It takes three or four minutes optimistically on Wi-Fi.
31:52Um so start it now.
31:54[snorts]
31:54Um if it fails for whatever reason, don't worry about it.
31:57The seed project that you loaded earlier uh contains is called Phoenix 101 golden.
32:03Uh, and it has my 16 runs that I did earlier on reliable Wi-Fi on it.
32:08So, you can use those instead if you can't get your batch to run.
32:13Um, here are some examples of the queries.
32:18Uh, there are obvious ones like Apple, Nvidia, and Amazon.
32:21And then there are some edge cases that I put in there to make things fall over.
32:25Uh, so one asks it to compare Apple and Microsoft, which is harder than just company covering one company.
32:31Uh, one is about Rivian.
32:32uh which is not a public company.
32:34So there's much less data out there than it does about uh public companies.
32:38Um Coca-Cola is a very different type of company from a tech company.
32:42Uh and ZZQ isn't a real ticker at all.
32:44I just made it up.
32:46Um and that is to see what the agent does when it sees a ticker symbol that it doesn't recognize.
32:51Does it, you know, invent a company?
32:53Does it guess what company you mean or does it say what I'm hoping it says, which is that ticker doesn't exist?
32:59Um, one of them is uh about Tesla, but the focus is yo is it a buy right now?
33:05Uh, because that is how real people send queries to agents.
33:08They do it in slang.
33:09Uh, and they don't often specify exactly what it is that they wanted.
33:14Um, so these are all, you know, realish looking queries.
33:21Um, and what you have to do is, like I said, you have to write down what good means before you measure it.
33:29You can't say that your agent doesn't work if you haven't specified what working means.
33:34So, for our analyst, uh, what does a good report look like?
33:39Um, it should use real current data.
33:41It should probably be about the right company.
33:43It should stick to what it actually found.
33:45Uh, and for my purposes, I want it to tell us what we should do.
33:50I want it to say buy or hold or sell this stock.
33:54I don't just want it to give me analysis.
33:57Um, those sorts of requirements sound obvious once you've written them down, but before that, they could just be vague or different people on your team could have different ideas
34:07about what a good financial an analyst agent would do.
34:10[snorts]
34:11Uh so writing them down turns vague disappointment into specific things that you can test and we are going to turn every single one of those requirements into an eval today.
34:21Defining good is not solely uh an engineering job.
34:26If you are defining good for your product, you have to talk to the non-technical stakeholders in your organization because they are the domain experts who are going to know what good looks like.
34:37You as an engineer are going to have specific guesses about what uh guesses and biases uh about what makes uh a good agent and what makes good output and you are not always going to agree with the domain experts.
34:51So uh the customerf facing people need to be in the room when you define what your agent looks like.
34:59Uh it's also great uh if you don't have users that you can make some up.
35:05uh you can get an LLM to generate lots of test queries and vary them on purpose.
35:09So research Tesla's financial performance is one way to ask what's going on with Tesla stock is another and yo is Tesla by right now is a third.
35:18Um then you can get a human to look at the list because LLM generated queries they are a useful way of seeding your initial data.
35:26They are not perfect u mostly because they are relatively conservative.
35:31They don't go as far a field as human users tend to actually go.
35:36Human users are like, you know, tell me what's wrong with, you know, my child's health instead of asking you for a financial analyst and you your agent has to have something to do with that.
35:46Um, but synthetic data is a great way to get started once you've got started and you have real users with real traffic.
35:53Real traffic is uh the gold standard for testing an agent.
35:57you want to get it incorporated into your data set as fast as you possibly can.
36:02Um, one aside before we look at the data that I'm hope hoping is arriving in your laptops right now.
36:08Um, I walked through tracing by hand because you should understand what it's doing, but you can, like I said, you can also get your coding agent to do all of this.
36:16Um, Phoenix has a really good CLI.
36:19Uh, and if you run it setup command in your project, it will give your coding agent um, it will get your coding agent to add tracing to your app.
36:27uh and it will work until your your app uh has tracing installed and you don't have to figure out how to add the register command and all of that stuff.
36:35It will work in every language.
36:37Uh it's really neat.
36:40So step two is reading your data.
36:43Who has any data at this point?
36:45Show hands.
36:46One person.
36:47Anybody else?
36:48Oh yeah.
36:48Cool.
36:49All right.
36:50Several people.
36:51Fantastic.
36:53Uh you have to read your traces before you write a single eval.
36:58Um reading a dozen or more of them end to end.
37:01Uh what was the input?
37:02What was the output?
37:03Um what specifically is wrong?
37:06This sounds manual and old-fashioned, but it's an incredibly high value high value thing that you can do.
37:12Um, when you're building an agent, uh, you have to manually inspect what is going on so that you can find things that you the unknown unknowns that you weren't expecting.
37:24Uh, Anthropics team regularly sits down and reads eval transcripts by hand uh, and have built a bunch of tooling internally to make that easy to do.
37:34[snorts]
37:34Um, if you automate before you understand how your agent fails, you're going to end up measuring what's easy to measure instead of what actually matters.
37:44Um, so you should read traces and we're about to do that.
37:47But reading traces doesn't scale.
37:49Reading 16 traces yourself is absolutely fine.
37:53Uh, but 1600 traces uh is going to give you a bunch of really great data and it's also going to take a week out of your life.
38:00Um, but it turns out that coding agents are aces at reading traces.
38:05Uh, so you can point your coding agent at your traces and get them to look for things that are weird in your traces.
38:11Get them to code stuff up.
38:13[snorts]
38:13Uh, but they're not always going to find what you expected.
38:17So manual inspection as a sampling technique always a really great idea.
38:24So, um, if you are using if you're getting the, uh, agent to do stuff for you, you want to install the Phoenix plugin or the Phoenix skill depending on which coding agent you're use you use.
38:38It was the last step in the setup readme if you've still got that.
38:42Uh, so you may have it already.
38:44Um, for cloud code, we have a plugin which is two commands.
38:47Uh, there are also plugins for cursor and codeex.
38:50Uh, and for any other coding agent, there is a skill file.
38:53Um, the plug-in connects your coding agent to your Phoenix.
38:57Uh, and it already points at localhost 606, so there's nothing to configure.
39:01Um, and Phoenix itself will hand your agent a skill called the Phoenix error analysis skill with instructions on how to do error analysis properly.
39:09Um, it should take about 10 seconds to install.
39:14Uh, and I'm not going to hang around waiting for you to install it.
39:18Um, now open your coding agent and ask it to do the error analysis.
39:26Uh, if you've got it.
39:30Who's got who's got the skill installed from previously?
39:33Fantastic.
39:34I'm going to pick on you again.
39:35Can you have you asked your coding agent to do things?
39:39Okay, great.
39:41Um, the prompt is on the slide.
39:43Tell it where Phoenix is.
39:44Tell it to read the traces.
39:45Come up with categories of failures.
39:47uh count them up and write each category back onto its trace in Phoenix as an annotation.
39:52An annotation is just a human readable string of text attached to a trace.
39:58Uh they are a way of adding additional information to traces so that either yourself or your agent can figure out what was going on.
40:07Um your your plug-in is going to ask permission to use Phoenix's tools.
40:11So you should say yes.
40:12Uh when I ran it, it took about 9 minutes.
40:16Uh, so I'm not going to pick on you quite yet.
40:19Um, uh, also if it asks to write notes into the repo, uh, that's fine.
40:26Just let it write stuff to disk.
40:30Now, let's look at Phoenix again.
40:35Um, I'm going to very dangerously try and remember my syntax for search.
41:00Ha.
41:01Okay, so these are all of my runs uh that are the financial reports.
41:06Uh we can click into individual ones of them.
41:08Let's see if we can find the interesting ones.
41:12Uh ZZQ, what was my output for ZZQ?
41:18ZZZQ does not appear to be a valid or active financial ticker symbol.
41:21Hooray.
41:21The agent did the thing that we were expecting it to do.
41:24Um how about uh Coca-Cola?
41:31First, it starts with the apology.
41:33It's running in restricted mode and it cannot write files directly.
41:35We're going to have to fix that definitely.
41:37Uh then it comes to a conclusion.
41:39Coca-Cola represents a high quality, lowrisk dividend investment.
41:43Um KO is an excellent long-term income vehicle for riskaverse investors.
41:49This one's going to be fun.
41:50Show of hands.
41:51Who thinks that this is an actionable report?
41:57[laughter]
41:57Okay, I primed you already.
41:59Uh it does not say buy, sell, or hold.
42:02So it just says that it's a good investment.
42:04That doesn't say what you should do.
42:06Uh I'm going to make my standard of actionability for these reports that it should tell me whether I should buy, sell, or hold the stock.
42:13I'm not going to leave it to be ambiguous.
42:16Um a whole bunch of them do different things in this set.
42:21Uh you can see this one had the same problem.
42:25It tried to write to disk and it was and it was and it failed.
42:29Um, uh, the Rivian trace, I made a note that Rivian was interesting.
42:39There it is.
42:44Uh, it's full of emojis for some reason.
42:46Um, it opens by asking, "Would you like to Yes.
42:52Would you like to create the report file?"
42:53We saw that earlier when I ran it live.
42:55Uh, it's asking me to do things and it can't ask.
42:58I told it I told it it's not allowed to and it's asking to do things anyway.
43:02Um, all of these things are uh different ways of failing.
43:10Lots of these are failures.
43:12Lots of these are sort of okay.
43:14Um, but then uh there's Tesla.
43:23Tesla is particularly fascinating.
43:25Uh specifically the yo is Tesla a buy right now one.
43:36So in this one uh it says that the analyst consensus is a buy.
43:43Uh it says 50% say buy, 52.6% 6% say hold and 13.1% say sell, which adds up to 116% of analysts saying things.
43:55And it did not catch that its math was completely wrong.
43:58What it did was it did two different web searches.
44:00One where it found 50% of people saying something, another one where it found 13% of people saying something.
44:06It took those two web searches and it munched them together even though they were not from the same data set and it came up with 116%.
44:14But it confidently wrote down that 116% of analysts say something.
44:18This is exactly the kind of agentic failure that you see.
44:22Uh it is confidently wrong having done uh bad math.
44:27Um if you go into these web searches, you can see the specific places where it got the data and how it uh put those things together.
44:37Um so when you find a failure, you have to ask why it failed.
44:43the answer was wrong is a symptom.
44:45It's not a cause.
44:46Um with the traces you can see what the actual answer was.
44:50Did it get bad search results?
44:52That would be a retrieval failure.
44:53That is not what happened here.
44:54This each search result was fine.
44:56Uh did it have the right data but the wrong conclusion?
44:59Then that is a reasoning error.
45:01That is uh definitely what happened here.
45:04Right?
45:04It saw two different bits of data and it did not reason correctly about them.
45:09Um sometimes it will make numbers up completely.
45:12that is a hallucination.
45:13Uh and sometimes it will answer a question that isn't the question you asked.
45:16So a scope problem.
45:18Each cause has a different fix.
45:20Uh but if you don't know the cause, you don't know the fix.
45:23Which is why reading traces is useful.
45:26Um there's a really great way to do this which is let's switch back to slides.
45:35Uh you should write notes first and categories second.
45:38So look through all of your traces and just write down in human language what you see happening.
45:42So this is bad because it does this.
45:45This is good because it does that.
45:47Uh that is called open coding.
45:50Uh another term borrowed from ML.
45:52You're just saying uh what you saw.
45:55Once you've done that uh you can then do what is called axial coding where you look through all of the notes you've written to yourself and you group them into categories.
46:04It is very very tempting as an engineer to g try and guess in advance what the categories are going to be and come up with categories and just put things in categories from the get-go
46:14uh because that is less work but you will do worse at it because you will not know in advance what your failure nodes are going to be.
46:21Uh so the categories you're going to pick are going to be either too narrow or too broad or just not what not the kind of stuff that actually went wrong.
46:29[snorts]
46:29Um, while we're talking about this, I talked about how you stack up uh um Kod valves and LLM judgy valves and human evals.
46:44Uh this is uh a model barred from the security world called the Swiss cheese model.
46:50Um every layer of defense that you have is going to have holes in it.
46:56your your every eval you have is going to have a blind spot.
46:59What you want to do is uh stack as many eval as you can on top of each other such that it is very hard for an error to get all the way through all of them.
47:10Um because they've all got holes but the holes are not in the same place.
47:15So if you can uh if you have enough evals, if you have enough different types of check going on, they will stack up on top of each other and be much more reliable.
47:27Um so code evals can catch formatting problems, but they will miss problems with meaning.
47:32LM judges will catch reasoning gaps, but they will miss some subtle hallucinations.
47:36Humans will often catch the subtle stuff, but they can't look at every single trace.
47:40So no single method uh is going to catch everything.
47:44Um, so today we're building several small evals, not one giant super eval, which is also another failure mode.
47:51When people first get into evals, they're like, I'm going to think of every possible thing that can go wrong and I'm going to cram it into one big LLM as a judge, and it's going to find everything.
48:00And that is uh a very unreliable and very complicated way of not solving the problem.
48:08So, whoops.
48:14Uh so the agent may call uh the agent is going to do a whole bunch of stuff.
48:26This is what the agent output looks like.
48:28I showed you already in Phoenix.
48:30Um we've I've already showed you uh the various things that uh it came up with.
48:39Um, in this case, uh, my coding agent looked through all of my traces for me.
48:46It came up with, uh, both, uh, open coding and aial coding, uh, and it came up with a category, uh, called harness leakage.
48:57Uh, harness leakage is what it decided to call that thing where it's constantly asking us to give it permission to do things that it doesn't have permission to do.
49:07Um, so, uh, that's a ton of that's a ton of times for that specific thing to happen.
49:16[snorts]
49:16Um, there are also a bunch it also found a bunch of other things.
49:19It found hallucination.
49:20It found uh, reasoning errors like the Tesla one where it had too many it added up percentages.
49:26Um if you're doing if you've run this data yourself and you've come up with your coding agent has uh produced its own analysis, you're going to get a different analysis and a different set of uh labels.
49:38Uh because all of our agents are different.
49:41Who has got their coding agent to complete an analysis yet?
49:45Anybody still running?
49:48All right.
49:49Well, we'll keep going.
49:50Hopefully, you'll get some, but if not, you can just use my seed data.
49:56Uh once you've got data, you can count the categories.
49:59The most common failure is going to be where you look first, but the most common failure is not necessarily the first thing that you're going to fix.
50:06What you want to do is address uh frequency and severity.
50:11So if uh you know 40% of the time it hallucinates, that's a problem.
50:16If 1% of the time it gives your user instructions on how to build a bomb, you fix that immediately even though it happens only 1% of the time because that's absolutely not what a financial analyst is supposed to be doing.
50:28Uh so you multiply frequency by severity to get your priority.
50:32All right, so hopefully your agents will catch up to where we are, but I'm going to keep going.
50:37Step three is Kodi valves.
50:39Kodi valves are the simple ones.
50:41There's no model, there's no API call, there's just TypeScript.
50:45Uh the simplest useful eval that I could think of was does the report actually mention the stock that we asked about.
50:52Our agent is supposed to analyze a specific company.
50:54If it's doing that, it should mention the ticker symbol of the company that it is analyzing.
50:59Uh if I ask about Tesla and it writes about Microsoft, which is a thing that it really did on at least one occasion, um that is a completely useless report.
51:08So either the ticker report, the ticker is in the report or it isn't.
51:11and using an LLM to check that would be complete overkill.
51:15Uh so to run that check, the first thing that we need to do do is get our data out of Phoenix so that we can run the analysis.
51:23This is how that is done.
51:25Um we ask for spans for our project and parent ID null means that only spans that have no parent.
51:32Um so those are the top level spans.
51:34So the ones that I was showing you in my traces.
51:37Um, the repo runs this in a little helper called get agent runs, which hands back each run span ID, its input, and its output, plus the research turns output,
51:46uh, which we're going to need later.
51:49Uh, and then this is the ticker check.
51:52Um, create evaluator comes from the Phoenix evals package.
51:56You can just give it an ordinary function.
51:57You don't have to build an evaluator function yourself.
52:00Uh, all you have to do is tell it that this is a code eval so that no LLM is involved.
52:05The function gets the input and the output.
52:06It uh splits out the tickers we asked about.
52:09Uh it checks that each one appears in the report.
52:13Uh and it returns a label and a score.
52:15And if something's missing, it also it also says which ticker.
52:18So the failure explains itself.
52:21Um the structured inputs that I talked about earlier, how we the input being sent uh to our tool was JSON, that is helping us out here because it's making it really easy to write this Kodi val because it's already JSON.
52:33So we already know uh which thing is the ticker and which thing is the uh description of the uh report that it should be writing.
52:42[snorts]
52:42Um so if you can get uh your inputs to be structured data that's going to make your evals really easy to work with.
52:50But if not you're going to have to get another LLM involved to read your input figure out what the input was and then pass it to your eval.
52:58Uh then we run it and we log the results back to Phoenix.
53:01So the script runs the evaluator on every agent run and then it calls log span annotations which attaches each result to its span.
53:09Um that's going to be the same pattern for every single eval we run today.
53:13You get the runs uh you evaluate them and then you log the results back to Phoenix.
53:18Uh so uh the command here is npm run eval code.
53:24Let me see if I can get that to work.
53:39All right, it worked and it ran almost instantly because it's Kod valves.
53:43They don't have to wait for an LLM to do anything.
53:44They just go
53:45[snorts]
53:46uh and it has reported 15 fastes and two failures.
53:51One is Tesla is missing Tesla and one is Amazon is missing Amazon.
53:55Uh so uh this is the dumbest possible check that we could have made.
54:02Did the LLM actually write a report about the thing that we asked it about?
54:06Write a report about and it has already found some errors.
54:09It has found cases uh where it wrote about the wrong thing.
54:13I know for instance that the am the problem with Amazon is uh that it asked it about AWS profitability and the agent got focused on AWS and wrote a report that was only about AWS and what AWS was doing
54:27uh and not about what Amazon, you know, the shopping company does.
54:32Um so this is a test worth doing and it cost you absolutely nothing to run.
54:39So Kod valves aren't toys.
54:40So, did the output parses JSON?
54:42Is it under the length limit?
54:44Does it avoid phrases you never want to see like as an AI language model?
54:48Um, and it doesn't have to be a string check.
54:51Kodval just means that the grading is deterministic.
54:54So, uh, your Kodval can be querying your database to make sure that the state is correct.
54:59It can make sure that the tool calls did what the tool calls were supposed to do.
55:03uh it can call an API and double check that the stock price uh that the agent thinks is happening is really the stock price.
55:11Um anything that gives the same answer for the same input every time can become a Kodi val.
55:16So your Kodi valves can be quite advanced.
55:19Um but one thing you should be wary of is grading the outcome and not the path.
55:24Uh there's a tempting mistake here um to check that the agent took particular steps.
55:29Did it call the search tool first and then this other tool?
55:33Uh, Anthropic advises that this is too rigid an approach.
55:37Uh, like we saw earlier, the agent can be creative.
55:41It can do things in an unexpected way.
55:43You don't want to assume that it's always going to do exactly what you would have done.
55:48Uh, so you don't want to grade the path that it took.
55:51You want to grade whether or not the outcome was good.
55:55um our ticket check doesn't care how care how the agent found its data right it just cares that the right company is in the report and that is the right level for a lot of evals we will talk later about
56:06trajectory level evals where we do actually want to grade the path but that is a very different uh kettle of fish so that brings us to step four the LLM as a judge
56:18uh and we're going to start with Phoenix's builtin evals Um code can't check whether code can check whether the ticker is in the report but it can't tell you if the analysis is any good.
56:30Uh are the numbers accurate?
56:31Is the report report complete?
56:33Is the recommendation well reasoned?
56:35These are questions about the meaning of the report not the textual content of the report.
56:39And for those you need uh something that has judgment.
56:43Uh an LLM judge has three parts.
56:45It has a judge model which is the LM that is doing the judging.
56:48Uh it has a rubric which is the prompt that tells it what good looks like.
56:52And then it has the data which is the examples that it's grading.
56:55Phoenix keeps those three separate so you can mix and match.
56:58You can try the same rules with different models.
57:00You can try the same judge with different rules.
57:03Uh the good news for you is that Phoenix ships a bunch of built-in evals.
57:08So you don't always have to write your own evals, at least not right off the bat.
57:12Uh there's correctness which checks if the answer is accurate.
57:15Uh there's faithfulness which checks if it sticks to the source material.
57:20Uh we're going to see the difference between those two in a second.
57:22Uh there's completeness, hallucination, refusal, PII, and toxicity.
57:26Lots of things that lots of agents need to check are built in.
57:30Uh and for agents, there are evals for tool use.
57:34Did it pick the right tool?
57:35Did it pass the right arguments?
57:37Um did it do something sensible with with the result?
57:41Uh these eval are not magical.
57:45They are just uh prompts that we've already written and battle tested and made sure work for you.
57:52Uh so let's find the judge model.
57:57Um uh Phoenix evals in Typescript uh use the Versel AI SDK to talk to models.
58:06So any model that the AI SDK supports can be a judge.
58:09Uh in this case I'm using Claude Sonnet 5.5. Um, the agent runs on Haiku.
58:16Like I said, it's cheap and fast.
58:18Our judge is running on Sonnet because you want your judge to be more capable than the agent itself was.
58:23So, give it Sonnet or give it Opus.
58:25Uh, or give it uh even better um give it a model from uh a different provider entirely.
58:33If you've got an OpenAI key, get OpenAI to judge the work of Claude.
58:36Uh because Claude has what is known as uh same model bias.
58:41So Claude tends to like the output of other versions of Claude uh more than it does the output of uh of OpenAI or some other provider.
58:50Um the last line here uh in case you're wondering is suppressing tracing on the judge because we turned on tracing across the board.
59:00The judge is itself a copy of Claude.
59:02Uh so by default and in fact in my test data you will see that I forgot to turn on uh uh telemetry suppression.
59:12So my traces are full of the the judge running itself and and seeing all of the traces from the judge which you don't need.
59:20Um so every evaluator in this repo gets it gets this telemetry setting uh so that the judge stays out of uh our agent.
59:31Um let's start with correctness.
59:34Uh which is the obvious first choice.
59:36Is the uh is this report correct?
59:39Um that is one import and one line of setup.
59:42You don't have to do anything.
59:43You just say hey I'm running the correctness evaluator.
59:46Run it on my stuff and let's see how it goes.
59:48Uh then you call evaluate and you get back a label, a score, and an explanation.
59:53And the script runs it on every agent run and logs the results back to Phoenix.
59:57Uh the same way we did the ticker check.
60:00Uh, I should be able to run that as well.
60:12That's going to take a little while because it has to actually do some LLM stuff.
60:16Uh, if it works, it works.
60:19If it doesn't, I have stuff already built in.
60:26So you can see uh in my annotations column you can see the results of various evaluations I've already run.
60:33In this case it's showing you the uh actionability one but you can see uh correctness, faithfulness, human actionable uh and mention sticker.
60:42We're going to talk about what all of those are.
60:43But first you're going to see correctness.
60:45Uh correctness on all of these is zero.
60:51Uh, all 16 reports have been judged as incorrect.
60:57And the reason that they've been judged as incorrect is because Claude judges correctness by what it knows.
61:04That's the only thing that it has available to it.
61:06And Claude's knowledge cut off uh at the moment is in something like January.
61:12So uh my agent is talking about stock prices in October and Claude is freaking out and going, "You're talking about stuff that's in the future.
61:19That's obviously a hallucination.
61:21This report cannot possibly possibly correct.
61:24So our built-in check for correctness fails every single report for that reason.
61:29Um which is not terribly useful.
61:32Um so the eval is half wrong and half right.
61:37Seven of in my data, seven of the 16 explanations complain that the judge can't verify fresh data.
61:43Uh which is the eval definitely being wrong.
61:46Um, but lots of them also point out something real.
61:49Things like the numbers don't add up or dates that contradict each other.
61:52Um, so a score of zero is telling us nothing here.
61:55It's telling us across the board that this that this eval is probably not the right choice of eval.
62:01Um, what we want is a question that the judge uh can actually answer.
62:08So uh oh, I told you that already.
62:14All right.
62:15uh what we're going to use instead is faithfulness.
62:17Faithfulness is asking a much better question.
62:19It's saying given the data that you had, did your did your report make sense?
62:24Did is the answer based in the data that we were given?
62:28This works much better because it can work with data this exactly the same data that the agent was given and see if it gets a report that makes sense.
62:36Uh so that's a question the judge can actually answer.
62:40Uh and the context is already in the trace.
62:42Every web search is its own span like we saw.
62:46Um, every research turns output is the agent's own notes on what it found.
62:50So my first version, my first eval uh only gave the judge uh the research notes and then I read the explanations and the judge was flagging numbers as made up
63:00uh because my version of the eval hadn't told it that oh I I searched the web and actually found this data.
63:06Uh so uh in the version you see we've given it the full context.
63:11um our get agent runs helper is going to pull both uh the raw search results and the notes so that it can see uh what the research agent found and how it found that data.
63:27Uh so like I said faithfulness takes three things.
63:29It takes the input which is what the user asked.
63:31It takes the output which is the report and it takes the context which is the search results and the notes that I was just talking about.
63:37[snorts]
63:37Uh and you run it using the same pattern as before.
63:40You create the evaluator, you evaluate every run, you log the results.
63:44Um, like I said, it's slower than correctness uh because the judge is reading every search.
63:52Uh, who has got any results out of their uh eval so far?
63:58Again, you've got a faster Wi-Fi connection than everybody else.
64:01Anybody else?
64:04Cool.
64:06Um, so let's look in Phoenix at my faithfulness reports.
64:12Um, I got a lot of zeros on faithfulness, but not all of them.
64:22Um, let's read the Microsoft explanation.
64:26That was the one that I read that was most interesting.
64:30Uh, as you can see, next to each one, you get the explanation.
64:34Um, so in this one, the response adds several claims that the context does not support.
64:39Azure is the number two cloud provider.
64:41The backlog gives three years of visibility and there are implied FY202 targets.
64:47Uh, it is telling me what it doesn't like about uh this thing.
64:54It does it's telling me why it thinks this thing is unfaithful.
64:58Um, it's complaining that uh some of the 2027 targets are fabricated because they're not in any search results.
65:08Uh, the Nvidia one is also interesting if I can find that one.
65:20Right.
65:20In this one, it says the response conflates two different data points.
65:23This is a reasoning error that the agent that the judge has uh accurately found.
65:28Um it says the context information this is fabricated.
65:34These are much more useful results.
65:36We've got uh 14 out of 16 uh reports marked as unfaithful here, but the explanations are way more useful.
65:43Uh and it's actually doing the thing that we wanted it to do.
65:47Um so for agents, you can also grade the tool calls.
65:52Uh so we saw the web search spans inside each trace.
65:56Phoenix has built-in evals for all of those as well.
65:59Tool selection asks did the agent pick the right tool uh or search when it didn't need to.
66:05Tool invocation asks did it pass sensible arguments to those tools like a search query that's actually about the company.
66:11And tool response handling asks did it do something sensible uh with the result of that call.
66:17Uh these are the decisions that can make a break make or break an agent.
66:22Um, and they all happen before the report actually gets read.
66:26Uh, and we already know our agent makes strange tool choices sometimes, like it tries to save the report to a file.
66:32Uh, so we can see if our built-in eval catches that.
66:36Uh, I'm not going to run this one, but you can run that one.
66:39Um, tool selection takes three things.
66:42What the agent was asked, which tools it could have used, and which ones it picked.
66:45Uh and the script goes through every agent turn and every trace research and writing separately and collects the tool spans underneath each one.
66:52Um the trace records which tools the agent called but not which ones it could have called.
66:58So we have to write that list ourselves uh in tool selection.ts.
67:03Uh then it is uh the same pattern as every other run.
67:08Uh so npm run eval tools uh and then you're done.
67:12Uh that one should take about 15 seconds.
67:17So, let's look in Phoenix at tool selection.
67:38Tool selection.
67:39It's helping me out.
67:44All right.
67:45So, these are ones where it has decided that my tool selection was incorrect for whatever reason.
67:50Uh, the user asked for a concise report based on prior research.
67:53Writing it to a file with write is plausible and appropriate since write exists and is safe.
67:59Uh, however, the LLM made a second redundant bash call.
68:03Awesome.
68:04Uh, yeah, it sure did.
68:06Um, the judge failed six that tried twice.
68:12It passed seven.
68:13Six of the seven that tried once.
68:15Uh uh overall 13 of my 16 writing turns tried to save a file.
68:21Um and only one of these explanations said that the user never asked for a file to be saved.
68:30So uh is this judge incorrect?
68:34Is the judge uh accurately judging that the tool call was correct?
68:39Um, our writing prompt said, "Write a concise financial report based on your research above."
68:46Our prompt is genuinely ambiguous here.
68:48We said, "Write, we didn't say where to write it to.
68:50We didn't say what to write it for.
68:52Uh, we didn't say who the audience was going to be."
68:55So, the tool selection knows that and is looking at the input and is going, "Well, it seems a reasonable thing for the agent to try and write to disk here."
69:03Uh so again what you're telling the agent to do and what you're telling the eval to do interact with each other.
69:12Uh you can't necessarily trust that the tool call invocation is going to be right because in this case the tool call is citing the tool call eval is siding with the agent.
69:20It's like no the the agent is right.
69:22The agent gave us instructions uh the agent received instructions and did its best is basically the eval's position.
69:30Uh, eval results live in the trace as well.
69:37Um, so if you're looking at the traces in Phoenix, uh, you can see them here attached to which trace they got, which span in the trace they got attached to.
69:48So you see actionability and correctness got attached to the root.
69:51Uh, whereas tool selection is getting attached to the specific uh, turn that made that tool call.
69:58Um, you can filter your traces by any eval.
70:02Uh, you can sort by score to find your worst runs.
70:04You can click straight from a failing score into the step that caused it.
70:08Uh, from score to from score to cause in one click.
70:11Uh, so all of that is the built-in eval.
70:15Um, and the lesson here is that choosing the right eval beats tuning it.
70:20Choosing correctness was entirely the wrong eval.
70:23It didn't tell us anything.
70:24It just told us a bunch of stuff was wrong.
70:26tool selection did tell us something uh and faithfulness told us quite a lot.
70:31Um so checking the right eval beats tuning it uh and knowing which question you're actually asking is really the core of getting this right which brings us to step five which is writing your own judge.
70:45This is where the eval get specific to your application.
70:49Uh, and again, this is when people are thinking about evals, this is what most people are thinking about, but I wanted to make sure that you got the basics in first.
70:57You don't have to jump straight to writing a custom rubric.
71:00You can write code evals to get a bunch of stuff out of the way.
71:03You can write uh you can use built-in eval to do a bunch of obvious stuff without having to invent your own tool calling correctness uh uh evaluator.
71:13Um, what I wanted my report to do was give me a buy, sell, or hold recommendation.
71:20There is no built-in eval that checks for that because that would be a very specific thing indeed.
71:24So, we're going to write our own.
71:27Uh, every eval prompt I've seen that works well has five different parts and we're going to go through them one at a time.
71:34The first part is you give the judge a role and context.
71:38Uh, a thing you'll see all the time is you are an expert blah blah blah blah blah.
71:43That does actually work.
71:45That's why everybody puts it in there.
71:46Like it helps a little bit to tell the to tell the agent what it is.
71:51Uh, telling it it's an expert as opposed to just telling it that it is running up something else also seems to make a difference when they do sideby-side comparisons.
72:00Um, it's just one of the weird things about LLMs.
72:03You need to tell it it's really smart and it gets like confident and tries.
72:07Um uh but what really matters is the context.
72:11Um this is a financial report.
72:14Uh we are expecting a report that that analyzes financial data and tells us real things.
72:20Um and specifically what we care about is whether or not you could act on it.
72:25So uh that gives us the explicit criteria that we include in our rubric.
72:30Um so this is where a lot of people don't put in enough effort.
72:34So don't write a good response is helpful and accurate because uh a judge doesn't know what helpful and it doesn't know what accurate means in this context.
72:43Um a judge can't make a yes or no decision from that.
72:47Write down exactly what makes a report actionable.
72:50Uh so it makes a specific recommendation.
72:52It names concrete risks and it looks forward not just back.
72:56Those are things that a judge can look at and actually check.
73:00Uh, and note that we're not just making up these things blind.
73:05Um, your criteria come from your error analysis.
73:08Uh, only summarizes data with no interpretation is a problem that we saw when we were reading our traces.
73:14We are putting that in there because it's a problem that we are trying to avoid.
73:18Um, that is the whole point of reading the data first.
73:21Specific criteria give you consistent judgments and you get specific criteria by knowing in advance what the data is going to look like because you read it yourself.
73:31Part three of a good rubric is to mark the data clearly.
73:34Wrap each piece of data in its own XML tag.
73:37Claude was trained on billions and billions of XML tags and it's really good at working with them.
73:42Um, if you look into the code at our own rubrics, that's what we're doing.
73:48uh our own prompts
73:49[snorts]
73:49uh wrap everything in XML tags.
73:52Um and the double curly braces are template variables that Phoenix fills in uh with each span's input and outputs.
73:59Uh the reason you do this is because you want clear boundaries between what is the input, what is the output, and what is your instructions.
74:06Uh this helps against prompt injection, but it also helps uh the agent just getting confused.
74:12uh in particular it helps you get con uh if you include examples which we're going to see mark the example as an example and not the data otherwise it will get very confused about that
74:23um I love to include labeled examples but it is in fact a controversial proc uh a controversial practice uh in the eval world um criteria tell the judge what you mean in the abstract but examples show it exactly what you mean as a human, I find it much easier to do something when somebody has shown me an
74:44example of how it's supposed to work than if they've just given me a list of instructions of how to work.
74:50Uh, and, uh, a lot of LLMs feel that way as well, apparently.
74:55Um, so when you test your rubric, uh, which we're going to do in the next step, uh, test it with examples and without examples and see which one does better.
75:07Uh, because sometimes, uh, examples are going to help it do a better job.
75:11And sometimes, uh, an example will make it overfit.
75:15It will look at your examples and say, well, it doesn't look exactly like my example.
75:19Or it will try and make every output look like your output example.
75:22Uh, which is less helpful.
75:24Um, I ran exactly that test on mine.
75:27Uh, so in the rubric that I used, um, this was fascinating in a really nerdy way, which is why I included it.
75:35Uh, I threw out my examples and with the examples, the judge agreed with me, uh, 15, 15, and 15 times out of 16 in three different runs.
75:44Uh, and if I threw out the examples, it agreed with me 16 times twice, all 16 examples, and 15 times once.
75:52Uh and so I was like I was very curious as to how throwing out the examples made it better.
75:58Uh and the specific problem that it was having is that in one of the reports it did not include um uh it did not include a list of risks.
76:10So as a human I read the report and it was like Microsoft is a buy and it didn't include any risks and I was like that's fine.
76:17It's still actionable.
76:18It included a buy.
76:19Uh so I marked it as uh an actionable report.
76:23Uh but my examples said you must include risks.
76:27And so the agent uh the LLM judge was looking at the rules and and looking at the examples and saying there must be risks in here.
76:34So this is not actionable because there's no risks.
76:37So that's genuinely an open question as to whether or not my human judgment was better or worse than the judges.
76:44The judge was being very strict and I was being loosey goosey when I as a human gave myself golden data.
76:51Sorry, can I ask a question?
76:53Sure.
76:53So what's your hypothesis then as to without the examples it agreed with you so it saw that Microsoft as being actionable?
77:03Yeah.
77:03Is that is that your hypothesis then that like simply because you didn't say risk was a necessary element it felt that that was a good actionable?
77:12I mean, it's not a hypothesis like the LLM judge comes with an explanation.
77:16So, it says
77:17So, it said that that was actionable,
77:19right?
77:22Yeah.
77:22So, like I said, in this case, it's genuinely unclear whether or not I was doing the right thing or the judge was doing the right thing.
77:30The judge decided to interpret the the rubric very strictly and I when I was marking these things as human actionable or not, uh was not being as strict.
77:41Um, here's an example of an actionable one.
77:46Uh, it has specific numbers.
77:48It has revenue growth.
77:49Uh, it has valuation compared to its sector.
77:51It names a concrete risk.
77:52And it ends with a recommendation you could actually follow, including when you'd change your mind.
77:57Um, and this is what a not actionable one looks like.
78:00It isn't wrong.
78:01Everything in it is is true.
78:03It's just not very useless.
78:04Investors should consider various faction uh factors is not a recommendation.
78:09Um, now that the judge has seen both, the hope is that it's more consistent on reports that fall somewhere in between.
78:16Uh, and whether it actually is, you find out by testing.
78:20Uh, part five of putting together a rubric is you constrain the answer.
78:24You ask it for things that are actionable or not actionable.
78:27A thing you used to have to do is you used to have to say output a string and the string must be the word actionable or not actionable.
78:34Uh, you don't have to do that anymore.
78:36what uh modern evaluators use tool calls uh to put their output.
78:42So those uh the choices available for that tool call are defined in the tool call and not in your rubric.
78:50Uh which makes them a lot more reliable.
78:54So now it is your turn to write the rubric.
78:56Uh open evalactionability.ts.
79:00Uh it has my rubric in it with all five parts.
79:03Um and you can make it yours.
79:04You can change the criteria.
79:06You can change the examples.
79:08Uh what would make you act on one of these reports?
79:11What did you see in your own traces?
79:13Write that down and that should become your rubric.
79:16Um it will take three or four minutes to run.
79:19Who has managed to stay up with me so far?
79:22Like who is at this point?
79:24Excellent.
79:25I have multiple hands.
79:27Uh like I said, the last time I did this workshop, everyone was completely lost because the 200 megabyte repo.
79:34So the fact that anyone's got it this is just great for me.
79:38Um this is how you wire it up.
79:41You create a classification evaluator which does all the plumbing.
79:44Uh you give it a name, you give it the model.
79:46Uh you give it your template.
79:50Uh and like I said uh out of band you give it its out its options.
79:54So you give it its choices which are actionable with a score of one and not actionable with a score of zero.
80:00Um, so let's look at actionability in Phoenix.
80:06[snorts]
80:22It's so cold in here that my fingers have gone all numb and I can't type properly.
80:28Okay.
80:36So, the report analyzes JPM's rate exposure and loan growth uh and a forward-looking output, but makes no buy, hold, sell recommendations.
80:44Fantastic.
80:44It's doing exactly what it's supposed to do.
80:46Uh it is complaining about uh the lack of recommendation of concrete recommendations.
80:52Um before we look at all of these recommendations, we should look at the split in the recommendations.
81:04Uh in my data, five of 16 reports were actionable and 11 were counted as not actionable.
81:11Uh and that is one of the most useful types of evals you can get.
81:15An eval where everything is perfect is not particularly helpful.
81:19An EVA where everything is wrong is also equally unhelpful.
81:22An EVA where you've got an even split that is a hill that you can climb.
81:26You've suddenly got uh some you've got a def a set of examples where things are actually working, a set of things where uh stuff isn't working and you can start figuring out the differences between them.
81:39Um so if you click through these traces, I'm not going to bother clicking through all these traces.
81:47In Rivian it says it never makes a buy, hold, sell recommendation or any equivalent next step.
81:52For Nvidia, it said the closing investment thesis risk level is moderate, is vague rating, not guidance.
81:58And uh for Meta, the verdict is the is only that the position is solid.
82:03So this is the same problem over and over.
82:05It is refusing to give concrete buy, sell, hold recommendations.
82:10Um, if we uh we can open the Phoenix prompt template.
82:19I'm not going to do it now.
82:20Um, but uh there's one more rule for rubrics, which is that you should treat eval prompts as code.
82:28Your eval prompts going into your into your custom eval are extremely sensitive to small changes in wording.
82:36Um, so, uh, one example of that I ran into while making this was, uh, I told you that you should put XML tags around stuff.
82:45I forgot to put XML tags around my examples to begin with.
82:49Uh, and with my examples as plain text, uh, I got a string of not actionable stuff, uh, because it was getting confused about what the data was and what my examples were.
83:00And when I wrapped them in XML tags, everything suddenly became actionable because it could tell what I was talking about.
83:07Um, this is there was no other change.
83:11I'd given it the same data, the same instructions, the same input and the same output every time.
83:16Uh, it just got less confused in the evaluator as to what was happening.
83:21Uh, so you have to treat your evals as code.
83:25You should version them, you should check them, and you should test them.
83:29Um, as I mentioned earlier, uh, there is a trap to fall into with evaluators, which is to try and write write one evaluator that does everything for you.
83:38It is called the god evaluator because it is such a a common pattern.
83:42Uh, you try and get it to check accuracy, tone, completeness, formatting, policy all in a single prompt.
83:48Uh, and it is a nightmare because if it fails, what thing failed?
83:52Was it inaccurate?
83:53Was the the tone wrong?
83:54Was the faithfulness wrong?
83:55which of the many many many things that you've told this god evaluator to evaluate is it actually wrong and you try and solve that by giving it 15 labels
84:04that it can spit out uh and the whole thing becomes very hard to track.
84:08Uh the best practice is to split this up split your evaluators into as many dimensions as you are testing.
84:15So one for faithfulness, one for correctness, one for actionability.
84:19Um, and you should also know which one of your evals is a guardrail and which one is a north star.
84:25So, a guardrail is something that should block the release.
84:28So, if the agent is making up a stock price, that should halt shipping.
84:33Uh, but north stars are things that you're aiming for.
84:36So, suggests other stocks worth a look would be nice to have.
84:40That would be a good thing to include.
84:41Your eval can look for that, but it doesn't have to gate shipping on that.
84:47Which brings us to step six, which is testing your tests.
84:50We've built a judge and it says that some of our reports aren't actionable.
84:53Uh but can we trust the judge?
84:56How do we know that the judge is right?
84:58Uh the trick to remember is that your judge is a classifier in the classic ML sense.
85:04Uh it is looking at a report and it is predicting a label as actionable or not actionable.
85:08Predictions uh can be right or wrong.
85:11So like any classifier, you measure it by comparing its predictions with the truth.
85:16Uh, and here the truth is what a human expert would say.
85:19You get to judge uh you get to mark the judge's homework.
85:23Uh, in this case, the human expert is going to be me.
85:26I'm going to be deciding whether or not stocks are buy, sell, or hold.
85:30Uh, not necessarily the expert that you want.
85:34Uh, but we're doing our best.
85:36It's just a demo.
85:37Um, how you do this is very simple.
85:41You open up a trace.
85:44uh you click the little pencil button and you can add annotations.
85:49So I have created an annotation in advance called human actionable uh and I can set it to actionable or not actionable and I can very quickly uh click through all of my traces
86:01and set that on each one which I've done in advance.
86:05Uh this is where you would get uh a domain expert involved.
86:09You would throw them at Phoenix.
86:10You'd say here are 10 traces, here are 20 traces.
86:13there's 50 traces.
86:14Go through and read the output and give me give me your golden data set about whether or not these things are good.
86:20Um,
86:24[snorts]
86:25I sat down and labeled all 16 of my reports properly against the same criteria that I gave the judge.
86:30Uh, and those are sitting in your seed project so that you don't have to do that because that would take you a long time.
86:36Um, what you're building when you do this, like I said, is called a golden data set.
86:40Uh it's examples where to the best of human knowledge you know that the answers are right.
86:45Um it is a lot of work so you should make that work count.
86:50So you should give yourself the same specific criteria that you gave the judge.
86:54That was the error that we were talking about earlier.
86:57Uh where I was not exactly reading the rules that I had put into my eval criteria.
87:03Right?
87:03I was disobeying one of my own rules.
87:05my rule said uh must include risks and I said that this report is fine even though it doesn't include risks.
87:13Uh and as a result me and the judge disagreed and I feel like the judge was actually more correct than I was.
87:18It was being stricter.
87:19Um so uh your enemy here is fatigue.
87:25Uh if you give yourself a hundred traces to judge by trace 50, you're going to be really bored of judging traces and you're going to get lazier about it.
87:34Uh so it is better to have 50 really well-labeled uh examples than a hundred where you were just clicking through and saying ah this looks fine.
87:44Uh the the higher the quality the data in your golden data set uh the more reliable everything is going to be.
87:52Um and one of the best things that you can do is that uh is continuously add to your golden data set.
87:58when you see a tricky example, when you see something strange happen in production, figure out what the answer should have been and put it into your golden data set.
88:06Uh, a few good rules for human go for a good golden data set is that you should write test tasks a human could pass.
88:13Uh, if your agent fails a task every single time, that is almost always going to be a broken task, not a broken agent.
88:20Um, and for each task, you should have an answer that you know is good, which proves the task can be done.
88:26Uh, you should also test in both directions.
88:28If you only check, for instance, that the agent searches the web when it's supposed to search the web, you're going to accidentally train your agent to search the web every single time, all the time, even when it doesn't need to.
88:40Uh, and here's a rule from machine learning that applies to judges as well.
88:43Don't tune on your test set.
88:45Uh, split your labels into two groups.
88:47One that you use to tune your evals and one that you use to test that they are working.
88:54Uh when you only have 16 answers in your golden data set, that makes it your test set very small.
88:59Uh because the rule is usually 75% in the tuning set, 25% in the test set.
89:04So there's only four in our test set.
89:06Uh but if you were up to 100, that would be fine.
89:10Um Phoenix data sets have splits built in.
89:13So this is very easy to do.
89:16Uh see what that looks like.
89:20uh so first it builds the golden data set with splits.
89:25It finds every report with a human label.
89:28Uh uh and each one becomes an example in a Phoenix data set.
89:32The input is the report.
89:33The expected output is the human's label.
89:36We sort them by span ID.
89:37Then every fourth example goes into the test split and the rest go into dev.
89:42Uh sorting first means that a report always lands in the same split however many times you run this.
89:47Uh, and span
89:48[snorts]
89:49ID links each example back to the trace it came from.
89:51Uh, so you can always click through to the original run.
89:57Then we run the judge against the humans.
89:59This is an experiment which is something that we're going to dig into in a little bit.
90:03Um, but for now uh an experiment takes a data set.
90:06It takes a task task to run on every example uh and evaluators to score the results.
90:12The task here is our judge and the evaluator is a tiny Kodi val which checks whether the judge's label matches the humans.
90:18Uh notice that we only run it in the devs split.
90:22Uh you can run this yourself by running npm run meta.
90:28[sighs]
90:30Let's see if we can get it to go.
90:44It's going to run an LM.
90:45So, uh, I'm going to assume that it does the same thing that it did last night.
90:50Um, the script should print a little table at the end.
90:53Uh, and on my 12 dev reports, the judge and I made the same call every time, which is great, but it's also only 12 reports.
91:00Uh, so read that as, you know, no sign of a problem yet.
91:04you're going to want a larger test set when you're running this in production.
91:08Um, cool.
91:11It actually completed it.
91:13Uh, the more interesting part is the two ports that I the two reports that I found hardest to label.
91:20Um, so in my Phoenix data, uh, there's Coca-Cola and there's Microsoft.
91:38Um the the first one is Coca-Cola.
91:43The Coca-Cola is the one that flipped uh when I changed from the XML tags to the uh from not having XML tags to having XML tags.
91:52Uh this is where when it got the examples it got better.
91:55Um borderline for me turns out to be uh borderline as well.
92:01the it the report calls KO ideal for conservative income oriented pol uh portfolios uh which sounds like a recommendation.
92:09So as a human I went back and forth on whether or not I should count that as a recommendation.
92:14Ultimately I decided it did not uh but the judge also goes back and forth on whether or not that counts as enough of a recommendation.
92:21Um and the second one was Microsoft uh which I'm not going to bother to click into which is the other way around.
92:27The report says buy with reasons uh and a time horizon.
92:31Uh and it never mentions a single risk.
92:33This is the one that I was talking about earlier.
92:35Uh my rubric lists name concrete risks as a criterion.
92:40Uh and I decided that was fine and the judge decided that that was not fine.
92:45It said it was weak on risks.
92:46There is no risk section.
92:47It clears the bar for actionability but though not by much.
92:51So, the judge looking at this stuff came up with exactly my human judgment on whether or not this was an actionable report, including the ambiguity.
93:02Um, and that ambiguity is still in our rubric because we haven't changed it yet.
93:10Uh, so it's very tempting to fix it.
93:12So, I tried I added one sentence uh which was a report is actionable only if it meets all four criteria because previously that was ambiguous.
93:20I said a report should include all of these things.
93:23I didn't say that it must.
93:26Uh so you can change the you can change the rubric and be more specific and say yes this absolutely must contain all these four criteria.
93:34Uh and on dev uh that broke Microsoft my 12 out of 12 agreement became 11 out of 12 agreement.
93:41Um and uh it also in the test split it broke Apple uh because Apple didn't have enough uh a big enough risk section as far as the agent was the judge was concerned.
93:53Um what that tells you is that I could change my rubric and make my eval worse.
93:59Uh I can make my eval less helpful.
94:02I can make my EVA less like what a human would do.
94:04Uh even though I'm trying to be specific.
94:08Um the script also prints precision and recall.
94:12Uh precision these are very important things again borrowed from the ML land or rather from stats land.
94:18Um precision asks when the judge says not actionable how often is it correct?
94:25And recall asks of all the reports that aren't actionable how many did the judge catch?
94:30Those two sound very similar but measure different things.
94:35Um, in most cases, you're going to want to prioritize recall because what recall gives you is it catches everything, but it gives you high recall catches everything, but it will give you
94:48uh false alarms.
94:49A false alarm costs you a minute or two when you look at a report that turns out to be fine.
94:54Uh, but a missed failure uh would give you a bad report that reaches a customer.
94:59Um, so you have to trade one for the other.
95:03If recall goes up uh um then uh precision will go down and vice versa.
95:10So for your application you have to decide whether being accurate uh as in throwing away some things that were actually correct uh but but uh you decided they weren't is better or worse
95:24than getting false alarms where you say something is important and it turns out not to be.
95:30Um judges also have biases.
95:34Uh I mentioned one of them already.
95:36Um one is position bias.
95:38Given two options, they tend to prefer depending on the model either which one comes first or which one comes last.
95:44It depends which way the model around which way around the model was changed.
95:48Uh but you should swap your options so that it's not always looking at those uh things in the same order.
95:54Um they have length bias.
95:56Uh, judges prefer long answers and they also have a confidence bias.
96:00Judges prefer answers that sound fluent and confident.
96:04Um, and the one that I mentioned already is self-preference.
96:08A model judging its own output tends to rate it higher.
96:12Uh, which is why we judged Haiku with sonnet.
96:15Uh, and ideally you judge haiku with something from open AAI.
96:21Um, but before you get too worried about how much you can trust a judge, it's worth noting that human judges also disagree with each other a lot of the time.
96:30Um, if you give two experts the same outputs in the same rubric and measured how often they agree, uh, there is a standard score for that called Cohen's Kappa,
96:38uh, and for this kind of task, it's often as low as 02 or.3.
96:42Uh, which is to say they agree with each other not very much more often than just random chance.
96:48Um
96:49[snorts]
96:50so the bar for your human judgment is h for the bar for your judge is human performance and h human performance uh is not actually that good when you compare humans to each other.
97:00So having a judge that disagree you don't do not need a judge that agrees with you 100% of the time.
97:07Uh your judge is never going to agree with you 100% of the time.
97:10What you want is a judge that does better than another human doing the judge would be.
97:14uh because your judge can run across thousands of examples and help itself out that way.
97:20Um one last principle on judges uh is that failures should seem fair.
97:25When something fails, it should be obvious what the agent did wrong.
97:28If you look at the failure and go, "Well, actually that looks fine to me," then the problem is probably your eval.
97:34Um there's a famous example of this uh when Claude Opus 4.5 first scored 42% on a benchmark called Corebench uh which tests whether agents can reproduce scientific papers.
97:48Uh anthropic were upset about that low score.
97:50Uh so a researcher ran the actual transcripts and found uh that uh Claude was just being uh right in a different way.
98:00Uh it the test expected uh six digits of precision and Claude gave the right answer to two digits of four digits of precision rather.
98:09Um and the test said that was wrong.
98:11There were ambiguous task descriptions.
98:13Some of the tasks couldn't be reproduced at all.
98:16Uh so Anthropic went through and they fixed all of the tests and they said these tests could be better.
98:20Uh and magically they got 95% on the eval.
98:24Um this happens all the time.
98:27SWEBench and all of the other uh benchmarks that you see are constantly being re-evaluated and manually tweaked and uh that is why you see version two, version three, version four of of
98:40uh the famous benchmarks uh because people are constantly finding that the benchmarks are not actually as good as they could be and your evals are going to end up with the same problem.
98:49Uh so you need to be constantly re-evaluating your test set.
98:55Uh which brings us to step seven which is uh experiments.
98:59We've got evals that we trust.
99:00Now we are going to use them to actually improve that the agent and prove that the fixed worked.
99:07Uh I'm going to go a little faster now as I've been going on a very leisurely pace.
99:12I'm getting beginning to run out of time.
99:14Um so the here's the problem.
99:17We know that some reports aren't actionable and we know why.
99:20So we change the prompt and then what?
99:22How do you know that it worked?
99:24Uh, how do you know that you didn't break something that was working before?
99:27If you run it on a couple of examples and eyeball it, then you're back to vibes.
99:31A prompt change that you haven't measured is just a vibe.
99:34What you want is a controlled comparison.
99:37You want the same inputs, the same eval before and after.
99:40That is an experiment.
99:42Um so uh what you can do in Phoenix is you can take a bunch of your evals that you have filtered down to not actionable say uh and you can select them and you can add them to a data set.
99:58Um that gives you a set to test against uh against specific things.
100:04Um I'm not going to do that since I've already done it over here.
100:07Um Um once you have a data set you can create turn that into an experiment and run uh your experiment against only the data set only against the evals that were failing as opposed to against your entire set every single time which you know if you've got a decent sized split and a decent sized test set is
100:31saving you thousands and thousands of tests every time.
100:34Uh now we're going to fix what the eval found.
100:39So, uh, if you're still coding along, congratulations.
100:43Open prompts.ts.
100:44There is an improved version of each prompt.
100:47Uh, or you can write your own.
100:49Um, and every change maps to a finding.
100:53So, the judge said, uh, makes no recommendation, so we asked for an explicit recommendation.
100:58It said mentions risks with no evidence.
101:00So, we asked for risks with supporting data.
101:03Faithfulness found invented numbers, so we said use only numbers that appear in your research.
101:08uh and reading traces found that the agent uh was trying to save files all the time.
101:13Uh so we tell it to reply with the report instead.
101:17We're not guessing here.
101:18We are coming up with specific things for the agent to do as a result of the previous results of our evals.
101:24Uh this is what the experiment script looks like.
101:29An experiment is a data set, a task and evaluators.
101:32The data set is the one that we just created.
101:34Uh the task is a function that just runs the agent on each example.
101:38Uh in this case, we're running the entire agent, but we could be running it on a specific subset of the agents behavior or a specific subtask uh if we wanted to.
101:49Uh our task hands back the report and the research.
101:52So each eval gets a small wrapper that picks out the fields that it needs.
101:56Uh and then faithfulness gets the research as its context.
101:59The same as before, uh Phoenix runs the task on every example.
102:03It scores every result and it stores everything so you can compare.
102:07If you want to run this, you can run npm run experiment and it will take about four minutes to run.
102:12Um, but I'm running out of time so I'm not going to wait for four minutes.
102:17Uh, I ran mine ahead of time.
102:21How many examples do you need?
102:22Uh, is a really good question.
102:24Uh, 10 to 20 examples gives you a rough direction of where things are going.
102:28Is very useful in dev.
102:30Um, if you're getting into production, uh, you're going to want hundreds of examples.
102:35Um, that is going to take a long time, but is, uh, very worth it for accuracy.
102:42Um, for every, uh, to have your error bars on your accuracy, you need to quadruple the number of examples.
102:52So, uh, going from 25 examples to 100 examples is really, really helpful.
102:58uh and is not that much effort going to get that big a jump again you'd have to go to 400 examples which begins to get into the territory of you're never going to bother to do this
103:08uh but the bigger the better uh and the more production failures you throw into your data set the better um so I ran mine three times and the reason I did that
103:19uh is because the agent is non-deterministic each run of each example uh can mislead you in either direction um And there are two useful ways of thinking about how an agent runs
103:31uh when it runs multiple times.
103:33The first is called pass at K which is uh does it succeed at least once in K tries and then there is pass to the K which does it increase
103:44every single time uh does it pass every single time I run uh K times.
103:50Um so as K goes up those two numbers diverge.
103:56Uh the more tries you give it, the more likely pass at K is to succeed because it succeeds at least once.
104:02The more tries you give it, the more likely it is to fail at least once and then pass it K pass to the K will go down.
104:08Um a coding assistant gets that gets it right on the second try is fine.
104:12So pass it K is fine.
104:14Um a support bot that's wrong runtime in every five is a huge problem.
104:20And a financial anal analyst uh that gives inaccurate recommendations is also a huge problem.
104:26Uh so you want pass to the K. You want it to pass every single time for those.
104:31Um when you're improving an agent, there's an order of operations based on how useful making a change is going to be.
104:41The first one is data quality.
104:43If your underlying data is bad, if it's hard to read, if it's difficult to interpret for the agent or for the judge, uh then nothing else you do, nothing else you tweak is going to be useful.
104:54Once you've got good quality data, then you move to the prompts.
104:58Uh that is what we've been messing with mostly today.
105:01Uh is to make things better at the prompt level by changing our rubric, by changing our agent.
105:07Then there's model choice.
105:09Uh model choice has a lot of leverage, but it also changes a whole bunch of stuff at the same time.
105:18uh and also the usual way that you're going to change your model is you're going to change it to a bigger model and that makes things more expensive.
105:25So that is uh creates a tradeoff uh that you don't always want to make.
105:30Uh and finally there's hyperparameters.
105:32Hyperparameters are things like temperature and stuff like that.
105:35Uh they're so useless that uh people have that modern models have stopped allowing you to tune them.
105:41You can't pass temperature to model to modern claw models.
105:45Um so if you're tempted to it is the least useful thing that you can pass.
105:53Um so if we're looking at my experiments you can look in your seed data and get see see the same experiments.
105:58Um you'll see that uh as I improved as I improved my runs I went from uh 16 out 15 out of 16 to 48 out of 48.
106:13and went from 5 out of 16 to 48 out of 48.
106:15And faithfulness went from 2 out of 16 to 3 out of 48.
106:19Uh, which is not that good, right?
106:21Like we really should have done better on faithfulness with my improved prompt.
106:25Um, the uh important thing is did anything that used to pass break?
106:35We can filter down to the passes split.
106:37I'm not going to do it in the UI right now.
106:39Uh, but all of the five reports that passed previously as I improved my prompt still passed.
106:45Uh, so actionability went straight to perfect.
106:48Um, but faithfulness was bad.
106:50Faithfulness uh went from 2 out of 16 to 3 out of 48.
106:54Uh, even though the prompt literally says only use numbers that appear in your research.
106:59Um, so we fixed what we measured in a single line of the prompt.
107:05Uh, didn't fix the agent making things up.
107:07That is the next thing that we have to fix for this agent.
107:10Uh we have fixed whether or not it is making a good judgment call.
107:14Uh but we have not fixed uh that it is uh inventing the data some of the time.
107:22Um if we want to we can look into uh each of the runs and you'll see that each of the because it's nondeterministic each of the runs are coming up with different advice.
107:33Um, but I'm not going to do that because I'm running out of time and I want to get to some other stuff.
107:38Uh, let's get to Okay, cool.
107:46Um, a good eval like the 56 the 511 split that I was talking about on my 16 data sets.
107:54That is what is called a capability eval.
107:56A capability eval is a thing that our agent can't do yet.
107:59And it gives us a hill that we can measure ourselves climbing.
108:03Uh when it passes reliably, when you get a test that passes 100% of the time, you don't throw it away.
108:09You turn it into what is called a regression eval.
108:12That is the kind that you can put into your CI and say this used to pass 100% of the time.
108:16If it suddenly stops passing 100% of the time, then we can call that a failure and we should stop release.
108:22Um how do we climb this hill is the question.
108:26uh we could work on the prompt again or uh we could do the thing that I said that we would do which is we can upgrade the model.
108:35Uh so you can run the experiment again by passing in a new model haiku 5.5 and then run npm experiment.
108:42Um, I said at the start of today that you can't switch models without eval.
108:47And this is what it looks like when you can switch a model with an eval because we know now that we are measuring all of these things.
108:54Um, with three repetitions, it takes 13 minutes, which is more time than we've got left.
108:58So, I'm just going to skip to the results.
109:00Uh, here is Haiku 4.5 versus 5.5. Uh, faithfulness improved.
109:06It got more than three times better.
109:08It is still terrible.
109:09This agent is still making things up all the time.
109:12Uh, but consistent calls got better.
109:14So, it is at least making the same call over and over.
109:17Is that good?
109:18Uh, it's also costing four times as much money.
109:21Is that worth it?
109:23Uh, is a question that you need to make for yourself in production when you're doing this kind of thing.
109:28Um, one of the ways that it got more consistent, one of the ways it stopped varying between buy and sell between calls, uh, is it just got more conservative.
109:39Haiku 5.5 uh it turns out just says hold all the time.
109:43Uh so uh is that better or worse?
109:48Probably worse.
109:49Uh probably we need to iterate on Haiku 5.5 being too conservative uh about things.
109:55But at least we know now that something has changed and that the conserviveness is a problem with Haiku 5.5. Um that is the problem that I found.
110:05It went it went to 41 of out of uh 45 calls it said uh hold 41% of the on 41 of the times.
110:13Um which is perfectly consistent between runs.
110:17Uh so it it got better at the uh consistency check that I was talking about over here the consistent call.
110:24Uh but it got there by just picking the same answer every single time.
110:28[snorts]
110:28Um but that is the eval iterate cycle.
110:34Uh and the cycle is where the value is not anyone's score.
110:37You run your evals, you read the failures and the explanations, you spot the pattern, you change the prompt or the tools or the data and you run the experiment again.
110:45Did it get better?
110:46Uh your evals can now answer that question for you.
110:49You can go from I think it works to I can show you it works.
110:54Uh I have just enough time to talk about how to do the last step which is eval in CI.
111:00Uh more and more teams that ship AI quickly write the eval before the feature.
111:05That's called eval driven development.
111:07Um say you want your support agent to always check who the customer is before giving a refund.
111:13Uh you can write the eval that checks for that first and now done has a definition that you can measure a hill that you can climb from the start.
111:21Um, we talked already about capability capability evals and regression evals.
111:29Um, here's how uh to create a eval that runs in CI.
111:36Um, if you've never used VEST, VEST is a JavaScript testr runner and Fenix can plug straight into it.
111:42Uh, if you open agent.eval.ts, you'll see that it looks like an ordinary test file because it is one.
111:48There is a describe block and a test for each query in our regression set.
111:52So Coca-Cola, Apple versus Microsoft, Rivian, uh, and the fake ZZZQ.
111:57Um, each test runs the agent, logs the report as the output, uh, and then runs our evals on it.
112:03And the same ticker check and actionability judge that we wrote earlier.
112:06We don't have to add new evals for CI.
112:09Uh, then at the bottom, we gate the build on acceptance criteria.
112:13This is where I talked about guardrails and northstars.
112:16Mention sticker is a guardrail.
112:18It has to be talking about the right company.
112:21Um actionability is a target.
112:23The average has to stay 08 or better.
112:26Uh because an LLM judge can be a bit noisy on a single report.
112:29So we gate on the average uh not on just one.
112:33Uh and then you can run npm test uh and it runs like any test suite.
112:38Um and at the end the reporter prints out a scorecard.
112:42Um, and that'll take about four minutes, which is more time than I've got left.
112:46Um, a red build.
112:49If a failed build, if your build fails, this is what you'd get.
112:52All four tests ran and every report mentioned its ticker, but Rivian's report wasn't actionable.
112:56So, the actionability average came out at 75.
113:00Uh, it doesn't say you failed.
113:01It just says that you missed.
113:03Um, but, uh, a green run, uh, is where all of those pass.
113:11Um, I'm not going to show the experiment view because again, I'm running out of time.
113:17Uh, but when a build fails, you don't just get a red cross.
113:20You get an explanation that says whether things are good or bad and why they went wrong.
113:25And that is the important thing.
113:27Uh, this same test suite can be run in GitHub actions.
113:30Uh, it starts Phoenix as a service container.
113:33Uh, but now we've done everything that I've said that you can do.
113:40uh you can run your evals, you can test things, you can make sure uh that it matches your human judgment, you can put it into CI and you can make sure that your builds are gated on whether or not
113:51uh the LLM's judgment of what you've encoded as the definition of good uh really works.
113:58Um and there's something I love about this which is that anyone who knows what good looks like can write evals now.
114:06uh your product managers, your support team, your person in sales who keeps hearing the same complaint.
114:11They don't need to write TypeScript.
114:13Uh they just need to describe what a good answer looks like uh and give it to you or even better just give it to a coding agent which will then write all of the evals for them.
114:24Uh you shouldn't run your whole agent on every single commit because uh it's going to take a really long time and cost you a whole bunch of money.
114:33Um, but now you've got the repo I promised you.
114:37You've got all of the code.
114:38You've got all of the examples.
114:39You can see how everything is done.
114:40Uh, you can point your agent at it and say, "Do this, but for my codebase," uh, and your agent will figure it out.
114:47Um, there is one thing that we didn't get time to build today, uh, which is, uh, once you're live, you should keep evaluating.
114:56You should run your evals on a sample of real traffic, uh, and alert when scores drop and stay down.
115:03um not on every single failure because one failure would be noise.
115:06Um but if you get a trend in failures, uh production monitoring is incredibly useful and eval can get you that as well.
115:15So this is the full cycle that we built.
115:18We traced, we read, we evaluated, we checked, we improved, uh and we locked it in with CI.
115:24Uh but you don't have to do all of the stuff that you've learned today on Monday.
115:28You can start small.
115:30You can just instrument your agent.
115:31you can just read some traces.
115:33You will get information reading 12 traces that you wouldn't have got uh by just looking at the output.
115:40Um so run them.
115:42Look at the results and build from there.
115:45Uh so go try it on a app of your own.
115:49This is the repo that I said that everything was in.
115:51Uh the Phoenix docs are there.
115:53Uh Phoenix itself is open source.
115:56So you can mess with the code and tell us that we got stuff wrong in there.
116:00Uh and that just on time is it.
116:03Thank you very much everybody.
116:06[applause]