From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

AI Engineer · 111 min · 1407 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hello everyone.
00:15Uh I have a packed room for a two-hour workshop which is amazing.
00:21Welcome to AI Engineer.
00:22I know no one's officially welcomed you but uh welcome to AI Engineer.
00:26Uh this venue is amazing.
00:28This crowd is really quite surprising and I have uh a ton of content for you today.
00:35Uh my name is Lori Voss.
00:36I'm head of developer relations at AriseAI.
00:40Uh some of you may remember me from when I used to co-found npm Inc. Uh so you may remember me from like JavaScript days.
00:47Uh but these days I talk about AI uh and how to test it uh and how to make it work.
00:53So what are we going to be covering today?
00:55Uh we've got a really good stretch of time together.
00:58Uh so we're going to cover a lot of ground and I'm going to start with fundamentals.
01:02That's why this is called a 101 session.
01:05Uh we're going to start with what eval are, why you need them, uh and why agents make evaluation harder than simple LLM applications.
01:14Uh then we're going to set up tracing in Arise AX, uh which is how you capture the raw data you need to run evals in the first place.
01:21Uh, we're going to build a simple AI agent uh with the claw agent SDK.
01:26We're going to run it and we're going to look at uh the traces it produces.
01:30Can I get a show of hands if you've already built an agent?
01:35All right, great.
01:36I'm going to spend no time at all on how the agent works because I assume everybody knows how an agent works and you're here to learn how to evaluate them.
01:44Uh once we've got some data, uh we're going to do something that a lot of tutorials skip, which is we're going to actually look at the data.
01:50Uh we're going to read our traces.
01:52We're going to categorize what went wrong.
01:54Uh and we're going to figure out what to measure before we write a single eval.
01:58Then we're going to write three kinds of evals.
02:00There are code evals, which are simple deterministic checks, uh very similar to unit tests.
02:07Then we're going to use built-in LLM evals uh like faithfulness uh where a second LLM judges the uh output of your first one.
02:16Uh and we're going to do a custom eval uh LLM as a judge from scratch.
02:21Uh and we're also going to test whether our judges are judging correctly, a process called meta evaluation.
02:28Then we're going to finish with data sets and experiments, which is how you iterate on your agent uh and actually measure whether it's getting better.
02:35uh including we're going to use cloud code to automatically improve our app in response to feedback from our evals.
02:43So, first things first, uh get your notebook.
02:48Uh I messed up earlier and didn't have permissions set correctly.
02:53So, if you scanned this earlier and it didn't work, it works now.
02:56Uh grab this notebook and make a copy for yourself.
03:00That is where we're going to be living all day.
03:04And I'm going to wait for that to happen because otherwise I'm going to have to keep swapping back to this slide.
03:11All right, most of those are down.
03:15No, no, I'm still seeing people taking pictures.
03:19All right, cool.
03:22We're also going to need a certain amount of AI infrastructure.
03:26I'm going to be using anthropic and cloud code today.
03:30Uh, Arise AX works with OpenAI.
03:33It works with every other uh, lab provider, but I had to pick one in order to do a workshop about it.
03:37So, I've used Claude.
03:38So, this is going, this notebook is going to expect you to have uh, an anthropic API key, which you can get from console.anthropic.com.
03:45If not, you can, if you are very clever, do a fast swap to OpenAI.
03:50Um, you're also going to need an Arise account.
03:53Go to arise.com and start a free trial.
03:56Um, from there you will need an Arise Space ID and an Arise API key, both of which you can get from the settings page.
04:05Uh, my lovely assistant Dat is here.
04:08Uh, that's him in the corner.
04:10If you run into any problems getting your API keys or anything like that, uh, just flag him down and he will run over to you and help you figure it out, which is,
04:21uh, incredibly nice of him.
04:23Thank you, Dad.
04:24Um, yes.
04:30So, uh, who who had all that stuff already?
04:34Anybody already got an Aris X account?
04:38All right, I'm going to let you hang out for this.
04:41I'm going to let you set that one up for a little while.
04:48The way you do it is you go to arise.com and you hit Oops, it's sent me to the docs because I always live in the docs.
05:02You go to arise.com.
05:04Thank you.
05:05Uh and you hit the big pink get started button and then you sign up from here.
05:16Do not accidentally sign up for Phoenix.
05:18Phoenix is our lovely open- source project which also has a hosted service underneath Phoenix.
05:23Uh sometimes when you Google Arise, you get sent to Phoenix instead.
05:27Don't sign up for Phoenix.
05:28You're doing AX today.
05:34All right.
05:35While you are signing up, uh I'm going to say a little bit more bit more about what AX is.
05:39It is an observability and evaluations platform.
05:42It captures traces from your AI application.
05:45It stores and runs your evaluations and it watches your app in production and alerts you when something goes wrong.
05:51Uh by default, we host it for you.
05:53That is why we're using it today because you don't have to like run some software to make it run.
05:58Uh so there's no infrastructure to manage.
06:01Uh though if you are from an enterprise that is super precious about your data and you're like no, we need to run it ourselves, you can absolutely do that if you want to.
06:09Um, and like I said, uh, if you've used Arise Phoenix, which is our open source tool, a lot of this will feel familiar.
06:17Uh, they have the same core ideas.
06:19One is, uh, built for open source accessibility, and the other one is built for, uh, enterprise with production and teams and things like that.
06:30So, how are we doing on signing up for keys right now?
06:35Wi-Fi sucks.
06:36The Wi-Fi sucks.
06:38That is predictable really, isn't it?
06:49All right.
06:50Is anybody like desperately in need of assistance right now?
07:05All right.
07:12Okay.
07:12Well, the advice is to switch to your phones as a hot spot.
07:14I apologize for the Wi-Fi.
07:18I'm really worried about my 2PM workshop where I have a Git repo that I know is 200 megabytes large and I'm like, no one is going to get the repo because it's too big.
07:29Yeah, start cloning that repo now.
07:32Anyway, uh let's talk about these these fundamentals.
07:35There's a lot of theory in this workshop which gives you time to uh catch up.
07:39Um here's the mental model.
07:42Evals are testing for AI and traces are logs for EI AI.
07:46Why we had to come up with two completely new names for those things?
07:49I don't really know.
07:50Uh you write tests for your code.
07:53Given this input, I expect this output.
07:55Uh eval do the same thing, but for AI outputs where the outputs are never exactly the same twice.
08:02Just as logs record what your server did at runtime, uh, traces record what your AI did.
08:08So, every agent call, every tool call, every LLM invocation with the inputs and outputs at each step.
08:15The building blocks of a trace are called spans.
08:19Uh, in this little screenshot, you can see a trace in Arise AX.
08:22It's made up of a bunch of spans.
08:24It's got this tree structure.
08:26Uh, and each span represents one step in the execution.
08:29An LLM call is a span.
08:30A tool call is a span.
08:31And a full agent turn is a span that contains other spans inside of it.
08:35Each span records its input, its output, uh timing, token counts, and a whole bunch of metadata.
08:40Uh everything that you need to understand what happened at that step.
08:44Traces tell you what happened.
08:45Evals tell you whether it was any good.
08:47That is the model for mental model for today.
08:50This is not rocket science.
08:54But let me talk about why we even need evals.
08:57Uh and that is the vibes problem.
08:59Uh AI features usually get shipped by building an AI feature, running a few queries, saying does this look right?
09:07Yeah, it looks good.
09:08And then you ship it.
09:09That is by far the most popular method of shipping AI applications today.
09:14Uh and the problem is that then it fails on inputs that you didn't test.
09:18Uh and the thing is it probably looks good on your test queries, but the problem is that you ran it three times.
09:23Three times is not a test suite.
09:25Uh the usual fix for uh flaky software is unit tests and they don't work here because the output is non-deterministic.
09:33Uh the same prompt produces different text every run.
09:36So two different responses might both be correct.
09:38So there's no expected string that you can assert against.
09:42So teams fall back on human review.
09:43They watch it run.
09:45It looks fine and they ship it.
09:47Uh that doesn't scale.
09:48It doesn't catch regressions.
09:50And most importantly, it doesn't run in CI.
09:54Here's some things you can't do without evals.
09:56Uh if you change your system prompt to fix a tone issue, then the tone would get better.
10:00Uh but now your uh bot is hallucinating product features.
10:03Without eval catch that uh until a user reports it.
10:08With a faithfulness eval, you would have seen the spike before it shipped.
10:12Uh and every prompt potentially affects every kind of input that users send.
10:17So you can be changing the prompt for one thing and have a completely unexpected effect at some totally different part of your application because the prompt is just a huge block of text that you're sending to your AI.
10:28Uh you can't manually test every combination.
10:30So without eval you're playing whack-a-ole.
10:32You fix one problem and you break something somewhere else.
10:36Evals instead give you a number that you can track and compare and act on.
10:41You also can't switch models without eval.
10:44This matters more and more because this field moves so fast.
10:47Uh new models drop every few months.
10:49Uh without eval switching models means weeks of manual testing.
10:53Is it better?
10:54Is it worse?
10:54Did something get did something break?
10:56You don't know.
10:57With evals, you run the suite.
10:59You compare the scores uh and you know within hours.
11:02And that is the difference between flying by the seat of your pants and flying by instruments.
11:08And this is not theoretical.
11:10uh real teams, Dcript, Bolt, Anthropic's own clawed code all follow the same arc of shipping on vibes and then discovering that vibes don't scale uh and then uh building evals and that is the same arc that we're going to follow in today's workshop.
11:27As I mentioned earlier, there are two broad types of evals.
11:30Code evals are deterministic functions.
11:32They run in milliseconds and they cost nothing and they give you an immediate unambiguous answer like did the output parses JSON?
11:39Is the output less than 500 tokens?
11:41Did it mention the thing that I asked about?
11:43The big advantage of Kodi valves is that they are fast and cheap and totally reproducible.
11:48They get the same they get the same input and they give the same answer every time.
11:52That makes them very similar to old style unit tests.
11:55The downside is that like old style unit tests, they can be brittle.
11:59Uh we have some strategies that can help avoid that and I'm going to be covering those later.
12:04um because two perfectly correct answers might be worded completely differently and you have to adjust your expectations of unit tests to be able to handle that.
12:14The second type of eval is an LLM as a judge eval.
12:17These use a second LLM to grade your outputs against a rubric.
12:21Rubric being yet another word that we made up uh that just means rules.
12:25It is your set of rules that you put in your prompt about what counts as good.
12:29Uh LLM evals are much more flexible.
12:32uh they can handle the kinds of questions that Kodi valves can't answer.
12:36Uh is this response uh factually accurate?
12:39Did it stay faithful to the source material?
12:41Is the tone right?
12:42Uh for a customer service context, the strength of LLM judges uh is that they understand meaning uh not just the strings that are in your output.
12:51But LLM judges have trade-offs.
12:53The weakness is that they cost money to run and they are nondeterministic themselves.
12:58Uh so they can be wrong which means that you need to calibrate them against human judgment and we're going to do that later today.
13:05Um there's also a third type of eval which is human evaluation.
13:10Uh a domain expert can read your output and grade it.
13:13Uh and that is the gold standard for quality.
13:16Uh but it is slow and expensive and humans get tired which is why we can't use it all the time.
13:21Uh fun fact that's not very fun.
13:23Human annotators miss up to 50% of defects due to fatigue.
13:27So they're not perfect either.
13:29So when you build an LLM as a judge and you're like, well, it doesn't always catch my errors.
13:34Remember that if you were judging with a human, the human would also not always catch your errors.
13:39Uh but these are complimentary approaches.
13:41Most real applications are going to use a a collection of Kodi valves, LLM as a judge, and human evaluation.
13:49So when do you use which?
13:51Use Kodi valves when the answer is deterministic.
13:54So, format validation, length limits, forbidden phrases, uh required fields, use LLM judges when you need semantic understanding, a correctness, eval accurately, a faithfulness eval asks, did it stick to the source documents without hallucinating?
14:11Uh, and you should keep humans in the loop for failure modes that you haven't seen before.
14:16Uh, and to verify that your LLM judges are actually judging correctly.
14:19LLM judges can be wrong.
14:21uh judging them against human judgment is how you know whether or not you can trust them.
14:27All of that is true of any application that uses LLMs.
14:31Uh but agents make things even harder in some ways that are uh worth calling out.
14:35You can think of it as a ladder of complexity.
14:38A single LM call is relatively simple.
14:40You get an input, you get an output.
14:42When you're talking about an agent, an agent is making a series of tool calls and a series of decisions, which means that at every single step in the uh chain,
14:51it is the chances that it has made a mistake are getting higher and so the chances that it will go off the rails uh are going to get uh greater.
15:00So you have to judge the intermediate steps as well.
15:03You have to uh check whether your agent is using the right tool.
15:06Did it pass the right arguments?
15:08Did it know when to stop?
15:09getting stuck in loops is a real failure case for agents.
15:13Uh and then multi- aent systems if you are building those add yet another level of complexity.
15:18They add handoffs.
15:19One agent decides to pass work to another.
15:22Did the triage agent route correctly?
15:24Did the specialist agent handle the the handoff gracefully.
15:27Each layer adds new ways that things go wrong.
15:30Uh and non-determinism means that you need to have more evals because errors can cascade.
15:38Uh here's what an example of an cascading failure looks like.
15:41If you're you ask a query about Tesla, meaning the car company, your agent searches the web, finds a whole bunch of information about an 18th century inventor, passes that back to your agent, which writes a beautiful, well- formatted report about an 18th century inventor, and passes that to your boss.
15:58Nothing that it did there was wrong.
16:00It you told it to search the web for Tesla.
16:02It searched the web for Tesla.
16:03It got information about Tesla, and it wrote a report about Tesla.
16:06where where did it go wrong?
16:08Uh this is worse than an obvious failure because you have to be reading your inputs and outputs for that to work.
16:14You can't there was no unit test that would have caught that.
16:19Uh but agents can also do the opposite.
16:21They can get things right in a way that your tests weren't expecting and get graded wrong as a result.
16:26This happened to Anthropic when they ran a benchmark called Tao Squared Bench uh which simulates multi-turn customer to service tasks.
16:34uh they asked the agent to book flights uh and asked it to do something that they thought was impossible which was reschedule an economycl class flight.
16:44Uh but it turned out that the policies that they gave the agent allowed the agent uh to first upgrade the flight to first class.
16:52First class flights can be rescheduled which uh economy flights cannot.
16:57So the agent found a way to reschedule an economy flight uh and failed the test because it was supposed to be not able to do that and it was able to do that.
17:06Uh so this is an example of how agents being flexible agents finding things that you didn't think of uh needs to be accounted for in your tests.
17:15Uh there is a difference between creatively correct and wrong and we're going to come back to that when we talk about grading your agents later or rather grading your evals.
17:25Um there's also a second way of categorizing your eval into two categories uh which is uh capability evals and regression evals.
17:34A capability eval asks uh can my agent do this thing at all?
17:39Um capability evals are expected to mostly fail.
17:42Your agent is expected to get a very low score on them.
17:45Uh because they give you a hill to climb.
17:47It gives you something for your agent to try and get better at.
17:50Uh once you've climbed that hill, your capability eval turns into a regression eval.
17:54A regression eval is an eval that you expect your agent to pass completely or nearly completely every single time.
18:02So as your agent matures, you'll be giving it capability eval after capability eval and slowly turning them into a suite of regression evals that you run all the time to make sure that uh previous behavior has not regressed.
18:17This is what an eval result looks like.
18:20Uh every eval produces a score and a label.
18:22In this case, it's a numeric score just often just zero or one.
18:26Binary scores are very easy for uh LLMs to do.
18:29Uh and it comes with a human readable label like correct or incorrect or valid or invalid.
18:34Um LLM judges add a third thing which is the explanation.
18:38Um code eval produce explanations because they are just code.
18:42Um but LLM, an LLM as a judge will say not just that something is incorrect, but it will say why it decided that something is correct.
18:51And this is incredibly valuable uh because if you are living in a world of coding agents, you can take a whole bunch of explanations from a bunch of evals and pass them back to your coding agent and say here is all the reasons that you messed up in the last test.
19:05What could you do that would make you better at this?
19:07And your coding agent will just go cool thank you for the feedback.
19:10Improve your application uh and it will get better at it next time.
19:14This is something I'm going to show you uh in today's workshop.
19:17It is how you turn a failing eval into a prompt improvement.
19:22This is a real judge explanation.
19:24Um, in this example, it it's a travel planning agent.
19:28Um, and you're running a correctness eval.
19:31The judge doesn't just say incorrect.
19:32It tells you exactly what's missing.
19:34You asked for a budget flight.
19:35It didn't give a budget breakdown.
19:37It says the agent gave destin destination info and recommendations.
19:41Uh, but the user asked about budget travel and there were no cost estimates.
19:45That explanation makes the eval actionable uh because you now have a concrete failure.
19:50You know what to fix in the prompt.
19:51Uh and the explanation is what makes evals into a useful debugging tool and not just a scoreboard.
19:57Uh and if you're seeing the same explanation across 50 different traces, uh you know you have a systematic problem and not a one-off edge case, which makes it the first thing you should fix.
20:07So you have to take your eval explanations uh and categorize them into types of failures uh and count up the categories.
20:14This is called coding.
20:15Again, we can make up words from the L from the ML word uh from the ML world.
20:21Um but categorizing things is known as coding.
20:24Um and of course each explanation is natural language.
20:28Uh so it is hard to do this coding uh unless you use yet another LLM to do the categorization for you.
20:34We'll be seeing how that works uh towards the end of today.
20:39Um, and like I said, you can hand a whole pile of these explanations to a coding agent and let the app fix it for you.
20:46Uh, this is the full loop that we're going to build today.
20:49We're going to, uh, instrument, trace, eval, and iterate.
20:53Each step feeds the next.
20:54You define what you want.
20:56Uh, you build it, you measure how well it works.
20:59You ship it, you monitor it in production, and you iterate based on what you see.
21:03Every traditional software product goes through a loop like this.
21:06Uh but for AI the measurement step is where most teams fall down and evals are how you measure.
21:12Evals are the connective tissue uh across this whole life cycle.
21:16Uh so let's get started building it.
21:19Step one is setting up tracing with Arise EX AX before we can run evals.
21:23We need something uh some data to run our evals on.
21:27Uh and you can't evaluate what you can't observe.
21:30So observation comes first.
21:33So hopefully you've conquered the Wi-Fi uh and you have our notebook at this point and that is running around to make sure uh that you do.
21:46Uh let's go to our very first code cell which is where I install our dependencies.
21:52Uh claude agent SDK is self-explanatory.
21:56That is the claud agent SDK.
21:58uh open inference open inference instrumentation claude agent SDK is the auto instrumentation package for the claude agent SDK.
22:05This is how it knows to capture traces from cloud agents uh without you changing your application code.
22:12Uh as I mentioned earlier, you can use open AI instead if you want to uh or some framework.
22:17Um we have packages for all of those as well.
22:19You just need to swap them in.
22:21Um, Arise and Arise Otel are the packages that send uh your traces to AX and let you read them back.
22:28Uh, and there's a Phoenix package in there that we use as utility package.
22:32Don't worry about it.
22:34Um, the anthropic package lets you use Claude both to power our agent and to judge its output later.
22:42Uh, so uh, go ahead and run this cell if you haven't already.
22:45And while it's installing, I'm going to talk about what it is that we're going to actually build.
22:50Um the claude agent SDK if you haven't already used it is Anthropic's framework for building agents.
22:55So Anthropic's answer to lang chain and things like that.
22:59Uh it can use tools, it can search the web, it can maintain conversation context across turns.
23:04Uh open AI has their own uh agent SDK and of course there are whole agent frameworks like crewa align chain and mastra and uh AX is compatible with all of those.
23:13So no matter what framework you've built uh your agent in uh it is already instrumented and you can just turn on logging and all of your traces will light up.
23:23Um hopefully your install is done by now which is uh optimistic timing on my part.
23:32Um next set your keys.
23:35Uh in this case I have stolen my keys from uh collab.
23:40If you are feeling naughty, you can just paste them in directly to your cell and put them there.
23:44Um, three things go in here.
23:47Like I said earlier, you need an anthropic API key, uh, which is going to power our agents LLM calls, and it's also going to power the judge later.
23:54You need an Arise API key, which you get from your settings, which authenticates your traffic to the AX service, and you need an Arise space ID, which tells AX which uh, workspace to put all of your data in.
24:06Uh if your keys aren't working, the usual culprit is a copype error.
24:10Uh and of course that is still running around to help you if uh your keys aren't working.
24:16That is has been described as a walking security hole.
24:20Uh so he will definitely give you a key if you can't figure out how to get your keys to work.
24:25So now we scroll to the register section.
24:29Uh I'm going to bump up my fonts here.
24:38One more time.
24:39The QR up one more time.
24:41Sure.
24:42Probably.
24:56Sorry.
25:01Uh, yeah.
25:02I didn't put my mine lives in Collab.
25:04So, Collab's secrets feature pulls it out.
25:09All right, I'm going to go back to where I was.
25:13Hopefully, I can do that.
25:24All right.
25:25Uh so we've installed dependencies, we've talked about the SDK, we've talked about the secrets.
25:33Uh and now we're at the register.
25:36So this is the magic.
25:38Uh anybody who works for Arise will tell you that you only have to add two lines of code to your application in order to instrument your call.
25:44And this is these are the two lines of code.
25:46Uh what we're doing here is we're passing in our arise space ID, our API key, uh and a name for it to trace everything to.
25:56Um the register function is setting up open telemetry.
25:59Open telemetry is the industry standard for application observability.
26:03Uh and it has a layer on top called open inference which adds LLM specific attributes.
26:08Things like prompt text, completion text, token counts, which model was called, which tools were invoked.
26:14um we point it at ax uh and we give it a project name and that is how your traces get grouped in the UI.
26:21Uh in this case we're also passing batch equals false because uh when you run in a notebook uh you want to send your telemetry as soon as it happens and not batch stuff up which it does for efficiency in production.
26:33Um that one line at the bottom, the claude agent SDK instrument tells the cloud SDK to send a span to ax every time it makes an LLM call or invokes a tool.
26:43Uh and that is now instrumented.
26:48The reason this works with so little code is because open telemetry is an open standard that nearly everybody uses.
26:53So the cloud the cloud SDK authors, the open AAI SDK authors, the lang chain authors, all of them have already written the code inside of their SDKs that calls open that calls open inference and sends data back.
27:06So all you have to do is say, "Hey, I'm an open inference collector.
27:09I live here.
27:10Send it to me."
27:13One more bit of setup is that the register call sends data to AX.
27:17Uh and the Arise client reads data back out of AX.
27:21Uh so we'll use it to pull our spans back into the notebook.
27:24Uh that is what this line is about.
27:26So you need to make sure that you've put in the same keys there or uh copied them across.
27:40Oh, you need to make a you need to make a copy of the notebook from the file menu to be able to edit it.
27:55Cool.
27:56Um, all right.
27:59So, if you've run that run those cells, then we are ready to build.
28:03Uh, before we build, one thing worth knowing about, uh, everything we're going to do by hand today, you can also get your coding agent to do for you.
28:10The reason we are doing it by hand is so that you have a fundamental understanding of what it is that you're doing and you're not just vibing your way to success.
28:16Um but we publish a set of skills called the Arise skills uh which know how to do all of this for you at the skill level.
28:24So you can install your Arise skills uh and um run evaluations, create data sets, uh instrument your app in the first place.
28:33Um they all use uh AX's recommended patterns and you install them once with a single npx command.
28:39They work in cloud code, they work in cursor, they work in codeex and dozens of other coding agents.
28:44Um, so we're going to do stuff manually today, but in production in a real life environment, we are using our own skills every day to do this stuff.
28:54Uh, so now let's build our agents.
28:57Uh, when I asked if everybody had built an agent before, everybody put their hand up.
29:01So I'm not going to spend a lot of time explaining how this agent works.
29:03I'm going to assume that you already have an agent somewhere and you're just trying to instrument it and make it work for you.
29:09Um, the fake agent that I'm using today is a financial analysis chatbot uh using the cloud agent SDK.
29:16You give it a stock ticker and a focus area and it searches the web for the latest information about that stock ticker.
29:22Uh, researches real current financial data and writes a report for you.
29:26Uh, this is a a real use case that whole startups are built around.
29:29Although uh my agent version is extremely simple.
29:32Um, I'm assuming uh that you built an agent already.
29:36Our agent works in two turns.
29:39Um there's first a research turn that uses tools to search the web and gather data.
29:43Then there is a writing turn that compiles the research into a readable report.
29:48Uh so let me walk you through that.
29:51Uh here is my research prompt and my write prompt.
29:54These are deliberately extremely simple.
29:55They are going to cause errors later and those are the things that we are going to use our evals to debug.
30:00Uh and then I'm setting up my claude agent options.
30:03I'm using uh Claude Haiku uh as the agent underneath because Haiku is you know capable but it will make mistakes and we want some mistakes so that we can fix them.
30:13Uh and it's also fast and cheap so I don't burn a lot of money uh of your money when as we do this.
30:19Um the permission mode controls uh what the agent is allowed to do autonomously.
30:27There is a catch hiding in the in the specific choices that I have made and the specific allowed tools.
30:33uh that I am giving this agent right now as we're going to find out later.
30:38Uh and now let's look at our two turns.
30:40Um turn number one is research.
30:44There is a bunch of boiler plate in here that I'm not going to explain which is just about outputting the output so that we can see what it's doing as it's going.
30:51Uh turn two is writing the report with again a pile of boiler plate.
30:56Um the prompts are critical.
31:00So this is where I want you to start doing your own changes in your own notebook.
31:04I have given it extremely basic prompts at the top.
31:06If you can see if you can think of a better prompt just off the top of your head uh that is going to do a better job than the prompts that I put in
31:15uh at solving this problem.
31:16What would be better than what I put in to uh do the research in the first place?
31:21What would be better than what I put in uh to do the report writing?
31:26Um, and if you oneshot it, then you're going to have uh less stuff to do later on.
31:32Um, one of the things we do here is we wrap the whole thing in an open telemetry span.
31:40Uh, this is because if I don't do that, then claude inter Claude sends this as two the two turns as two separate spans and we wanted them to be grouped together in the UI.
31:51Uh, so I've wrapped them in a span together so that they come out as a single agent turn.
31:55Um, this is just saving us some time later.
31:58I wanted to be clear about why I'm doing this uh bit of code.
32:02Um, and now we can run our agents.
32:06Uh, I have run my agent in advance because I wanted to know exactly what it was going to do.
32:10Um, but now is the time to kick off your own agent.
32:15Um, I've asked it to analyze Tesla, specifically its financial performance and growth outlook.
32:21Um, you should go ahead and run this cell.
32:24It'll take a minute or two because the agent will actually search the web.
32:27Uh, and the LLM is doing multiple rounds of reasoning.
32:30Um, the agent receives our research prompt.
32:33It decides it needs to search the web, which is a tool call.
32:35It reads the search results.
32:36It decides if it has enough information and maybe searches again.
32:40That is the agentic part.
32:42You didn't say do a web search, take the web information from the web search and write a report about it.
32:47The agent is deciding is this web search enough information?
32:50And if not, I will write I will run more web searches.
32:53It will run an in indeterminate number of web searches to get enough information until it has decided what enough means.
32:59Uh and then it will write the report.
33:02This execution path is non-deterministic by design and that is the key here to understand.
33:07Uh if you run this again with the same input, your agent might search for different things.
33:11It might find different results.
33:12It might write a different report.
33:14Uh and both reports might be good.
33:17Um or one might be good and one might be bad.
33:18And that is exactly why we need evals.
33:21We can't predict the output from the input alone.
33:23Uh, and every single one of those decisions is being captured as a trace.
33:28Uh, so I'm just going to pop open my last 12 hours or my last 24 hours, I guess.
33:38Cool.
33:39This is a pile of traces.
33:40You're going to understand what all of this is later, but here you can see the Tesla one that I ran earlier.
33:45This is what it looks like when I go through.
33:47You can see every single turn.
33:49Uh it's calling a skill.
33:51It's doing a tool search.
33:52It's doing a web search.
33:54Uh you can see every single thing that it does.
33:57You can see that it ran four web searches before finally deciding that was enough information.
34:01Then it did the second uh call in the step and it wrote the report.
34:06Uh one of the things that it does uh that we're going to see later is uh I told you there was a catch in the tools that I gave it.
34:15Uh here's this step that was unexpected when we were putting this demo together.
34:20It tried to write the report.
34:22I told it to write a report.
34:23It tried to write the report as a markdown file to disk.
34:26It is living in a collab notebook.
34:27So there is no uh there's no file system for it to write to.
34:31Uh so the write step would fail.
34:33Uh and this is something that you need evals for.
34:36You need evals to detect when your agent is doing something helpful but incorrect.
34:41Uh because this fails silently.
34:43It gives you output anyway, but it's doing this unnecessary extra step of trying to write to disk uh without you knowing that it was there.
34:52Um, so back to the collab, this is a pretty good report.
34:59Um, it's a financial performance table with real numbers, growth drivers, risk factors, analyst price targets, and a conclusion.
35:06Um, it's not bad for a first pass, but it not bad as a vibe.
35:11And we are here to replace vibes with actual numbers.
35:14Um, so like I said, uh, you can see all of that stuff in the uh, traces.
35:25We didn't have to do anything to get all of that information.
35:27We didn't have to like instrument every single line to get the LLM call and get the tool call and all of that stuff.
35:32All of that is built into cloud agent SDK.
35:34just it is like it's built into open AIS SDK.
35:37Uh and each one of these rows in this table is a span.
35:42Um traces reveal every decision the agent made.
35:46Without them, all you see is I gave it a prompt and I got a report.
35:49With traces, you see every step.
35:51Uh if you click into any span, like I said, you'll see exactly what the model received as input and exactly what it returned.
35:57Um, and that is what observability means in practice.
36:04Not just did it work, uh, but how did it work and where exactly did it go wrong?
36:09Uh, when we run our eval in a few minutes, uh, we're going to pull the input and attribute output attributes out of these spans and feed them to our evaluators.
36:16The input becomes the evaluator's input.
36:18The output uh, becomes what gets graded.
36:26Uh now we need to generate some test data to run meaningful evals.
36:30Uh we need more than the one trace that you've generated so far.
36:33We need a body of data.
36:34Uh in the notebook there are 12 test queries covering different tickers and analysis types.
36:39Uh you should kick those off now.
36:41They take about 5 to 10 minutes to run assuming that you've got Wi-Fi and everything's working for you.
36:46Um I'm going to show you what they look like.
36:50Uh and oops this is them here.
37:01So the key thing here is uh diversity.
37:04I've given it a bunch of obvious queries that should work.
37:08I've also given it some edge cases.
37:09For instance, I've given it one where I gave it two tickers, Apple and Microsoft, and told it to run a comparison.
37:16Uh which is something it could theoretically do.
37:19uh I've also given it the same uh tickers multiple times uh in places so that we can see uh the non-deterministic output uh when we say uh Amazon profitability
37:30trends and outlook versus Amazon AWS performance and profitability um different tickers and different questions different levels of complexity are important uh to generate test data that's going to cover the range of things that users will actually ask although spoiler alert in reality your users are going to ask really weird things that you're not going to be able
37:51to predict in advance and that is another one of the reasons that eval are important.
37:55Um the uh Rivian query rivn uh asks about a company with much less public data than Apple or Microsoft and the Coca-Cola dividend yield query is a very different kind of analysis from a growth growth stock query.
38:10Um again this is diversity that is important.
38:14Um, diversity in your test set is how you catch uh things that your agent is going to be not good at.
38:22Uh, so while yours are running, mine have already loaded so I can show you all of my data.
38:29Uh, these are all of my spans from all of those runs.
38:33Um, you can see uh at the bottom there are 13 uh the which is the initial Tesla run and 12 test queries.
38:41Each one is a complete execution of the cloud uh financial analyst.
38:45So now we have the data but before we write any evals uh we need to actually uh look at the data.
38:53Um this is error analysis.
38:56Uh and the whole step is one instruction.
38:58Read your traces before you write eval.
39:01Before you write a single evaluator start with your data.
39:04Uh read or a dozen or more of your traces end to end.
39:07This is the most important practice in this entire workshop.
39:11uh focus on the ones where something went wrong.
39:13What was the input?
39:14What was the output?
39:15What specifically is broken?
39:17This sounds extremely old-fashioned.
39:19Just read the input.
39:21Uh but it is one of the highest value activities in agent development.
39:25Uh Anthropic, for instance, invested in tooling specifically for viewing eval transcripts and their team spends a whole lot of time just looking at eval transcripts every day.
39:34Um, a trace tells you whether the agent made a genuine mistake or whether your graders rejected a valid solution.
39:40Uh, if you automate before you understand your failures, then you're going to create an eval that measures what's easy to measure instead of what actually matters.
39:54You need requirements first before you can categorize failures.
39:57You need to know what success looks like.
39:58You can't say it doesn't work if you haven't defined what it works means.
40:03Uh for our financial analyst, what does a good report look like?
40:07Uh it should reference the correct ticker.
40:09It should include some re real recent actionable financial data uh and include actionable recommendations.
40:15It should distinguish between forward-looking analysis and historical summary.
40:18Those are our success criteria and they seem obvious uh when you write them down, but lots of teams never do.
40:26Um they ship an agent and then react to complaints instead of defining the bar up front.
40:31Uh, writing requirements down turns vague disappointment into specific testable criteria and those criteria are what turn into your evals.
40:40Here's the thing about defining success that it's important to note though, which is that it is not something that your engineers can do or not something your engineers can do alone.
40:48The definition of good usually lives in the domain knowledge and the domain knowledge lives in the people that your company will often laughably refer to as non-technical.
40:57So your product managers, your QA, your support team, those are the people who know what the definition of good really is.
41:03They know what the failures are going to look like.
41:06Those are the people who should be helping you write your evals.
41:09Uh OpenAI put it really nicely.
41:11They said that people management skills are AI skills.
41:14So clear goals, uh direct feedback, knowing what your value proposition is, uh those skills matter more than ever when the system is probabilistic.
41:23Uh and this is one of the things that AX is built around.
41:26Anyone on your team can read traces or add annotations and contribute to your criteria.
41:31This is not just an engineering surface.
41:33You're expected to have other members of your team in here.
41:38A quick note on where to get this test data.
41:40Uh we have these traces because we already ran the agent ourselves.
41:44But what if you're building something new and you don't have real traffic yet?
41:48Uh you can do what we just did, which is you can use synthetic data.
41:51You can have an LLM generate diverse queries across your expected categories.
41:55So, uh, research Tesla financial rep performance is one phrasing.
41:59What's going on with Tesla stock is another query that's asking the same thing.
42:03Yo, is Tesla a buy right now is a third way of expressing the same query.
42:08Uh, they're all the same intent, but they look very different.
42:10So, vary the phrasing, the complexity, the level of specificity.
42:14Uh, a domain expert should review your set of queries.
42:18Uh, because LLM generated queries tend to cluster around obvious phrasings and miss the weird stuff that real users write.
42:25You also need to include edge cases in your test data.
42:28So things like non-existent tickers, multi-part questions, jailbreak attempts.
42:32Uh those might be 1% of your traffic, but they are the 1% of your traffic that ends up, you know, being a PR disaster in the press.
42:39Your test data should look like production data, not what you wish production looked like.
42:44Uh so synthetic data can get you started, but as soon as you have production data, production data should be what you're running on.
42:52So let's actually examine our traces.
42:55Uh like I said, the first 13 traces are the first 13 traces that I wrote.
43:01Um most of them produced long structured reports in line.
43:06So Tesla, Apple, Nvidia, stuff like that.
43:09Uh 3 to 7,000 characters uh of report right there in the trace output with executive summaries, valuation tables, explicit buy or hold recommendations.
43:18On the surface, most of these look fine, but like I said, three of the 13 were doing something weird, which was uh they were uh the rep output in them is much shorter
43:29because it just says, "Oh, I wrote this to disk for you."
43:32Uh and the output went to disk.
43:36Uh so instead of having a full report, uh it has a summary of the report and the the actual body of the report went to the non-existent disk.
43:45This is what the eval is designed to find.
43:48Um the agent decided to write a file with no permission.
43:52The write silently failed and it told us the report was saved.
43:55Uh and that is the kind of pattern that you only see if you actually read the trace.
44:00Um so this is I believe one of the three sources.
44:05Nope, that one had worked.
44:09Oh no, the Wi-Fi.
44:10Here we go.
44:24So, this one's showing another kind of failure where it just got stuck in a loop forever.
44:28It was trying to find information about who was it trying to find information about?
44:31Microsoft.
44:32Uh, and it just did an endless series of web searches and took forever.
44:36Um, this is what it tried to write to disk.
44:41Uh, and this is what it actually wrote to output.
44:45Uh, right here we go.
44:46This is the report is saved as Microsoft cloud segment financial report.mmd.
44:50This report is useless because it's referring to a report that doesn't exist.
44:54Um, there's also uh an example of a confidently wrong.
45:00If we go to the Rivian report uh which is RIBN, where did it go?
45:08There we go.
45:09Uh, Rivian has the same problem that the Microsoft one does, but even inside of the full report that it tried to write to disk, there's a whole bunch of really confident information about a private company that you can't possibly verify.
45:24Uh, these might be hallucinations, they might not be hallucinations, but we haven't done anything to check whether or not uh this report is really grounded in reality.
45:33Uh, so we're going to do an eval about that later.
45:37There is a structured way to do this reading uh which as I mentioned earlier is called coding.
45:43There's open coding and aial coding uh and it comes from qualitative research.
45:48Open coding means you read each trace and you write down what you see.
45:52So stuff like vague recommendation made up a number talked about the wrong ticker.
45:56Uh you're not trying to be neat.
45:58As a programmer, it is very uh tempting to try and come up with categories in advance and say, "Oh, this is a you know, retrieval failure or whatever."
46:07Uh don't do that when you're doing open coding.
46:10Just write down what the problem was in as natural language as possible.
46:14Uh and uh then when you find the failure, sorry, and then uh you want to do aial coding afterwards.
46:23AIAL coding is when you look at uh all of the open codes that you've put together and you then you do the categorization.
46:29You say okay now there are five things that look similar.
46:32I'm going to give them all a single category and uh turn them into categories.
46:37Um the second pass the the uh aial coding pass is structural.
46:44Uh and like I said the tend the tendency the temptation is to try and go straight to a coding and you should resist that.
46:51When you find a failure, you should ask why it failed.
46:54So the response was wrong is a symptom.
46:56Did it get bad search results?
46:58That is a retrieval failure.
47:00Did it get the right data uh but the wrong conclusion?
47:03Then that is a reasoning error.
47:04Did it make up a stock price?
47:06That's obviously hallucination.
47:07Did it answer outside of its domain?
47:09That is a scope violation.
47:10Did it do something that you didn't tell it that it should be able to do.
47:14Each root cause, each axial code uh points to a different fix.
47:18If you don't know the right call, if you don't know the cause, you can't pick the right remedy.
47:23Once you've categorized uh which I did here, uh you can sum them up into uh a table.
47:37So, uh confession time, I didn't actually read all of those traces and code them.
47:44I just gave them random axial codes.
47:46Uh but what you end up with is this table of root cause frequency things like looks good possible hallucination reasoning gap unverifiable data missing recommendation.
47:56It's tempting to look at the top one the possible hoodation and decide that that is the most important one to fix.
48:02But in reality uh you're going to want to uh balance between uh frequency and severity.
48:12So if one time in a hundred instead of doing a financial report, it gives the user instructions on how to make a bomb, you fix that one first because that is the most severe possible failure.
48:23Uh so you have to uh multiply the severity by the frequency to get to which ones you decide to fix first.
48:30Uh and that is why you look at the data before you write your evals.
48:36Uh one more concept before we get started on that is the Swiss cheese model.
48:41um which is a concept from safety engineering.
48:43Imagine each layer of defense as a slice of Swiss cheese.
48:47Each slice has holes uh gaps where problems can slip through.
48:51Uh but if you stack enough slices, the holes don't line up.
48:54So what gets through one layer gets caught by the next.
48:56Uh your code eval catches format issues but misses semantic problems.
49:00Your LLM judge catches reasoning gaps but misses subtle hallucinations.
49:04And your human review catches the subtle stuff but it can't scale to every trace.
49:08No single eval in that set uh catches everything and the combination is what gives you coverage.
49:14Uh so stack the cheese.
49:18Um now that we know what's wrong, let's automate the checking.
49:23Um the next step is code evals.
49:26This is no model configuration, no API calls, uh just Python.
49:30We're going to write the simplest useful eval that I can think of.
49:34Uh, our agent is supposed to analyze a specific stock ticker.
49:38What I'm going to do is write a Kodi valve that checks whether that stock ticker actually showed up in the report.
49:43Uh, which is a test that my initial demos absolutely failed a number of times.
49:48Uh, an LLM judge would be overkill for this.
49:51You're looking for a specific string.
49:52It's somewhere in the report.
49:53You don't need an LLM to judge whether or not you mentioned the ticker.
49:58So, there's two ways to do this.
50:01There's two ways to do everything in AX.
50:03one is programmatically uh and one is via the UI.
50:07Uh in the uh notebook you can see the programmatic way.
50:12What I'm going to show you on screen is how you do it in the UI.
50:19So this big button in the top right is what you want.
50:22You want to add an evaluator.
50:23In this case, I've already added my mentions evaluator, but you will want to add an evaluator which will give you this screen.
50:33Uh here you're going to see basically some Python.
50:38Um in the notebook I've given you the Python that you can use uh to either do your uh programmatic version or your version in the UI.
50:47Uh this is a very simple Python function which returns uh an evaluation result which is looks exactly like those evaluation results that I showed you earlier.
51:00Uh it has a label, it has a score.
51:02Um so uh once you've written what your code evaluator does, you're going to have uh parameters to your code evaluator.
51:18In this case, I've called them query and report.
51:20And you're going to have to tell AX uh which variables in your trace are those variables.
51:26So there's this extremely fun UI, the single trace UI, uh where you look at a single trace, and you can literally just click uh the specific thing in your output that you want to be that variable, and you tell it, okay, that one should be the query, and that one should be the report, and then it's mapped.
51:47Um, you should also uh be changing your evaluator to uh work on uh a trace as opposed to a span.
51:57That is what gives you that UI.
52:00Um, once you've done that, you need to save your code evaluator to our evaluator hub.
52:06Uh, and then you hit save to save the evaluation.
52:17So assuming you've done that.
52:20Yeah.
52:28Sure.
52:30Uh so he asked for more detail on how code evaluators work.
52:33Uh so code evaluators are they're completely deterministic functions.
52:37You're just taking input and output and saying was this output good according to some definition of good.
52:43So in this code uh I'm doing very very simple Python.
52:47I'm just saying uh look for all of the words that are in capital letters.
52:52Uh strip out the ones that are obvious acronyms that I already know and of the remaining words are any of them the uh the acronym which is the stock ticker that I'm looking for?
53:02That's all it's doing.
53:04Uh lots of code evaluators are even shorter than that.
53:07They're something like is this output 500 characters long or less?
53:11Uh is this output parsible JSON?
53:14You're doing something very simple, very deterministic uh for the purposes of having that first line of defense in your Swiss cheese.
53:23Um so I talked about how to add stuff already.
53:31Uh in the notebook you can run the eval programmatically.
53:38Uh this is how to do this is these are the instructions on how to do it via the UI if you followed that.
53:43The alternative is to do it in the code where you can just run it directly which is uh easier to do if you're following along in a notebook.
53:51Um, a code evaluator returns a label and a score.
53:55Like I said, there's no LLM needed.
53:56There's no API call.
53:57It's instant.
53:59Uh, and uh if you're wondering why we're wrapping it in suppressed tracing, it's because uh the evaluation is itself uh a call to the uh agent SDK which would then get
54:12stored as traces and I didn't want traces of my tracing happening because that gets too meta.
54:17Uh so I told it to suppress tracing when I'm running an evaluation.
54:21Uh if you look at this programmatic run uh you will see that 12 passed and one failed.
54:28Uh there was a missing ticker for Amazon.
54:31Uh the specific trace that went wrong uh is when I asked you'll note you'll notice I told you earlier that I made two queries to Amazon.
54:39One where I asked about its profitability and one I asked were about uh AWS's profitability.
54:45Uh if you click into the traces uh which I could attempt now um you'll see that the report that asked about AWS uh wrote a report that was entirely about AWS
55:00only about the AWS branch of Amazon rather than uh all of Amazon.
55:05Uh and so it failed to mention the Amazon stock ticker entirely.
55:08It talked only about AWS.
55:10So it's a really subtle failure because it did a bunch of research.
55:13It found out a whole bunch of stuff about what was happening at Amazon.
55:16It just didn't think about what was happening about Amazon as a company.
55:19Uh only about AWS because I gave it that extra prompt like I also want to know about AWS profitability.
55:26Um so uh looking at those results back in AX um they show up as annotations next to our traces.
55:38So you can click into any one of these traces uh and you'll see evaluations.
55:43I've run a lot more evaluations uh than you have so far, but you can see my mention sticker evaluation has run with the span.
55:51It's run at the span level.
55:52It has hit a pass uh and it's given a score of one.
55:56Um the notebook has a little helper called log eval to ax which is doing that work for you.
56:02takes uh the spans that it ran offline and the tests that it ran offline and pushes them back to AX for you.
56:08Um so now every score now every eval has uh a mention sticker evaluation uh that you can look at and filter by uh and look for the failing failing ones.
56:22So why does this matter?
56:23Because it is your first line of defense.
56:25Uh if your financial analyst agent writes a beautiful report about m Microsoft when the user asked about Tesla uh which is a thing that actually happened to me when I was building this
56:35uh then no amount of answer of eloquence from your agent uh matters because the answer is wrong.
56:40Uh so you can catch this with five lines of Python.
56:43Um the ticker check catches uh really basic failures but it catches them really cheaply.
56:51That is why it is important.
56:53Um, Kodi valves aren't just toy examples.
56:56Often, you're going to want to know that your output is JSON, uh, or that it has a certain length, or you're going to want to avoid forbidden phrases like as an AI language model.
57:05Uh, and those are production critical checks.
57:07A code eval doesn't have to be a simple string operation.
57:10Kodval means the grading logic is deterministic.
57:14Uh, but it can do complicated things.
57:15It can query a database for you.
57:17It could hit an API and get the actual stock price to make sure that it wasn't hallucinating a stock price.
57:22uh anything with a grading always gives the same answer for the same input can be a code evaluator.
57:29One important principle is to grade what the agent produced and not the path that it took.
57:34Uh I referred to this earlier.
57:36There's a common instinct to check that the agent followed a specific sequence of steps.
57:40Did it call this tool first and then that tool in that order?
57:43Uh and uh practitioners have found this to be in practice too rigid.
57:48uh agents regularly find valid approaches uh that the eval designer didn't anticipate.
57:54Uh so uh if you grade the path um you punish creativity.
58:00Instead grade what the agent produced.
58:02Did the output have the right information?
58:04Did the final state match what you wanted.
58:06Uh our ticker check doesn't care how the agent found the data.
58:09It just checks is the ticker in the output.
58:12Uh that is the right level of abstraction for most Kodi valves.
58:17So now let's get to built-in evals.
58:19LLM evals.
58:20This is where it begins to get more interesting.
58:23Uh code can check whether the ticker appears but it can't check whether the analysis is good.
58:28Is the financial data accurate?
58:29Is the report complete?
58:30Are the recommendations well reasoned?
58:32Uh these are semantic questions and for semantic questions you need a judge.
58:37Every LLM as a judge has three parts.
58:40Uh it has a judge model which is the LLM that is doing the grading for you.
58:45It has a prompt template which as I mentioned is also called a rubric.
58:48Uh which is the criteria that the judge applies the rules of what defines good.
58:53Uh and it has the data which is the examples being evaluated.
58:56AX keeps these three things separate which means that you can mix and match.
59:00You can try the same criteria with different judge models.
59:02The same judge model with different criteria.
59:04Uh it is modular by design.
59:07The good news is that AX ships with a B with a bunch of built-in eval so you don't have to write your own.
59:13There are a bunch of types of evals that are always uh that are often going to apply to uh most AI applications.
59:21So we've already written the prompts for those and you don't have to write them yourself.
59:26Uh correctness, for instance, checks whether a response is factually accurate.
59:30Faithfulness checks whether the response stays grounded in the source material.
59:34Uh and there are evals for tool selection.
59:36Did the agent pick the right tool?
59:38Uh and tool invocation.
59:40Did it pass the right arguments to that tool?
59:42stuff like document relevance, refusal detection, many of the things that you'd want to check in an agentic application uh come out of the box.
59:50Uh so the first eval that we're going to run uh is a correctness eval.
59:58Um and spoiler alert, it's not going to work.
60:02Um setting it up is very easy.
60:06We give it an LLM.
60:07In this case, uh I've chosen uh the anthropic LLM.
60:11uh sonnet 4.6 because it is better than haiku which is the thing that is doing the work.
60:16You often want to pick a bigger slower model to be your judge.
60:20Uh and then I've just uh given correctness evaluated the LLM uh and uh run it right here.
60:29Um like I said uh once we've run the evaluation with evaluate data frame uh we've also uh turned on suppressed tracing here so that our evaluation doesn't send a bunch of extra traces to uh AX.
60:48Um it gives us this output which is a bunch of evaluation rows.
60:53Um these are not very readable.
60:56This is why you need AX to look at them.
60:58Um the judge reads the input and output from each span and grades it against the rubric.
61:03Um and we can see that every score is zero.
61:10If I scroll across, status completed, name correctness, score zero.
61:19The reason it's doing this is because correctness is judging against the uh LLM's definition of of what is correct and what is true.
61:29Uh which means that it is relying on the LLM's training data.
61:34What we're doing in this agent is a whole bunch of live web search to get current stock prices and up-to-date financial information, which means the agent is talking about things that happened in 2026,
61:46but the LLM that's doing the judging only has information that stops in January.
61:51So, if you read the explanations of our correctness eval inside uh inside of AX, you can scroll to evaluations, you can look at correctness, you can say that it is incorrect.
62:02uh I will save you a bunch of reading of a very densely packed column.
62:05What it's doing is complaining that there's a bunch of stuff that comes from the future that it couldn't possibly know.
62:11Uh so our correctness eval here has completely failed.
62:14This is not what you wanted to do.
62:16So instead we are going to use a faithfulness eval.
62:19Uh faithfulness is a better eval for our use case because uh the judge actually has context.
62:25In faithfulness, you give the uh LLM as a judge the information that it is being based on uh and then say based on the same information that the jud that my agent collected,
62:37did it do a good job of writing the report?
62:40This is this way they are on a level playing field.
62:42They both have the same amount of information and your slow complicated judge agent uh can judge whether your uh quick fast uh worker agent uh was uh doing a good job.
62:55Um so let's look at the faithfulness cells.
62:59Uh just like everything else, these can be done online as well as offline.
63:07So uh there is a faithfulness built-in evaluator uh which isn't in this view.
63:14There is a faithfulness built-in evaluator which you can pull up uh and you can give your input, output and your context to it.
63:22Uh again, you can do this in code as well.
63:25And if you're using the skills, this is probably what's going to happen.
63:27Your skills will say, "Oh, I'm not going to use the UI.
63:30I'm going to use the code."
63:31Uh which is why I show it here.
63:34Um so the first thing we do is we extract the research context from the traces.
63:39As you'll remember, we do uh a two-step process.
63:43So I take the output from the first step where it does the research and I attach that to all of my spans uh as uh the context for it the judge to work on.
63:55Uh and then I run the faithfulness evaluator.
63:58So uh here you can see spans with context.
64:02These are the same spans that I just pulled out of AX.
64:05I attached at the column level all of my context.
64:08Uh and then uh I called evaluate dataf frame with those spans and I gave it my faithfulness evaluator which I instantiated here just like I instantiated the correctness evaluator.
64:21This gives us uh much more interesting data.
64:25This gives us unfaithful seven and faithful six.
64:29Uh so uh roughly half the time my very simple report is hallucinating.
64:38It is coming up with things that were not in the source data according to sonnet.
64:43Um and that is this is a useful eval.
64:48This is an eval where we're failing half the time.
64:50This is a capability eval where we can climb the hill, modify our prompt and get it better at writing a grounded report.
64:56This is exactly the kind of eval that you want.
64:59Um, so that's two built-in evals and two very different signals.
65:07And there's a really important lesson here, which is correctness gave us zero out of 13 and faithfulness uh told us that reports aren't grounded in their sources roughly half the time.
65:16The difference isn't that one eval is better than the other.
65:18The difference is uh the use case.
65:20There are lots of use cases where that correctness eval that I showed you earlier works just fine for some use cases.
65:26Uh it's just not in this specific use case where we were doing live real-time research.
65:32Um so it's important to know the question that you're actually asking.
65:38Built-in eval are your starting point.
65:40They give you an immediate signal without any prompt engineering.
65:43Uh but even with a good built-in like faithfulness, uh there are a bunch of things that it can't check.
65:49uh our financial analyst uh should produce actionable recommendations.
65:54It shouldn't just summarize data.
65:56Um so not just you know give you a financial report but tell you what to do with the financial report.
66:02Should I buy, sell or hold?
66:04Uh no built-in template checks for that.
66:07This AX is does not cover every single use case.
66:09So you have to build a specific eval that checks for actionability.
66:13You have to build an custom eval that is about your domain.
66:17If you are writing real eval, you're absolutely going to have to write a custom eval with a definition of good that we didn't think of.
66:25So that brings us to the next step, which is writing a custom eval rubric.
66:28This is where you make eval truly specific to your application.
66:32And writing a good eval is complicated.
66:35So I'm going to spend a little time talking about what makes a good eval rubric.
66:39Uh we recommend four parts of a custom rubric.
66:43The first is to define the judge's role.
66:46Um, you should spell out ex explicit pass and fail criteria.
66:50You should label the data with XML tags.
66:52Uh, and you should define the output choices outside the prompt itself.
66:56Uh, I'm going to add one more thing on top of those which is actually kind of controversial amongst practitioners, which is I tend to include examples of good and bad output.
67:06Uh, some practitioners believe uh that including examples will make your uh, eval overfit.
67:13Uh but let's walk through all of those.
67:16So the first part is defining the role.
67:19Um you'll notice this uh what is by now a tired trope which is you are an expert financial analyst evaluator.
67:26That is not the important part.
67:27Tests show it makes some very small difference uh to whether or not uh your eval works.
67:34The important thing is to give it the context of what it is looking at.
67:37You are looking at a financial report and what I am expecting is financial analysis.
67:41Those are the important pieces of context that you are giving the LLM there.
67:45Um you're also telling it that it should be that what you want is something actionable.
67:51So that brings us to part two.
67:53So uh this is where most people underinvest.
67:57Don't say a good response because that is uh an aspiration.
68:01That is not a criterion.
68:02A good response is helpful and accurate sounds like you're adding more detail, but you're still being just as vague because helpful and accurate don't have any meaning as far as the LLM is concerned.
68:12Uh instead, you should be listing exactly what makes a report actionable.
68:15So list exactly what makes it actionable and not actionable.
68:19Every criterion should be specific and observable.
68:21Something that if you were a human looking at this report, you'd be able to know whether or not it was there and you'd be able to say yes or no.
68:28Uh so contains specific recommendations.
68:32That's something you can definitely say yes or no. Includes forward-looking analysis, not just historical data.
68:37That's a clear distinction that you can say yes or no to.
68:40Uh and the important thing here is that each of these criteria maps to something we saw in our analysis.
68:48If we when we were reading the reports uh we saw we actually observed when we read our traces or at least when I read our traces uh that uh these are ways that these reports are going wrong specifically that is not a coincidence.
69:00Your error analysis is what tells you what criteria to write.
69:04You don't come up with them from whole cloth because it's not going to occur to you.
69:07Where you get that data from is from reading your traces and going oh it didn't do that.
69:11My axial coding said it does this specific wrong thing.
69:15Uh so specific criteria produce consistent judgments uh while vague criteria produce coin flips.
69:22The third thing you should do in your rubric is you should label the data with XML tags especially if you are using anthropic.
69:28Anthropic loves XML and finds it very easy to understand.
69:31Um your rubric has variables the input and the output and you want to make sure that it can tell the difference between your instructions and the input and the output and XML tags are an excellent way to do that.
69:43Um, so put a user query tag around the input, a financial report tag around the output.
69:48Uh, clear boundaries reduce the chance that the judge confuses the question with the answer.
69:52It is especially important if you're including examples uh to label the examples with XML tags because otherwise it will go, "Oh, this is exactly what I'm supposed to write and just spit out the example back to you."
70:04Um, which brings us to the examples.
70:07This is the part a lot of people skip.
70:09Uh, in my opinion, it is the part that most improves quality.
70:12LLMs are okay at following instructions.
70:16They are increasingly good at following instructions.
70:18But what they are really good at is copying an example that you've already given them and replacing the things that are relevant to the current situation.
70:26So if you give it an example of exactly the kind of report that you're looking for, it's going to follow that format exactly.
70:32That might or might not depending on your domain be a good idea.
70:36If you if your example is too rigid, then the LLM will always give the same report and you have reduced its scope for creativity.
70:44Uh but on the other hand, if it is being too nondeterministic and your report is being crazy, uh then giving it an example will ground it in what exactly you are expecting.
70:55Uh here's what an actional example looks like in the template.
70:58Uh it has everything.
71:00It has uh percentage revenue growth, a PE ratio.
71:03It identifies a concrete risk with a number to back it up and it gives a specific recommendation.
71:08Uh that is what actionable looks like.
71:10Here is a non-actionable result.
71:12Uh it's not wrong.
71:13Uh everything it says is correct.
71:15Uh but it doesn't say anything useful.
71:16It doesn't have any specific data.
71:18It doesn't have any specific risks.
71:19It just says investors should consider various factors.
71:22Uh that's not a recommendation.
71:24Uh so the judge now has a concrete reference for what you actually mean by the labels that you were giving it.
71:31Uh and this can dramatically improve consistency especially on the edge cases where the report uh would have otherwise been somewhere in between.
71:40Uh and this is uh an important part.
71:43It is very tempting if you're writing this rubric to say and now spit out the word correct or incorrect because that is what you wanted to do.
71:51You do not need to do that anymore.
71:53Uh the way that it actually works under the hood is we have given the LLM a tool that says you call this tool when you are done and it will call that tool with correct or incorrect when you're done.
72:03So you don't need to tell it what output and in fact telling it to spit out a specific string as output is more likely to confuse it and make your eval fail.
72:12So just in the UI uh set your uh choices uh or when you are setting the when you're creating it programmatically set the choices uh and let the LLM do its thing.
72:25Um when you are coming up with these choices it is best to keep those choices simple.
72:29Uh binary is best two clear labels.
72:32Uh numeric scales like a scale of 1 to five.
72:36Uh they are tempting but the LLMs are bad at them.
72:40what is the difference between a three and a four?
72:42You would have to write a whole bunch of rules for the LLM to know that.
72:45Uh it doesn't know that automatically.
72:47Uh whereas you can give it a pass fail and it will be pretty good at deciding what is a pass and what is a fail.
72:53Uh so if you really need more categories, you can do like a partial result.
72:58You can say a 0.5 score is where it did a a half-ass job.
73:03Uh but by default, go for a zero or one.
73:08Um, one more practical tip is chain of thought for judges.
73:11Ask the judge to reason before it scores and have it explain its thinking first.
73:16Uh, this measurably improves uh, accuracy.
73:20Uh, when the judge has to articulate why something is actionable or not or why it is correct or not.
73:25Uh, it makes fewer mistakes than when it just picks a label uh, without thinking first.
73:30Uh, this explanation is also very helpful for you.
73:33Uh, but it also helps the judge think better.
73:35Just one of those weird things about LLMs.
73:38Uh so if you are coding along, now is the time to find your actionability template.
73:45Uh this contains all of the example all of the instructions that I uh was just talking about.
73:51Uh can you do better than me?
73:53Can you oneshot your way to a perfect eval uh by modifying this eval template?
73:59Um, as with uh all things in AX, you can do this via the UI as well.
74:05So, you can create an online evaluator uh and you can create a new LLM as a judge.
74:10In this case, I've already got it, my actionability evaluator uh which has literally exactly the same text uh that I just showed you in the notebook.
74:21Um, and as before, I had to pick a trace uh scope.
74:27Uh, and because it's an LLM as a judge, I had to set up a template.
74:30Uh, sorry, I had to set up an LLM for it to use in the UI.
74:35Uh, which means that you need to give it your anthropic key.
74:37Again, uh, and I picked my inputs and outputs via the single trace method.
74:43Uh, and then I saved it.
74:45One of the things that you can do with online evals uh is you can set them to run continuously against new incoming traces.
74:52This is something you'll almost certainly do in production.
74:55Uh and if your evals are expensive or you have a whole lot of traffic, you won't want to do it on all of your traffic.
75:01You can uh set it to serve to uh rate only a percentage.
75:06That's the sampling rate.
75:07So you can say, you know, just look at 1% of my traffic and that will still give you uh a whole bunch of eval data without breaking your budget.
75:17Uh so back to the code version, wiring it up is just like wiring up the other action the other uh evaluators that we've given.
75:29Uh we've created a classification evaluator.
75:32We've given it a name actionability.
75:34We've given it an LLM.
75:35Uh we've given it that template that I just showed you and we've given it those choices.
75:39This is us programmatically doing what we would have done in the UI where we said your choices are correct or incorrect.
75:45Uh actionable or not actionable scores of one or zero.
75:48Uh and then uh we ran evaluate data frame.
75:52That gives us uh the data that I was skipping over because I was already run earlier.
76:02So you can see actionability actionable.
76:05If we look at our evaluations tab, we can scroll down to our actionability and we get this really great explanation about why the report is actionable or not actionable.
76:15Uh lots of our reports scored actionable uh but several are not which is awesome because it means that we have again we have a capability eval uh and a hill that we can climb.
76:32Um, so in the notebook, uh, this is where you log your results back to AX or if you've run it online in the UI, your evals, your eval results are already in the UI as you've seen.
76:44Um, so you can see now in in AX that every span has multiple eval scores.
76:50It has mention sticker, it has correctness, it has faithfulness, it has actionability.
76:54uh and you can filter and select uh for specific uh eval results.
77:03So, eval.actionability.label equals not actionable.
77:21that finds you the failing set.
77:23This is the set that you're like, okay, my eva my eval says that my code messed up here.
77:29I can turn this into a data set and I can start running uh new prompts against this.
77:33I can run experiments against this as we're going to see.
77:37Um before we move on, I want to flag a few a few common antiatterns uh that turn evals from a useful tool into noise.
77:49Uh the first one is to treat your eval prompts like code.
77:51And I mean that literally.
77:52They're very sensitive to war to wording.
77:55Small shifts in your eval prompt uh can lead to dramatic changes in your eval results.
78:01Uh so you should version them.
78:02You should test them on examples where you know the right answer.
78:05If the judge if the LLM as a judge disagrees with your human labels uh on 40% of examples, then your eval is probably wrong.
78:13It's not that the your results are wrong.
78:14It is that you have written the eval incorrectly.
78:17uh AX has a prompt playground where you can iterate on uh rubrics without touching your code and it's very useful for doing this kind of iteration of your eval.
78:27Uh the second antiattern is the god evaluator.
78:31Um it is tempting to build one big mega evaluator that checks everything at once.
78:36It'll look for accuracy and tone and completeness and policy compliance and formatting all in a single prompt.
78:41Don't do this because it is a nightmare to calibrate.
78:44uh when your god evaluator says fail, which of those five dimensions failed?
78:49You don't know.
78:49So suddenly you're like, "Oh, well, I'll have it spit out labels."
78:52And then you're like, "Oh, it's spitting out 15 different labels."
78:55Suddenly it's way too complicated.
78:57Uh you can't tell, you can't judge by it, you can't score by it.
79:01It is much much simpler and easier to split this up into uh one evaluator per dimension.
79:07I've already done this today.
79:09I have a ticker check.
79:10I have faithfulness.
79:10I have actionability.
79:11Three separate focused evals.
79:13and each one tells you something specific.
79:16Uh the combination gives you a richer picture than any single eval ever could.
79:21If you find yourself writing a rubric that lists six different things that the response should do, then stop.
79:26That is six evaluators, not one.
79:30Uh the third one is know your guardrails and your northstar metrics.
79:33Some evals are guard rails.
79:35They're ship blockers.
79:36If an agent hallucinates a stock price, that is a hard fail.
79:39Don't ship on that.
79:40Uh others are northstar metrics.
79:42they are aspirational.
79:44So always recommend complimentary investments is a nice to have in this case.
79:47It is not a dealbreaker.
79:49Uh so know which of your eval is which.
79:52It changes how you act on the results and later it changes how you set up your production monitors.
79:56A guardrail uh regression should page someone in the middle of the night.
80:00A northstar dip is something that you look at in a weekly report.
80:06Which brings us to the next section which is can you trust your judges?
80:09This is a question we've been dancing around.
80:11I've mentioned it a couple of times.
80:13Uh it's time to evaluate your evaluator.
80:16We built an LLM judge and it told us that some reports are not actionable.
80:20Uh but can we trust it?
80:22How do we know that the judge is correct?
80:25Uh the key mental model here is that your judge is a classifier.
80:29So you can treat it as a classifier.
80:31It takes an input in this case a financial report and it makes a prediction actionable or not actionable.
80:37That prediction can be right or wrong.
80:38So just like any classifier, you can measure its performance by comparing its predictions against ground truth.
80:44In this case, a human's judgment.
80:47Uh you can check the judge's homework.
80:50Um the thing about applying judgment is that it is a ton of work.
80:54You have to read actually every single output.
80:57Uh you have to think hard and apply a label.
80:59Uh AX does what it can to help by letting you define annotations and attaching them to spans uh right in the UI.
81:06So, let me show you what that looks like.
81:11When you're looking at a span, you can click annotate span and you can add and remove annotation configs.
81:17In this case, I've already created one called human actionable.
81:20And you can set this span as being actionable or not actionable.
81:23And then you can just use these little arrows to flip through all of your traces and set them as actionable or not actionable, which I have conveniently already done.
81:31Uh this is the kind of work that you can give to a nontechnical member of staff.
81:36That is why the UI looks like this.
81:38They can just click in, read the report and click a checkbox.
81:42They don't have to write any code.
81:43They don't need to know anything other than their domain expertise of whether or not this report should be considered actionable or not actionable.
81:52What we're doing here when we add human annotations is building a golden data set.
81:56Building golden data sets is incredibly helpful uh because it's how you measure uh whether your evaluator is actually doing its job.
82:03Uh the way to do it is the same thing uh the LLM did.
82:07Give yourself real concrete criteria and stick to them.
82:10So don't say this was good, this was bad.
82:12Uh be specific the same way that you've told the LLM to be specific.
82:15You failed at this specific thing.
82:17You asked I this thing was supposed to be present and it wasn't.
82:21uh and eliminate the chance to get lazy uh when you get tired of labeling.
82:26Um as I said uh the labeling is extremely uh timeconuming so I did it already.
82:36Um once I've labeled some spans in the UI uh I can reexport the spans into the notebook.
82:42Um, and this is a nice thing about AX is that the human annotations that I added by hand come back as columns uh in the export.
82:51Um, so we filter down to just the rows that I labeled here.
83:04Uh, and now we're going to compare uh what I said was actionable and not actionable to what the judge says is actionable or not actionable.
83:15Um, before we talk before we do that though, uh, a little bit more about golden data sets, uh, keep your tasks unambiguous.
83:25If your agent scores 0% consistently, that's almost always a broken task.
83:30That's almost always, uh, you've written your rubric incorrectly.
83:34uh if you've made your task something that only a human can do or worse that somebody something nobody could really do uh then you're going to get a zero score.
83:44Um so for each task create a reference solution a known working output that passes all of your graders.
83:49This proves that the task is solvable and you should also test in both directions.
83:54You should have cases where a behavior should occur and cases where it shouldn't.
83:58So if you test for instance does it search the web uh then you will end up with a uh an agent that always searches the web even when it doesn't need to.
84:09Whereas you should also have a test that says did it not search the web for this very obvious answer where it didn't need to do that.
84:16Uh and if you're doing this for real you should split your label data that you just created uh into a development set and a test set.
84:24Uh so use maybe 75% of your labels to it to to iterate on your eval uh tweaking criteria, adjusting examples until the judge agrees with you.
84:33Then hold out the remaining 25% to run the judge and see that it actually gets the scores that you are expecting.
84:40Uh that's how you know that the judge is actually generalizing to new examples instead of just fitting to the specific data set that you gave it.
84:47Uh this is the same principle as train tests, splits in machine learning.
84:51Don't overfit your evaluator to your golden data set.
84:56Now let's look at our actionability judge.
85:01I have run the judge on some examples and it has come up with uh a list of where they agree and disagree.
85:08Uh spoiler alert I just hit actionable and not actionable at random uh so that I would get lots of disagreement because actually Sonnet is pretty good at this stuff.
85:17Um that gives us uh an agreement rate of 46% six out of 13 times.
85:23Um in real life uh this is exactly where you dig in.
85:32This is where uh you would ask is my rubric wrong or were my human labels wrong.
85:37When the judge disagrees with you, you should read its explanation and decide whether or not the judge was right or you were right.
85:43Uh that is the most valuable output of meta evaluation.
85:46It usually reveals an ambiguity in the rules that you set down.
85:51To fix a rubric, you should read the explanations on the disagreements, find the ambiguity, uh, and tighten the criteria.
85:57So, instead of just includes forward-looking analysis, I'd write includes forward-looking analysis with specific recommendations or guidance.
86:05That's more precise.
86:06Uh, it rules out the case where the report talks about the future, but never tells the judge what to do about it.
86:12Uh then I'd rerun the judge on the same examples and see if the disagreement goes goes away.
86:16Uh this is rubric iteration.
86:18Your eval prompt is an application.
86:21Uh it is another LLM application.
86:23Uh so it needs testing and iteration just like your agent does.
86:26Uh and that is what I meant when I said that you should treat your evals like code.
86:32So now I'm going to talk very briefly about an ML concept called precision and recall.
86:37Uh, precision asks, when the judge says not actionable, how often is it really not actionable?
86:45Uh, and recall asks, of all the reports that are actually not actionable, how many did the judge catch?
86:52Uh, with a tiny sample like uh 12 examples, these numbers are going to judge jump around a lot.
86:57Uh, but even with the 12 that I have, uh, you can see whether the judge is in the right ballpark or completely off.
87:05In most scenarios, you want to uh you want to prioritize recall.
87:10These two measurements are at odds with each other.
87:14If you are if you prioritize recall, then you're going to get false positives where the judge has said something is wrong when nothing is wrong.
87:23That is much better than accidentally letting through things that are wrong.
87:27Uh however, some in some cases when you are tuning, you will say actually I want precision.
87:32I want it to be absolutely right when it's right and I don't care if it misses some things.
87:36There are domains where that is the output that you want, but most of the time uh you want to prioritize recall.
87:43Um a few known pitfalls with using an LLM as a judge.
87:50Um one is position bias.
87:52If you present it with two options to judge between, uh the judge tend to favor tends to favor whichever comes first.
87:58Uh this depends on the model.
88:00Sometimes the model depends uh prefers whichever comes last but it always has some kind of preference to position.
88:07Um there's also length bias.
88:09Uh LLMs like longer responses.
88:11They tend they tend to score those higher and that say that they are better even if the length extra length is just filler.
88:17Uh and the last one I love which is confidence bias.
88:20The judge gets fooled uh by a response that sounds confident just like humans do.
88:25If your LLM says things that are wrong, but it says them in a really confident tone, uh, your LLM judge is more likely to believe them.
88:33Um, there's also self-preference bias.
88:36If you use the same model to do the generation as you use to do the judging, they tend to like their own output.
88:41So, if possible, use both say OpenAI and Anthropic.
88:45Use one to do your agent and one to do the judging and they are less likely to agree with each other.
88:53uh self-preference bias is why I used sonnet as a judge uh for output generated by haiku.
89:00Uh but it works even better if you use a different uh labs model entirely.
89:06And remember that the benchmark here is human performance.
89:09Humans are not going to be perfect at judging whether things are or are not actionable.
89:14They are going to get it wrong quite a lot of the time.
89:17What you're trying to do is come up with an eval that is reasonably close to human judgment.
89:23Uh human judgment also varies from human to human.
89:26Human inter interrator reliability, so two humans judging the same thing uh is often as low as 02 or.3.
89:33So two experts with the same output and the same rules disagree a surprising amount of the time.
89:39So if your judge, if your LLM judge achieves that uh higher consistency than humans, then it is doing better than humans.
89:46the judge uh sometimes disagrees with me is not by itself a reason to distrust it.
89:52You what you're looking for is a judge that disagrees with you all of the time.
89:56That is the danger signal.
89:57A judge that disagrees with you some of the time is a judge uh correctly handling a genuinely ambiguous task.
90:06Uh and also failures should seem fair.
90:09That is a principle from Anthropic Eval teams that I want to leave you with.
90:12When a task fails, uh, it should be clear what the agent got wrong and why.
90:17If you look at a failing trace and you think that answer looks fine to me, the problem is probably the eval and not the agent.
90:26So, uh, now we've built evals, we've tested them, uh, let's use them to actually improve our agent.
90:34This is how you close the loop.
90:37Uh, here's the problem with one-off fixes.
90:41Um, if you found some failures, you've read some explanations, you know what to improve, you change the prompt, and then what?
90:47How do you know uh whether the fix actually worked?
90:50How do you know if it didn't break something that was working before?
90:53If you just run the agent again on a couple of examples and eyeball it, then you're back to vibes.
90:58Uh, you need a structured way to compare before and after.
91:02And as I mentioned earlier, this is what experiments are for.
91:06As I showed you earlier, uh you can filter your traces down to just the ones that are failing and you can turn them into a data set like this.
91:16So you select all of your traces, you click add to data set, and you create a new data set.
91:21In this case, uh I've called it AIEWF financial demo fails.
91:25Uh and I've already created my failure data set.
91:29But this gives you a smaller set of places where you know that your eval is failing so that you can then run uh your changed prompt against just them.
91:39This is faster and cheaper.
91:41Um this is a regression test set.
91:45uh sorry you should also save a regression test set rather you should get the ones where it is not failing and make sure that those are also a data set that you can check less frequently to make sure that when you've changed your model sorry when you've changed your rubric
91:59uh that you haven't accidentally made it fail in places that it was succeeding before um and your data sets are not static um pre-production when you don't have users yet your data sets are probably going to be synthetic queries you generated yourself
92:17or queries that you got an LLM to generate for you.
92:20That is fine.
92:20It gets you off the ground.
92:22As soon as real traffic starts coming in, you should turn your uh that into your eval sets.
92:28Uh real failures should replace imagined ones.
92:31The union of your failure cases and your passing cases is what people call your golden data set.
92:36Uh the curated labeled set you trust as ground truth for measuring quality.
92:42Uh now we're going to improve our agent.
92:45Uh but we are not going to handw write the fix.
92:49Uh as I mentioned, you can do all of this with skills.
92:52You can do all of this with a coding agent.
92:54I am assuming it being 2026 that absolutely everyone is using a coding agent to do all of their coding.
92:59Uh so why would you modify your prompts by yourself when you can get cloud code to pull down your failing traces uh modify your code for you uh and improve your application for you?
93:11That is what we're going to do here.
93:13Um Claude will pull down all of the failing traces.
93:17It will read all of the explanations.
93:18It will turn the explanations into action action that it takes on your codebase.
93:24Um and here's the important thing.
93:27Nothing here is guessed.
93:28Claude isn't inventing improvements out of thin air.
93:31You haven't just told it get better at this.
93:33You haven't just told it make no mistakes.
93:35Uh every change it proposes is traces back to a specific failure explanation.
93:40Uh so if the judge said that it lacks specific recommendations, uh Claude will read that and rewrite the prompt so that it does.
93:48Uh if the judge said that it presents risks without supporting evidence, then it will rewrite the prompt so that it fixes that too.
93:54This is datadriven prompt engineering.
93:57uh and now we can automate it.
93:59Uh so notice what we've passed in.
94:05Uh we've passed in our requirements, what we were trying to get done.
94:08Uh and we've passed in uh an improvement prompt telling it uh what you should be working on and how to get better at it.
94:17This is yet another LLM application and yet another thing that is nondeterministic and yet another way that it can go wrong.
94:23You could, if we had more time, meta-evaluate your meta evaluator of your meta evaluator and make sure that your meta evaluator is doing the right thing.
94:31But we're not doing that here because it gets too complicated and also we only have so much time.
94:36Um, but uh good requirements keep claud.
94:42So you should be spending time making sure that this uh evaluator uh is as accurate as you can make it.
94:50Uh so in this case we are taking this prompt and we are wiring them into uh our two pass uh we're taking the recommendations that we got from Claude based on pulling the
95:05uh failing traces uh and we are passing them back to anthropic and saying based on these explanations write a better prompt write a better research prompt write a better write prompt and then we're going to feed them back into our agent.
95:18Um, and now we're going to run an experiment.
95:24So, uh, over in the UI, we have data sets and experiments.
95:28We have my demo financial fails.
95:31Uh, we can, assuming I can scroll, wire up an improved agent, uh, using those new improved prompts.
95:44So, you can see the improved research prompt and the improved write prompt.
95:47this is otherwise exactly the same agent that it was before.
95:50Uh and you can create an evaluator uh with an experiment.
95:55Uh so what we're doing here is we're giving it a task.
96:01A task can be anything.
96:03So a task could be some subcomponent of what your agent did.
96:07So if we were running an evaluator that was about uh tool calls then your task could be just call this tool and see what the output of the tool is and then you would evaluate that.
96:17In this case my task is run the entire agent again from start to finish but it doesn't need to be uh and then we are running the same evaluation results
96:27uh against our uh against this new running of the task.
96:32So, we've changed the agent.
96:34We've rerun the agent, and we're rerunning the eval.
96:36And we're turning that into an experiment.
96:39This is what the experiment looks like.
96:41Experiments.run.
96:42We've given it a name.
96:44We've given it the data set that it should work on.
96:46Uh, we've given it the task.
96:48We've given it the evaluator.
96:49And we've told it to run.
96:51Uh, this gives us the results which are here.
97:00So for every single run uh we've got the output that it got and the new eval score.
97:08So uh as you can see this was a data set if you recall where previously all of them were not actionable.
97:16Uh the eval has run and you can see that some of them are now actionable.
97:19So we have improved and if we look at the overall score here actually I've nailed it because I had you know a lot of chances to get this right.
97:28I have oneshotted my prompt.
97:30uh from getting being wrong 50% of the time to being right right 100% of the time.
97:36Uh real eval won't work like that.
97:38Uh real iteration won't won't run so smoothly.
97:42That is why this is a graph.
97:43What you're expecting to do is run experiment after experiment and slowly watch the graph climb as your evals get more and more capable.
97:51That is the hill that I'm talking about that you're trying to climb.
97:54Uh so the key abstraction behind experiments is that a task is just a Python function.
98:02It takes an example from the data set.
98:04It runs whatever you run want and it returns an output.
98:07So AX doesn't care uh what the task does internally.
98:10So it could run the full agent or just one component like I said or it could call an API as long as it takes an input and returns an output.
98:17You can run experiments on it.
98:20The power of experiments is controlled comparison.
98:23So you take the same inputs, the same evaluators and the only thing that changed is the agent's prompts.
98:29Uh that means any differences in scores is attributable to your change.
98:33You're not wondering did it score higher because of my prompt change?
98:36Uh or did it score higher because the web search happened to return better results this time.
98:41Well, actually you're still wondering that a little bit because the agent is nondeterministic.
98:45So it could have done 10 web searches instead of five web searches.
98:48But you've eliminated a major source of variation which is the test cases.
98:55So like I said, uh my agent has one shot at this uh which is great, but in reality uh it's not going to be that simple.
99:04Um the eval iterate cycle that we're talking about here where you go through uh run an experiment, change your prompt, run the experiment again over and over and over.
99:15That is where the real value lies.
99:17Uh not in the score itself, but in the cycle.
99:20You run the evals, you look at the failures, you read the explanations, uh, you identify a pattern, and you adjust your prompt or as in this case, you get Claude to adjust your prompt for you.
99:30Um, did it get better is a question that can only be answered by evals.
99:36You have taken the definition of good and you have codified it in a way that is incredibly useful.
99:42Uh, so this allows you to move from I think it's working to I can prove it's working.
99:48A practical question that comes up fast uh is how many samples do you need?
99:52Um for workshop scale experiments like today, 12 to 20 examples gives you uh directional signal.
99:58For actual shipping decisions, you're going to want uh more.
100:01You're going to want something like 200 to 400 is a good target.
100:04But note that 200 to 400 is still like a humansized number that you could possibly accumulate over a couple of weeks.
100:10Uh it's not tens of thousands or millions like you would use if you were training an ML model.
100:15um there are diminishing returns.
100:18Uh to have your your margin of error, you have to quadruple your sample size.
100:23Uh so at some level, at some point, you have enough signal to be able to get along uh to get along.
100:30Um when you're iterating, where do you invest?
100:33Uh there's a hierarchy.
100:35Um data quality fixes have the highest impact.
100:38If your agent is searching the wrong sources or your knowledge base has stale information, then no amount of prompt iteration is going to help.
100:44you should fix the data first.
100:46Uh then prompting improvements are next.
100:48Few shot examples, explicit instructions, the kind of stuff that I mentioned today, uh constraints on what the agent should and shouldn't do.
100:56These are often the highest ROI change uh to your agent.
101:00Um model selection comes third.
101:03Sometimes a more capable model solves problems that prompting can't.
101:07Uh but it will also cost more and be slower.
101:09So that is a trade-off that you have to make.
101:11Uh the thing to not spend a lot of time on is hyperparameter tuning.
101:15Things like temperature and top p. It's very tempting to just go for them because they're easy to tweak.
101:20Uh they very seldom make a high lever change like prompting would or improving the quality of your data would so you can try them but they should be the last thing you try.
101:31A practice worth knowing is eval driven development.
101:33Write the eval before you build the feature.
101:36Uh if you want your agent to always verify customer identity before it does something before processing a refund for instance then write the eval that checks for that first.
101:44That gives you a clear measurable definition of what good looks like of what done looks like.
101:49Then you can build a feature uh until the eval passes.
101:53This is the same philosophy as test-driven development.
101:59Uh and something I love about eval driven development is that the people closest to product requirements are best positioned to define success.
102:06So product managers, customer service uh people, even salespeople can contribute to eval tasks.
102:13They can tell you what looks good and what doesn't look good in practice.
102:17Uh they don't need to write code.
102:18They just need to describe in natural language what good looks like to them.
102:23And you can boil that down into your Eval rubric.
102:26Uh, and so far, uh, I've shown you how to do things both offline and online.
102:36Uh, we've been doing everything in a notebook.
102:39Uh, but you're not really done when your dev experiment tra pass passes.
102:43Uh, you're done when your agent uh, keeps performing in production on traffic that you've never seen.
102:48Um and it's worth spending a couple of minutes talking about that even though uh we're going to stick in the notebook uh for this workbook for this work session.
102:58Um the first thing is online eval uh like I said uh and like I showed you earlier online evals are exactly what they sound like.
103:05They take the evaluators that you wrote today uh the ticker check the faithfulness the actionability uh and run them automatically on incoming production traces.
103:14The same a val running on new data.
103:17Uh I showed you how to run this on a trace.
103:19AX lets you do this at three scopes.
103:22You can run your evals at the span level.
103:24So you can say this particular LLM call, this particular tool call, run a trace on it every single time.
103:30Sorry, run a eval on it every single time.
103:32Or you can run it at the trace level where it has the full tree to look at uh and it can evaluate everything from start to finish.
103:39Um you pick the scope that matches the question that you're asking.
103:44Um, and you don't need to run evals on every single prediction trace I uh production trace rather.
103:50Um, LLM judges cost money and at production uh scale those costs can add up.
103:55So you sample 10% of your traffic or 1% of your traffic.
103:58Uh, and that gives you a directional signal without you having to run an expensive LLM as a judge on every single iteration, every single uh, instance of your application.
104:08Um if you don't want to write your evals by hand uh we have good news for you which is that uh we have built an agent into AX itself.
104:19It is called Alex.
104:20Um, you can describe your eval in plain English or you can get your non-technical uh colleagues to describe a pro describe the definition of good in plain English and Alex will actually manipulate the UI for you
104:34uh create an eval write the rubric set all the scores correctly get all the configuration correct and then just tell you that it's done.
104:42Uh this is especially useful for non-engineers.
104:45Um and the full loop in production looks like this.
104:50So you go from application uh which produces traces to online evals which grade those traces and produce labels and explanations.
104:57The eval labels feed into monitors which we didn't cover today.
105:01Um the monitors can alert your team when something goes wrong.
105:05Um the team can investigate, find the failing traces, save them as a regression data set, um improve the agent, run an experiment to verify and ship the fix.
105:14And then the loop can run again.
105:16Um, as new production traffic accumulates, you will accumulate new failure modes, you will accumulate new edge cases.
105:24Uh, you're always going to be uh iterating on your evals and iterating on your production monitoring.
105:32But it is a payoff that compounds.
105:34Every failure that your online evals catch becomes a new test case.
105:38You save it to a data set and now it's a regression test for your next round of changes.
105:42Over time, this creates a data set that is unique to your application.
105:46Uh so your specific failure modes, your specific users, your specific quality bar, nobody else has that data.
105:53Uh and that is a competitive advantage that grows every single time your agent runs.
105:59There's one last pattern that I want to leave you with uh which is the most powerful one I know.
106:03We did the simplest version today.
106:05We fed the judges explanations back to Claude and had it rewrite two prompts.
106:09This works beautifully for two prompts.
106:11Uh but a real system isn't two prompts.
106:13prompts scattered across many files, retrieval settings, tool definitions, all tangled together.
106:18Rewriting that isn't a single API call.
106:20It's a job for a coding agent uh that can see your entire repository.
106:26Um so this is the pattern.
106:28You want to export your failing traces from ax along with their explanations uh and hand the whole batch to cloud code or cursor or another coding agent as context.
106:37Uh this is where the Arise skills plugin that I mentioned at the beginning uh works really well because you don't have because you don't have to know how to call our API.
106:47You don't have to know how to do any of that stuff.
106:49You give it the skills and then tell cloud code, hey, pull down some traces, see what's wrong, and make make my application better and it works.
106:57Uh there should be two guardrails on that approach first.
107:02However, first you should feed it your requirements too and not just the failure explanations.
107:06So the goal isn't make the eval pass because otherwise it will cheat.
107:10It will include all of your test data into its eval uh and automatically pass all of your evals.
107:16So you have to make uh um your you have to make sure that your requirements to your cloud code are clear that you don't want it just to pass the evals.
107:25You want it to do this specific thing which then makes the evals pass.
107:29Um and second, you should tell it to find themes and not chase individual failures.
107:34It's very tempting for claude code to look at one particular failed trace and write a whole bunch of code to fix that particular failed trace when there are actually 10 failing traces that are all the same co cause that it should be focused on instead.
107:46Uh I'm not going to do all of that time all of that because I only have 12 minutes left.
107:50Uh but um once you have eval this is one of the highest leverage things that you can do with them.
108:01So let's step back and look at the whole shape of it.
108:03Uh production produces traces.
108:05Online eval them and produce explanations.
108:08A coding agent reads the explanations and proposes fixes.
108:11And experiments verify that the fixes work and then you ship.
108:14The production traffic that comes back gets graded by the same online evals that started the loop.
108:20Uh and this is the software development life cycle closing in on itself.
108:25uh your AI software is helping you improve your AI software and AX is the substrate uh that ties it all together.
108:31Your traces, the evals, the explanations, the data sets, the experiments, the online evals and the monitors.
108:39So just to recap, we instrumented an agent with two lines of code.
108:43We traced it and we read our data.
108:45We wrote a code eval and built-in LLM evals.
108:48We wrote a custom rubric from scratch.
108:49We validated our judge against human labels.
108:52Uh we saved failures at the data set.
108:54We had Claude improve the prompts.
108:56We ran an experiment to prove that it worked.
108:59Uh and we saw how AX takes all of that into production uh with online evals.
109:05Uh this is the full loop and now you know all of it.
109:09However, you don't have to do all of it at once.
109:11This has been two hours of extremely dense information and I'm aware of that.
109:16You can start very very small.
109:18Uh you can start by reading your traces.
109:20Just add those two lines of code and start looking at your traces and that is already that is immediately going to give you information about what your agent is up to that you didn't know.
109:29Uh 15 minutes of reading real outputs will teach you more about your application than an hour of building dashboards.
109:34Then write one code eval.
109:36Code evals are easier than LME evals and they're faster and they're cheaper.
109:40Uh and make it for the thing that matters most.
109:42So correctness, faithfulness, whatever fits your use case.
109:45Run them, look at the results, see what patterns emerge and build from there.
109:51Evals are infrastructure.
109:52They are not an afterthought.
109:53Some teams create evals at the very start of development.
109:56Add others add them later once they are at scale.
109:59Uh but either way uh the important thing is that you treat evals as a core part of your system.
110:04They should be as routine as unit tests and the value compounds but only if you keep investing.
110:10Each time a regression shows up before it show reaches your users instead of after you'll understand why all of this work is worth doing.
110:18Don't hope for great.
110:20You can get to great systematically by specifying it, measuring it, and improving towards it.
110:27So, now is the time to try this for real on an app of your own.
110:30Uh, you've already got an AX account, I hope, by the time the Wi-Fi kicked in.
110:35Uh, the docs at arise.com uh have companion notebooks with runnable code for everything that we covered today.
110:43Uh, and if you want your coding agent to do the heavy lifting, you can install the Arise plugins, uh, the Arise skills plugin with one npx command.
110:52Uh, and that is in that link is in today's notebook.
110:55Uh, if you've made it all the way to the end of this, you can also get a free year of Arise Pro uh, using that code that's at the on the screen right now.
111:04And that is it.
111:05Thank you all for your time and attention.