Your Old Prompts Are Making Claude Worse - Lance Martin, Anthropic

AI Engineer · 64 min · 861 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12We can kick it off.
00:13Thank you all for coming.
00:14I'm Lance.
00:15I'm member of technical staff at Anthropic.
00:18And I actually want to share some very simple things with you today.
00:22A few simple ways to get the most out of frontier models.
00:26So we typically think of cost and quality as a trade-off.
00:31So in order to get, you know, better performance, you have to spend more money.
00:35And in order to kind of reduce cost, you have to scratch sacrifice performance.
00:40And so you can kind of think about it on any given evaluation, you have models that kind of create this Pareto frontier.
00:46And you can imagine this case, Fable is kind of up on the far right of the curve, and then you have Opus, you have Sonnet, then you have Haiku.
00:54Now what I want to share is actually that many times you're not actually on this frontier.
01:01Your application is actually sitting inside this frontier that you could be on.
01:05And it's because of minor misconfigurations.
01:08Which I'm going to show you today how to fix.
01:11[snorts]
01:11And we've seen this repeatedly.
01:15Improve performance and reduce cost for many customers using some view very simple things.
01:22So first, I'm going to cover five themes.
01:24And then we can make it interactive.
01:26The slides won't take an hour.
01:27Not even close.
01:28So I question answering is actually much more fun.
01:32First, audit your prompts.
01:34This is a kind of important point that prompts accumulate debt just like code.
01:41So if you recall several generations ago, we used to use things like think step-by-step.
01:46We used to use verification rituals, double-check your work.
01:49We used to use thought from this boosters, be maximally thorough.
01:53Sometimes we would use kind of scratch pads, tell the agent how to think, give it places to think.
02:00You don't need that anymore.
02:02Frontier models are very good at at thinking, choosing when to think.
02:07They don't need to be told what to do and like kind of how to verify their work.
02:11They naturally verify their work.
02:14Um they don't need told to be thoughtful.
02:15They're already very thoughtful.
02:17Um and worse, when you do these things, it actually increases cost and can reduce performance in some ways I'll talk about briefly.
02:28Another interesting one with earlier models, if your prompt had contradictory rules in them, earlier models that weren't as good at instruction following would kind of ignore one of the two rules.
02:37So it kind of still worked.
02:39But with frontier models, they'll often get confused by contradictory rules.
02:44So I'm going to give you one very simple thing to do right now if you have any code that you want to check.
02:50In Claude code, you can run this command checkup prompt audit.
02:54You can also run Claude API prompt audit.
02:56These are two different skills in Claude code.
02:58They link to the same code.
03:00It's just checkup is typically how people do checkups on their Claude code configuration itself.
03:06So your skills, your prompts within Claude code itself, the Claude API skill is a separate skill bundled in Claude code that's meant for a kind of code that uses our APIs.
03:16But both of these commands use the same principle and they kind of link down to the same underlying skill itself.
03:21It's just two different ways to call it.
03:23[snorts]
03:24These skills will audit your code, anything in the working directory of Claude code.
03:29So that could be skills, that could be tools, that could be prompts for these anti-patterns and will remove them for you.
03:35This has become kind of very popular.
03:37Many people we all Many people use it within Anthropic.
03:39It's kind of become somewhat popular, you know, on Twitter as well.
03:42Um it's very simple and often it can improve cost and performance.
03:49So here's an example uh where it This is an evaluation that I built.
03:54It's a synthetic evaluation of customer support tickets.
03:57There's like 50 tickets.
03:58And what I do is I start with Opus 48.
04:03And I basically simulate a migration to Opus 55, okay?
04:09And in this process, I inject each anti-pattern individually and measure the result.
04:15That's this middle point, okay?
04:18So, what happens is it's taking the average of all the anti-patterns across the benchmark, basically.
04:23The key point is this.
04:25When I do the migration to Opus 55 from 48, initially with these kind of anti-patterns injected, we do see some cost savings and a little bit performance savings.
04:36You can see it goes to the left and it goes up a bit.
04:40If I run prompt audit on top of that, I get a further bump both in performance and cost reduction for a bunch of different reasons.
04:48Because this command is going through and it's removing verification rituals, boosters that are actually causing Opus 55 to expend excessive tokens.
04:58So, this little simple command often will basically reduce token use, saving cost, and it can improve performance in cases where things like contradictory prompts were confusing frontier models.
05:11So, this is a very simple command that I recommend.
05:13Um and we've used it very extensively within Anthropic and also many customers have enjoyed this quite a lot.
05:18It's very simple to run.
05:19You run it kind of in any code using the API or even on your cloud code configuration to make sure your prompts are kind of well-tuned for frontier models.
05:28That's point one.
05:30Now, point two is make sure you're using the prompt cache.
05:34This is probably pretty familiar to many of users, but I do just want to make sure it's extremely clear.
05:39So, agents, for example, operate across many turns.
05:43Every turn adds just a little bit.
05:45For example, one tool call.
05:47Much of kind of context is repeated across turns.
05:51You don't want to pay for it twice.
05:52That's why you can use the prompt cache.
05:54So, prompt cache is built at like a tenth the the rate of base tokens.
05:58So, it's much cheaper.
05:59Varies a bit per model.
06:00I think 10 a a tenth is I think the current pricing, but you can check the pricing page.
06:05The point is it's much cheaper.
06:07It's also much faster.
06:09So, it's very important in multi-turn applications to be making heavy use of the prompt cache.
06:14Now, the problem is prompt cache are a strict prefix match.
06:20So, any even a byte change in your prefix will invalidate the cache.
06:25So, you have to be careful in building agents or multi-turn applications that you don't break the prompt cache.
06:29That's the key point.
06:32And Claude console, if you're using the API, has nice diagnostics for you.
06:36So, if you're building applications with the Claude API, you can see for your application your prompt cache hit rate, which is very important to track.
06:43For example, with Claude code, we track the prompt cache hit rate very very religiously.
06:48It's very important for the economics of the application, obviously.
06:51And it's also in your cases, if you're building apps with the Claude API, it's very very important to make sure you're using making heavy use of the prompt cache.
07:00Now, in general, how to organize your prompts.
07:03You know, you can think about basically, um system instructions, tools, things that are globally cached across all of your sessions.
07:10You can think about Claude MDM memory.
07:11That's kind of cached per project.
07:14Session state is kind of per session, and then messages kind of incremented each turn.
07:17So, that's kind of how you want to structure your prompts.
07:20And the key point is this, run this in Claude code.
07:24So, if you have a project using the API, you want to make check and make sure that you're using you're making proper use of the prompt cache.
07:29You can run this.
07:30It'll run diagnostics to make sure that you're using the prompt cache effectively.
07:34It'll find opportunities to reorganize your prompts to make sure that they're organized such that you get a high prompt cache hit rate.
07:40That's really the key point here.
07:42[sighs]
07:43And this is kind of a fun exercise.
07:45So, myself and some folks on our Claude AI team ran um ran this command on a bunch of internal benchmarks that had kind of default harnesses.
07:53We found opportunities to reduce costs by up to 72% with no meaningful change in performance, just by running this command, because some of the harnesses on these Evals were not using the prompt cache properly.
08:05Really simple command.
08:06You can run it in your API code and it'll just look for opportunities to make sure you're saving as much cost as you should be, i.e. by using the prompt cache.
08:13It'll also check for other things.
08:14It'll look for um I'll I'll talk about it a little bit later, but it'll also check your effort levels, it'll check your prompt.
08:19So, it it'll do a number of other things, but in particular it'll look at your prompt cache to make sure you have a high prompt cache hit rate.
08:26Again, this has, you know, optimized prompt cache has no change in performance.
08:31It's like effectively just free cost savings.
08:34So, it's something to be very serious about when you're building applications with the API in particular.
08:38Um it's very something to very to very seriously make sure you're utilizing effectively.
08:45So, I want to get into to some other fundamentals.
08:49Um we talked about prompt caching, we talked about making sure your prompts are kind of well that are that are well-tuned.
08:55You can audit them using prompt cache.
08:58Now, let's talk about a few other things that are very important for working with frontier models.
09:02So, Claude, frontier Claude in particular, is very good at self-correction.
09:09So, if you give Claude a well-structured evaluation, it can basically just hill climb on that in that eval and I'll talk about that in a minute.
09:16But important in that is good eval design.
09:19And you have to give Claude an evaluation that is climbable.
09:23And so, actually we spend a lot of time thinking about eval design and making sure Claude is set up for success when building an eval and setting up hill climbing.
09:33So, some general features of good eval design.
09:36This is This is basics, this is nothing [clears throat] related to, you know, Anthropic, this is just kind of fundamentals, but it's good to revisit briefly.
09:43What you should see generally is that these are a number of different curves looking at basically tokens per attempt.
09:49This is kind of like thinking on the X and then score on the Y. Frontier model at the top, mid-size model on in the middle curve, small model at the bottom.
09:58So, think about that as like Opus, Sonnet, Haiku.
10:01You should generally see on your evaluation as you step up model and performance goes up.
10:06That's point one.
10:07As you increase thinking, performance should go up.
10:10That's point two.
10:11There should be headroom to improve at the frontier.
10:14So, that's point three.
10:15So, basically shouldn't be saturated.
10:18And there should be low run-run variance.
10:21Very simple points, uh but actually very important to get right.
10:24And actually even a lot of very popular evaluations that we see have many tasks that are ambiguous, effectively limiting that headroom at the frontier.
10:34And this is a problem I see all the time.
10:35You build an evaluation, it turns out you have a bunch of ambiguous cases that can't be solved anyway.
10:39So, it looks like, you know, oh, I've 20% more that I can get on this, but no model can actually achieve it.
10:45So, you could spend many tokens [clears throat] hill climbing on an evaluation that's effectively saturated.
10:49It's very important to get this right.
10:52[sighs]
10:53Try this command.
10:54Uh Claude API build eval.
10:57This is actually a very nice command because it's not opinionated at all about where your data lives.
11:03It doesn't care.
11:04You can point this at whatever tracing solution you have.
11:07It's just a skill in Claude code.
11:10It's more employing the principles I talked about to test and sanity check the evaluation for you.
11:16So, you can kind of give this skill, hey, here's a problem I want to solve, here's what I'm interested in, here is the raw data, for example, here's the traces that I want to work with, and this command can help you build an eval set.
11:26It'll walk you through a process and step you through building it.
11:30It'll it'll help you you'll show you the data, it'll show you the evaluation, it'll sanity it'll sanity check the grader the criteria with you.
11:37So, it'll kind of walk you through the process of building eval based upon some of our, you know, some of our best practice and we have a nice blog post on this as well that
11:44that I'll share.
11:46Um and it pairs very well with hill climbing.
11:50So, once you've built a good evaluation that has headroom at the frontier, low variance, scales as expected with thinking and model, you can then climb on it.
11:58Frontier Claude models are very, very effective at hill climbing.
12:01So, autonomously looking at an evaluation and autonomously kind of improving on it without you in the loop.
12:08So, you build a good eval, Claude can hill climb on it.
12:13The problem is there's many ways that this can kind of go wrong.
12:17And in particular, you can end up overfitting on the evaluation itself.
12:22So like some classic examples would be um like the task in your benchmark have particular needs that are bespoke or specific to those tasks.
12:32Like maybe one task needs OCR, so it happens your harness to solve the eval adds like an OCR tool.
12:39But in production, you don't see that problems.
12:40So you end up overfitting on the eval distribution, it doesn't apply to production.
12:44This is very common.
12:46Um yeah, like distinctive phrasing specific to your eval examples.
12:52For example, another classic is like it just kind of it like the prompt when you're basically hill climbing overfits the specifics of your eval distribution, it doesn't apply to to your to your test distribution.
13:05So overfitting is the major challenge concern when you're hill climbing.
13:08Imagine you spent a lot of tokens hill climbing an evaluation.
13:11It looks really good after hill climbing.
13:13You run it in production and performance is not better at all.
13:16That's frustrating because you spent a lot of money.
13:18You hill climbed and like, oh, it's not applicable to production because often you'll you've just kind of overfit to the eval distribution.
13:27So try this command.
13:30In Claude code is Claude API hill climb.
13:31So if you build an eval, it'll hill climb for you.
13:34And it tries to combat this with a few tricks.
13:37Now it's of course not perfect.
13:39You have to audit this carefully and and and indeed be in the loop and and make sure you really understand what's happening.
13:44But it can help you.
13:46And in particular, what it does is a process like this.
13:51It will look at your evaluation and it will split it into test and train sets.
13:56This is kind of very classic um kind of classic principles from statistics.
14:00Create a test and train split.
14:02What happens is when hill climbing, this is just on Claude code by the way.
14:05So Claude code basically has this skill.
14:08You run it on an eval.
14:10It will only ever look at the train split and reflect on those failures when it's making decisions about what to change to improve on the eval.
14:18So, what's nice about that is at every round, it never sees a test split.
14:24And it will basically check performance per round against test and train.
14:28So, what happens is a little it's a little early signal for overfitting.
14:32If you see train improving but test flat, it's a signal of "Ooh, you might be overfitting."
14:38So, basically it applies this check and a number of other more subtle things check to make sure while hill climbing's happening, you aren't just overfitting to your eval distribution, you're actually
14:48hill climbing in a way that will apply to production.
14:51Assuming you built your eval distribution well, and that's always something that we can address in that build eval phase.
14:58Now, I'll share some concrete examples that I've used here.
15:01So, this is actually the Claude API skill itself.
15:04So, if I back up a little bit, this is a skill in Claude code.
15:10And it is a skill that gives Claude full awareness of our APIs.
15:15Uh so, basically our SDKs, APIs.
15:18And it has to be grounded in our documentation.
15:22So, I built an eval set based upon our documentation.
15:26And I tested the skill.
15:28The skill initially was not very good at the eval questions in my set, 66%.
15:32So, I set up a hill climbing loop with giving Claude code access to the skill text and the documentation itself, and let it basically just iteratively look at the failures in the eval set, and try to figure out why
15:45the skill was misrepresenting what's in the documentation.
15:47And there's all sorts of little tricks, and I could, you know, talk about that more in more detail.
15:51But the point is, Claude can autonomously find, in this particular case, the right skill text to improve significantly on the evaluation.
16:00And I talk about this in the blog link below.
16:03Um but again, this is a good example of a kind of a well-scoped evaluation, a simple service I'm modifying a skill, and an autonomous hill climbing loop that resulted in a much better performance just when I set the system up this way with a carefully scoped eval,
16:16a surface that's easy to modify, claw can just rip through it and like significantly improve performance.
16:22This works well for prompts, it works well for skills.
16:25Simple text-based surfaces are very very good for this kind of hill climbing process.
16:30Now, this is another kind of interesting trick that I've seen a lot of customers like.
16:36You can also apply hill climbing to cost reduction.
16:39So, what's kind of nice about this is I've added guidance to this hill climb command.
16:44If you tell it to reduce cost, what it'll do is it will basically try to find a configuration that's that is at is at least at parity in performance and for sure reduces cost.
16:55And it'll search things like effort, model, and your prompts.
16:58Here's a simple example on that same um customer support benchmark.
17:02I start with Opus Opus 48 high with some baseline prompt.
17:06And I basically tell the hill climber reduce cost.
17:09So, what it does, it basically will search through model effort combinations and basically pick those that continuously reduces cost over the rounds without degrading performance.
17:21In this case, it moves to Opus 5 low.
17:23We see a big bump in performance and minor reduction in cost.
17:27The hill climber then autonomously tries flying at 5 low.
17:30We see again degrades performance a bit, but big cost reduction.
17:35And then it it then decide tune the prompt and we kind of risk rescue the performance.
17:39So, we basically got a 90% reduction in cost with notable improvement in performance autonomously using the hill climber.
17:46This is another very nice way if you have an evaluation, you can use the hill climber to basically say, "Okay, keep performance fixed but reduce my cost materially by tuning things like effort,
17:56things like model selection, and my prompts."
17:59Another very nice use of the hill climber.
18:00And actually, what's nice about this is you can do this if you have a saturated eval.
18:05So, you can basically say, "Okay, say my eval saturated, fine.
18:08It doesn't matter.
18:09I'm not I don't care about performance.
18:10I just want to reduce cost."
18:12So, then you can just tell the hill climber, "Okay, keep performance fixed and just try to burn down cost."
18:16And it can basically search for configurations and often it'll find very interesting things.
18:21Like in this particular case, there's generally an intuition that moving to higher models at lower effort is often cheaper and more performant than lower models at high effort.
18:30It's like giving someone who's smarter can like try less to actually get a better objective or or or to kind of solve the problem more effectively.
18:38Um and so it ends up actually using fewer tokens and as a result ends up being cheaper.
18:43That's kind of a broad intuition here that we've seen quite a bit.
18:46And that's baked into this hill climber skill.
18:49So, um another nice trick.
18:53Now, I'm going to open up the questions pretty soon.
18:55I'll just kind of close with some other comments.
18:57And this is a bit of a nod to the magic agent stuff you heard this morning.
19:01I just want to re-emphasize this a little bit.
19:02I do think um as part of unhobbling the frontier, it's very important to design harnesses that can take most advantage of the latest Claude models.
19:12And in particular, it's interesting if you look back the last few years, time is on the X, and this is task duration for human on the Y. You've seen these curves.
19:23They're just meter charts.
19:25It's interesting to think back, you know, Claude Opus 3 was capable of only like 4 minutes of autonomous work.
19:32And in that kind of regime, there were very limited products you could build.
19:36It's really auto completion or chat.
19:38Now, we saw over the course of 2025, models become more capable.
19:41They can do like hours of work.
19:42Like maybe an hour to 2 hours of human equivalent work.
19:45Claude code obviously kind of um gained significant over the past over 2025.
19:51And we're kind of entering a new regime now where these um task durations for models are are exceeding like a day to 2 days.
19:59I think this is kind of outside the scope of what meter can even measure now.
20:03But in this regime where you have models that can do the equivalent of like, say, days of work, what harnesses do you need?
20:10Like the considerations change quite a bit because you want models that can just do things autonomously over very, very long periods of time.
20:16Now, how do you make sure they're secure and reliable?
20:18The considerations change a lot.
20:21And I hear you heard about that morning, but this is actually the motivation behind one of our newer APIs called Cloud Manage Agents.
20:28Um and the principle here, I wrote this engine blog back in April on it, is decoupling the brain from the hands.
20:35And the idea is pretty simple here.
20:37You have an agent harness, which is effectively the brain.
20:39You have hands, which are sandboxes.
20:40That's where the agent harness writes code and performs actions.
20:43And you want those to be separate things.
20:46In particular, you want that you want it to be the case that if the harness dies or the sand box dies, the system can recover.
20:53Imagine you have an agent that you spin up overnight to just run and the sandbox dies midway through the night.
20:58It's very frustrating.
20:59You lose state, you lose your session log, and everything.
21:02So when we design this, we designed this for a world where Cloud would be doing work for very, very long durations.
21:07And so it's recoverable.
21:08So that means the session log is an independent object separate from the harness, separate from the sandbox, that is append-only, that is recoverable.
21:16So if the harness dies, the sandbox dies, you can always recover from the session log.
21:21Sandboxes are also recoverable.
21:22So if a sandbox dies, you can boot up a new one.
21:25If the harness dies, you can boot up a new one through this orchestrator.
21:28Another point here is safety.
21:30So one of the important things that Manage Agents does is it actually never gives Cloud direct access to secrets.
21:36Secrets are held in vaults.
21:38Um and that's actually stored in that kind of tools box you see at the top.
21:42So vaults ensure that Cloud can reference API keys or secrets, but it can't actually see them.
21:46Because obviously, if you have an agent that's working for, say, days on end, it's a little bit spooky to give it direct access to very kind of sensitive secrets.
21:54And so we ensure the security is is kind of very is is a very important consideration when we build Manage Agents.
22:02So reliability and security become much more important as agents can operate for hours, days, and maybe soon weeks.
22:09And this morning you probably heard a little more about this, but we have a nice uh recent blog that I put out with CJ that walks through uh an example
22:19of basically building an agent that can serve as like kind of a an an agent automation uh for regular context updates.
22:26It can reach into different sources that you specify, it can write updates to Slack.
22:30Very simple automation example, it's highly customizable, but it shows how all these different primitives come together.
22:35It brings together a scheduled deployment, all this all runs on Anthropic infrastructure, it secure- it securely stores secrets, it has memory, and um it's a very nice kind of end-end example.
22:49Now again, the final point is you can just run this in Cloud Code.
22:53So, run Cloud API magic sandboard and then give it the link to the blog, and it'll just boot up this tutorial for you, it'll spin up the example, and you can just play with it.
23:03Um so, the really the key point here is um there's a number of ways you can kind of harness and uh make sure that Frontier is effectively unhobbled.
23:13Uh it is very important to audit your prompts.
23:17It's very important to make sure that you're effectively using prompt cache, especially building agents for long duration, to make sure that cost is managed.
23:23It's important to think about harnesses that are designed for long duration in terms of reliability and security.
23:29And it's also important to think about evaluations and hill climbing.
23:32And when this all comes together, you can have agents that you can give a task, for example, an evaluation, and let it run for long period of time and hill climb on that evaluation, come back, for example, in the morning, and it's it's significantly improved.
23:43And that's really the way that I see many of my colleagues working now.
23:46People are spending much more time on eval design, and basically just specifying tasks that Claude can effectively hill climb on, and doing this in a way such that agent is highly secure when doing this,
23:58and is highly reliable for long duration jobs.
24:02And let's see where we are on time.
24:03With that, I think I'll just take it to questions, because it's more fun when it's interactive anyway.
24:08You can follow me on all the the socials here.
24:11And uh thank you for your time and attention.
24:16[applause]
24:25Sure, start there.
24:27[laughter]
24:27This is a very good question.
24:29The question was how automatable is eval design?
24:32So actually I was engaging a bit with with Hamel on Twitter.
24:35Extremely thoughtful with respect to evals.
24:38And some feedback he had on our skill was that it should indeed walk the user through the data first, which I actually agree with.
24:44I'll make it some minor modifications to the skill to do that.
24:48So I think that uh eval design, I'll go kind of go back to it.
24:56Claude can assist you when building evals significantly.
25:00So for example, in the example he gave when his walk-through, um he was looking at like a it was like a customer support app or something.
25:08And the first question was like, okay, like it's a bunch of traces.
25:13What are the failure cases here?
25:15What should I even like promote to an eval set?
25:17So Claude's really good at kind of identifying, ooh, okay, here's case where there might be errors.
25:21It's effectively like acting as like an online evaluator, but looking at your offline logs, scanning them saying, okay, here's some problematic examples.
25:29And Claude was very good at that in his example.
25:32Then it is actually important and quite useful if Claude then shows those raw like the raw data to you in a viewer.
25:40So I'm going to add that to the skill.
25:42I have a PR up and I'll add it.
25:44Currently, Claude goes directly to analyzing the data and like building the eval and showing it to you.
25:49I do think it's important and helpful for there to be an intermediate stage where you look at the data first.
25:54But I think having data annotated by Claude where Claude's giving you some ideas about what to look at is very important.
26:00So kind of like Claude assisted evaluation and scanning of the data is like I think like I I think a nice marriage.
26:06Um and then once that's done, you can kind of promote like kind of examples to your eval set.
26:10So I think there is still quite a role um for the human in the loop there, but Claude can assist quite a bit.
26:17Hill climb with human feedback.
26:19Um so hill climbing only requires that you have an eval and you have some signal that Claude can use to judge good versus bad.
26:29So if you've taken human feedback and you've rolled that into an eval signal, for example, then it should be perfectly feasible to hill climb on an evaluation structure that way.
26:37Yeah, this is a good one.
26:39So, for hill climbing, what's like the best surface to start with?
26:42So, I kind of have that here as like the thing we edit.
26:44Yeah, so so I've actually played with this quite a bit.
26:48Text is nice, in particular if things like prompts and skills are very nice because it's a very it is a very clear place that we know sign that we know significantly alters and affects model behavior.
27:01So, it's like a very constrained surface to modify, and we know it's highly coupled to the output, typically in terms of like what you're evaluating.
27:08Uh like here's a good example.
27:10Um like when I'm doing skill hill climbing, I'm testing something that is directly reading from the skill, so I know there's clear coupling.
27:16Now, I'm going to contrast that to I've also done hill climbing with agent harnesses.
27:21But, what happens there is it can be a bit more hard to interpret.
27:23So, you're modifying the harness code, and so the the the the changes per round are much larger diffs.
27:30And um there's just more surface area for confusion.
27:33You can be making changes that are actually decoupled from the output signal you're measuring.
27:38So, it's certainly possible, and indeed people have done quite a bit of work hill climbing on agent harnesses.
27:41I just think it's a little bit trickier, and text-based services are often a good starting point, prompts, skills, um because they're so clearly coupled to model behavior, and they're easy to modify.
27:53The catch actually is that is that things like prompts are very easy to overfit.
27:57I've seen that in in many of my uh hill climbing efforts as well.
28:00So, like um I don't know if I have a good diagram, but um you know, you have some eval distribution you're hill climbing on, even with these regularizations in place, like you you the eval test train split and so forth.
28:12Um prompts are very easy to like wiggle Cloud's very good at wiggling the prompt into a shape that does well in your eval that isn't really representative of what's in production.
28:22So, you have to be very careful with prompts, but they are very easy to to modify.
28:25So, it's a good starting point, you do need to be careful with overfitting.
28:29Uh let's see.
28:31Turning a prompt into a skill.
28:32So, they're kind of compliment like um maybe do you mean if you if you're like going through some standard operating procedure, when to like promote that to a skill?
28:41Okay, yeah.
28:42Um that's actually kind of interesting.
28:45I think often I and Claude is quite good at judging that.
28:48So, basically you can basically go through a workflow and then you can ask Claude, okay, turn that into a skill.
28:53I do that all the time.
28:54So, Claude is pretty good at like catching, okay, you're doing some repetitive thing, make it a skill.
28:58Uh I actually think you should just have Claude do that.
29:01Um just tell it, turn this into a skill, and it'll do it.
29:04So, usually in the in the case of evals, the setup will be I'll have an existing skill that I've kind of defined and maybe it has some boilerplate starting text and I'll say, okay, I want you to make this skill improved.
29:18I want to improve this skill text to basically perform better on this eval.
29:23So, that's usually the way I set it up.
29:24Like I have a skill boilerplate as a starting point.
29:28I know I want this thing to be a skill.
29:30Like in this in this case, right?
29:31Like I had this skill for the Claude API.
29:34So, and it was basically you can imagine it's basically distillation of our docs.
29:37It's like go to this page if you want to learn about this, go to this page you want to learn about this.
29:42Um it had a bunch of little gotchas and issues, and so my eval performance was pretty low to start.
29:47But I had a starting point, and that's what Claude was able to then modify.
29:51So, I think usually with skills you're kind of starting with some baseline.
29:56And it could be like a very crude default, but then give Claude something to like start from.
30:02That's usually the way I would do it.
30:03Right.
30:04So, verification in um domains that that have kind of open-ended criteria.
30:11So, like yeah, I think in general heuristic evaluators are preferable, of course.
30:15So, if you can have define like a a a deterministic check for good or bad, that's obviously preferable.
30:22So, in the maze that doesn't work.
30:23So, what I found is um well-structured rubrics that produce similar uh grades.
30:33So, okay, a few different things.
30:36Build a rubric that you as a human can reason about first.
30:41So, it's like let's say we're grading a piece of like text.
30:45Is this good or bad?
30:47Build a rubric that like I could look at as a human to like look at some text and say, "Okay, I can grade that."
30:52That's point one.
30:53Point two, don't use arbitrary scales.
30:56I see this all the time like grade it from zero to five.
31:00It's very ambiguous.
31:01Make it crisp and simple like ideally a binary classification good versus bad.
31:06You if you want to evaluate multiple things, you can set up like multiple binary classifiers rather than the worst thing is give Claude like from one to 10 rate this.
31:15That's extremely noisy.
31:17Another sanity check, test across different models.
31:19Have Sonnet, Opus, and Fable all grade using your rubric.
31:22If they all disagree, that's like a bad sign.
31:25Another test, run replicates.
31:28So, basically um like I show here how you want low variance, run replicates with your rubric and make sure it's tight.
31:35So, if you do all those things, you can definitely make it work pretty well um for yeah, like text-based answers that are squishy.
31:42Like there is no very definitive like way to heuristically check that.
31:47Uh but LLM as judge with a good well-structured rubric can be quite effective.
31:51Yeah, that's actually a good one.
31:53So, actually I I I I should have mentioned that.
31:55Uh pairwise comparisons can actually be quite good.
31:57It's much easier for humans even models to kind of compare two artifacts than it is to grade an artifact.
32:03Like you can typically answer, "Oh, I like this webpage better than this one."
32:06versus "I'm sure this webpage grade it between zero and five."
32:10It's much harder.
32:11So, pairwise comparisons are pretty effective and actually I think that is a good that is a good way uh if if your problem allows for that, that actually can be a effective way to deal
32:20with um more ambiguous gradings.
32:23Well, I think it goes a bit back to just looking at the data carefully.
32:27Kind of relate to the question asked over here.
32:30Um when I say the distribution, um I more mean ensuring that um you've sampled some set of cases for your eval set.
32:42And making sure those cases represent what you actually care about in production.
32:45That's really what I mean by distribution.
32:47And so, there are some interesting tricks here.
32:49Like one problem that happens is adversarial sampling, where you select like this is maybe a little bit confusing, but I'll try to explain it clearly.
32:58You select cases that are specific failures from a from a particular model class.
33:04So, models are jagged.
33:06And so, you just look at, okay, like Sonnet 5 can't do these examples.
33:09I promote them to my eval.
33:11Opus 5 may maybe it'll solve them all just because intelligence is jagged.
33:15And so, like a particular pocket of one model might just get solved by another.
33:20So, typically, a good recommendation is promote eval cases promote cases in eval set that are hard or interesting from first principles, from your own intuition, from looking at production data, not necessarily
33:33completely sampled from the failures of any given model.
33:35Cuz that can result in kind of this odd distribution that's shaped to the valleys of one particular like jagged model's intelligence that don't apply when you upgrade models.
33:44So, like can you use bucket-based evals or do you have to use binary classification?
33:47I'm being a little bit overly prescriptive.
33:50You obviously can, but just as a general rule, I see people going too far in that direction.
33:55So, there'll be some very arbitrary scale between 0 and 10 with unclear grading and rules that even a very small model can't resolve.
34:01So, you obviously, if you have a well thought through scale, you obviously can do that as long as it satisfies we talked about.
34:08It has like low variance, it's reproduced across models, a human grader can kind of distinguish between these two grades, that's totally fine.
34:15I'm just saying binary is like a shortcut of like if to simplify the problem, usually.
34:20And also for grading, you often want to use smaller, cheaper models.
34:22So, you really don't want Fable doing your grading and depending on your cost sensitivity.
34:27So, um, those are just the recommendations.
34:28But obviously, if it's well thought out, that's totally fine.
34:31You can think about using, um, yeah, like a multi uh, like a multi multi-category classification.
34:37Yeah, that's a good point.
34:38Like when when to build evals for what types of problems.
34:42Like, is it worth building it for my like personal like use cases or we only doing it for things that are kind of production grade?
34:49Um I think it depends on um I let's see.
34:56It probably depends a bit on the importance of the task.
34:58So, if it's even it's a personal task, you feel that it's quite important, then it would be perfectly reasonable to need eval for it.
35:03I think that's totally fine.
35:04I think people do that.
35:06Kind of varies by person.
35:07I personally haven't necessarily built many personal evals for like my personal automations, but um I think if it's a task that's highly important, it's in particular um I okay, I'll give a good example.
35:21Like if say people work in finance and like you've automated parts of your your personal workflow, but quality is extremely important.
35:28Like, this is a work output that's going to share externally or something, that would absolutely need evals.
35:32Even though it's something you've built personally.
35:34Um if it's my personal assistant, that's like, you know, it's like a a meeting it's like an app that I built to help me manage my calendar.
35:42Uh I may or may not build evals.
35:44It depends upon the context.
35:45But I do think obviously if it's something that's important for your work, it's like, you know, producing deliverables that are, you know, externally viewed or doing something very sensitive task, you obviously want evals.
35:54Even if it's a personal agent.
35:57I actually worked at LangChain before Anthropic, but I I I I can't advocate for any one or the other.
36:03Um Yeah, I I I plead the fifth on that.
36:06Um I mean, look, I've I've used many of them.
36:09Um I think actually if you're if you're working at relatively small scale, you obviously could roll your own.
36:14Like, you don't necessarily need a fancy observability system.
36:16In fact, we're going to be adding to the skill the ability to to um to to uh just like a simple data viewer or we're actually adding to the skill.
36:24So, Claude can kind of Claude's very good at kind of building you UIs and visualization surfaces.
36:30So, they can kind of roll If you have like a very bespoke problem you want to look at the data for, it can kind of roll you whatever app you want very easily.
36:37And and not very expensively, either.
36:39No, this is a key point.
36:40It's not guaranteed to converge at all.
36:42So, I've seen many cases in which the hill climber gets stuck.
36:45And actually, we built You can read the blog, but we built a fallback in the hill climber where if it gets stuck for multiple rounds, it'll actually go and basically try to cluster the the remaining failure cases, and it'll try to look for.
36:57So, for example, here's the problem.
36:59Oftentimes, you'll run the hill climber without having run like a build eval command or with without having built a very like carefully constructed eval set.
37:07So, the hill climber will go and look at your evals and try to find are these cases ambiguous like we talked about here.
37:14Maybe that last 20% are all faulty grader examples, ambiguous ambiguous instructions.
37:21This happens all the time.
37:22This is like very common.
37:23So, climber is just stuck not because Claude can't do better, it's because the eval is poorly structured.
37:28So, the hill climbing skill will try to find that for you.
37:31Uh so, we added that.
37:32In fact, when I did this hill climbing I talked about here, um this second stage actually went it kind of got stuck, and it went and basically clustered the eval cases and looked at, "Okay, why the failure cases?
37:43Why are they failing?
37:44What's going on?"
37:45It found a bunch of ambiguous cases.
37:47It rewrote them.
37:47And And anyway, so it'll actually do that for you, but that happens all the time.
37:51It's absolutely not guaranteed to converge.
37:54And usually it's because of failures in your eval or like kind of misconfigurations in eval.
37:58Uh so, that that's really interesting.
38:00I You have a lot of respect for the DSPy team and and Omar and and and and and um and colleagues.
38:05Uh this this actually just is a skill built like bundled with Claude code.
38:09The skill is also open source.
38:11So, you could consider taking that skill and and like vendoring it in any other surface.
38:15See if you want to could probably work with DSPy.
38:18It's just some very simple general instructions for setting up a hill climbing loop following basically these principles.
38:26Um so, um that [clears throat] it could be compatible with DSPy.
38:32I haven't used DSPy, so I don't know like the details and how it's structured, but um it's possible this skill guidance here could be compatible with it.
38:40If you want to like effectively implement this or run it in DSPy, the skill is open source.
38:46Uh you said choosing effort level.
38:49Yeah, this is a classic one.
38:50Effort's confusing, isn't it?
38:52Yeah.
38:53I'm also confused by effort.
38:55It's Yeah.
38:57It it is it's non-obvious.
38:59Um so um this is a so what's nice is if you have an eval uh the hill climber can absolutely try to configure effort for you.
39:13Now, I get it.
39:14It's like not that easy to have an eval.
39:16So, like it's not a it's not applicable to every problem, but if you happen to have an eval then this hill climber can search through your model effort configurations
39:22and it follows a very simple principle.
39:24The principle is just simply that typically the stronger model at lower effort is is like more effective uh or it sorry is more token efficient and therefore cheaper than like a weaker model at high effort.
39:36That's like the general principle.
39:37That actually turns out to be true a lot.
39:38So, it'll kind of search through the different models at low effort first and basically continue that search until you see a degradation performance below the baseline.
39:47So, for here I don't show it, but like when I tried Haiku for this was 4.5. Haiku we now have 5.5, which is quite a bit better.
39:54Uh 4.5 at low effort was a significant regression, so we admitted that that data point.
39:59But basically if you have an eval hill climbing is a very good way to try search for a good model effort combination.
40:05More general effort guidance if you don't have an eval is obviously tricky and subtle and like I think um people typically um I I will say the guidance I recommend generally on effort is try stronger model low effort
40:18in general.
40:19Opus 5.5 low effort.
40:22Try that.
40:23Basically.
40:24Uh better models low effort are often more token efficient and therefore cheaper than than weaker models at high effort.
40:29That's like the key point I'll I'll leave you with.
40:32So, not medium?
40:33Uh try low to start.
40:36Um you can always dial back up.
40:38But basically I like very smart model not trying too hard.
40:43That's a very that's often a sweet spot for for um kind of kind of cost performance.
40:50Obviously, you can dial up effort, but I think the the counter is people often start with something like max effort.
40:55This extremely token and like token hungry.
40:59So, it is actually wise often start like small smarter model low effort and kind of kind of go from there if you need it.
41:07Yeah, actually, we talked about this in the blog here.
41:12I was open that up.
41:14I don't have internet.
41:15That's fine.
41:16I'll just tell you then.
41:18Yeah, so it's basically telling Claude how hard to think.
41:21So, low effort saying like don't think as hard as you scale it up it's saying think harder.
41:27So, and that's kind of why more fundamentally intelligent model at low effort can actually be more token efficient and therefore cheaper than a cheaper model in terms of base tokens at high effort cuz that
41:40that for example, Sonic 5 at high effort is spending lots and lots of tokens thinking where smarter model low effort will spend fewer tokens thinking at a higher baseline of intelligence.
41:51That's kind of the intuition.
41:52Yeah, so the question is about setting budgets in hill climbing.
41:55So, basically when you when you run this skill and set this up, it'll ask you for a budget.
41:59So, um it will manage that for you.
42:03It'll track the budget you allocate for the hill climbing round.
42:06That can definitely happen though.
42:07And that's actually why this is a classic example.
42:10That's why designing a good evaluation is really important.
42:12If you have if you have a poorly specified evaluation, imagine for example, you have an eval where that final 20% is impossible.
42:19You burn a huge number of tokens trying to solve those final you know, 20% of problems that are effectively not solvable.
42:25Kind of an eval building, the question is about kind of human loop kind of human Claude collaboration.
42:32So, I think I think that um humans still have the role to kind of steer the goal.
42:41Like what are we actually trying to evaluate and why is it important?
42:44So, it's kind of taste is important.
42:45So, like what are you actually trying to see answer?
42:48For example, I'll give you I'll take an example from like Hamel's public kind of ant or a public dog fooding of our um build eval skill.
42:57So, it's like okay, it's a customer support app.
43:00So, let's say that's your application.
43:01What matters?
43:02What's important?
43:02What do you care about?
43:03Those are things that a human can guide Claude.
43:05Now, once those things are clearly specified, Claude um can look at like raw traces and determine effectively like okay, where is the app likely falling short of my goals?
43:18So, that's I think kind of nice collaboration you can have in place.
43:22And and and that's actually when like kind of looking at the data is very helpful.
43:25So, Claude can kind of help to steer you okay, I've looked at the data.
43:28Here's a bunch of cases that look problematic.
43:30And that's that's just like having an online evaluator.
43:32So, you like imagine you have a tracing system, you have an online evaluator.
43:36It has some like grading for like customer satisfaction.
43:39It's flagging like here's problematic traces for you.
43:42The same kind of thing.
43:43Like you're telling Claude, you're telling online evaluator, here's what I care about, here's what I want you check.
43:48It can look at the data, it can elevate a bunch of cases that look like these look problematic.
43:51Then given those cases, you can like review them, understand them, and then say okay, yes, build me an eval based upon these.
43:57That's typically a very nice marriage between like Claude and human when it comes to building evals.
44:03Yeah, question is like who's responsible for building evals?
44:05I mean, it depends on the problem, it depends upon I mean again throw back at everyone ever everyone is empowered to build evals.
44:10So, it's not really delineated per team.
44:13It's typically based on ownership like of the product and um I think in particular when you have skills like this, it make it easier to build evals to help climb, it really opens up the number of people that can
44:24feasibly kind of build and design evals, which is actually quite nice.
44:27So, we've seen like Anthropic many many many different types of roles build evaluations.
44:33Um because Claude's so involved and then has can help so much.
44:36Um much of the tedium of like um you know, building an eval runner and stuff, that's like Claude can handle all that.
44:42So, really it's about what we're talking about before, the human taste guiding the process.
44:47What do I care about?
44:47And that's very that many different types of roles can can kind of serve that guidance or provide that steering.
44:53Um let's see if I have a good one.
44:56Uh Right.
45:00Um Uh, off top of my head.
45:04Um I've seen Okay, I have a good one.
45:09I have seen um evals for generative UIs.
45:14Um, so that's kind of a good example.
45:18Uh, actually I think in that case it was a pairwise judge back to the point that you were making.
45:23Like so it's basically comparing like configuration A B, um what is better um between like two different generative UIs and like the greater would would do kind of pairwise comparison.
45:34I think then those results were rolled across many trials and scored with something like an Elo score.
45:39So that's an example of like generative UI with a pairwise evaluator I I've seen before.
45:45There's many other examples I can't think of at the top of my head.
45:48Um, but people use LM as judge all the time.
45:51Um, for like retrieval tasks or like writing tasks.
45:55Um Uh Look, categorizing traces, looking at traces and saying oh this is bad or good.
46:00So like it's extremely common.
46:03Yeah.
46:03So okay, that a lot of the motivation for the hill climbing work is kind of it is kind of flowing from auto research.
46:11Auto research is motivated heavily by the fact that models are getting very very good.
46:16They're very good at self-correction.
46:17If you give them a well-structured problem uh they can hill climb on.
46:21So I think Karpathy's case is a good example of I think I can't remember the precise problem.
46:26I think that general setup work with many different formulations.
46:28Like it could be um you know, like drive down loss or something.
46:32Um, so again there's like a very clear verifiable objective and clock can look at the objective and use that as a guide to hill climb.
46:39Um, I have Yeah, I didn't show it here but I have some other examples doing auto research um in this kind of hill climbing process and it would it would this kind of
46:49um This command in particular is actually designed more for eval-based hill climbing where you have like an eval set with a with a split.
46:56Auto research was looking more like you build an artifact and like grade it, like look at the loss or something, but the the setup could probably very be very easily kind of modified to work with auto research with the very same intuition.
47:08The point is that models are good at autonomous work over long horizons when given a clear um a clear objective or verifiable that that they can reward that they can kind of measure and use to inform what their next move is, they're very very good at autonomously solving problems.
47:24So, it's driven by the same intuition as auto research.
47:28hill climb and then extend into auto research.
47:31Uh the the methodology for hill climb would So, it's just a it's just a question of a problem formulation.
47:37This just to be clear.
47:38So, like the auto research from formulation was um was initially focused more on like on like designing, for example, an a like a network with, for example, um solving some problem where you have like a a verifiable objective like loss.
47:53That's really it.
47:54The the hill climbing process though was like very similar where like Claude runs some experiments.
47:58It measures loss.
47:58It looks at the results and then adapts, does something else.
48:01So, it's the same kind of hill climbing loop, but like in that particular case it's just applied to a slightly different domain.
48:07So, it's like you could probably take this skill and basically say, "Okay, set this up for if you had like like a the nano GPT repo, you could use this skill with it, I'm sure."
48:16It would Claude would figure out how to adapt things.
48:18Just If you're interested in that, just just like try it.
48:20But the the core process is the same.
48:22The core intuition is also the same that like models are very very effective at using um at self-correction and hill climbing.
48:30That's the key intuition.
48:32Yep.
48:33Yeah, I kind of try to do that in like this example where I'm looking at cost and um performance.
48:39So, you're like kind of jointly optimizing across those two.
48:42And yeah, you can just tell Claude the multi-objective.
48:46And usually it will just kind of measure multiple objectives and and take that into account.
48:51Um that's certainly feasible.
48:53And this is a good example of that.
48:55Have you checked like what's like the max you like
48:58Uh I mean, yeah, I haven't pushed that far.
49:01I think but I've typically looked at as cost and performance.
49:04And like speed's another dimension.
49:05So, you like cost performance speed.
49:07That feels like a pretty reasonable like trifecta of things that you would like kind of jointly measure.
49:11Um, yeah, I mean, I think that um, as you kind of scale up, it just hobbles the process of hill climbing a lot cuz there's lots of different objectives that Claus is trying to balance when making these changes.
49:21So, I think it'll just it'll make the problem harder, but like it is doable.
49:26Maybe you could think about kind of changing one thing at once.
49:29Doing doing them serially.
49:31Yeah, that's it that's a very good question.
49:33Uh, so the prompt on this stuff, which actually of all the things I talked about is the simplest thing to try and often is like extremely it's like do the simple thing first.
49:41It's very simple and often very effective.
49:43Uh, this general the general guidance in the skill applies to um, the the model delineation's a little bit on like it's not precise.
49:54These are well-known anti-patterns that definitively frontier models like the 55 series don't need anymore.
50:00How far back do you need to go until these patterns are still like appropriate?
50:04It's like not quite clear.
50:05I've tried to do some testing.
50:06It's maybe like Opus like 45 is the point at which these things start to like become less relevant.
50:13So, it's a little bit squishy.
50:14I don't know like precisely the model at which like you need to remove these, but my view is basically when you're doing a migration to the latest to like
50:22to like Opus 55, just do this.
50:24Like definitively like the latest frontier models don't need these patterns.
50:28Yeah, yeah, okay, good point.
50:30So, the skill does have some model specific caveats.
50:33So, the skill takes that into account a little bit.
50:34If you're for example going to like Haiku, I get it.
50:37Like maybe Haiku like um, isn't quite as autonomous.
50:40Maybe it does need slightly more boosters.
50:42I don't know if that's true.
50:43I'm just kind of yeah, kind of kind of suggesting something.
50:47Um, it it'll take that into account.
50:49So, it does have some model specific guidance built in.
50:52It's like smart to figure out like okay, here's the model you're working with and here's the anti-patterns that we know are are not needed anymore.
50:58Um, but it's it's a good call out.
51:00Yeah, uh, uh,
51:04Yeah, um it it is not a Yeah, I I should I should get back to to like to be really clear about this.
51:13Um it is not a hard cap.
51:16It is basically telling Claude emphasizing how much thinking to do to solve the problem.
51:22So, like for example, even if you're on very high effort, you give it a simple problem that won't just arbitrarily spend huge amounts of tokens verifying it.
51:29There's some Claude has some thoughtfulness about like what problems require, um you know, all this extra thinking.
51:36So, like um And and I don't know if I have any evals that show that.
51:41Uh yeah, maybe not.
51:42Um but it is suggested thinking.
51:47So, that's the kind of way I think it's basically telling Claude how hard to think about a problem.
51:50The problem comes up when you have like a harder problem, like a frontier coding problem.
51:54That's where at for example hot extra high effort or max effort, you can see huge differences in token usages from like medium or low without much difference in actual outcome because Claude will do more work to verify the result for example.
52:05So, I think out in the frontier regime of problems, that's where these effort differences can really result in like big big token spend differences.
52:13And that's where we need to be more careful.
52:15Yeah, I mean, this um this is So, if you kind of think about what's happening, um it's not dissimilar to to training.
52:25And basically you're giving a model some some reward or objective, and it it is indeed trying to make modifications to the system to improve the objective.
52:34Um so, yeah, philosophically that is similar.
52:37And in fact, even where we're used, you know, we employ some simple principles like test train splits that are often employed like at training time.
52:44But the nice thing is these are kind of run time modifications you can make.
52:47So, this is basically a way to employ some of those principles, but basically like once a model has been deployed and like you're modifying services like prompts or skills,
52:56um you can employ some of those same principles that we use in training, um but like in your particular domain.
53:02Uh that's kind of the the big idea.
53:04Yeah, so like is Claude aware this being evaled?
53:07Um, right, exactly.
53:09That I think that's something we've published on a fair amount.
53:12That is a concern um, in the that that is certainly a concern kind of in in training uh, in different stages of training, absolutely.
53:22What we're talking about here um, I I do think um, it could be somewhat of a concern.
53:32I think the bigger concern than what we're talking about here is typically more overfitting.
53:36So, Claude may know it's kind of being evaluated.
53:40Um, but it will find ways to hill climb the objective based upon what's been provided in kind of this eval distribution that has.
53:49The classic example being it'll find um, kind of prompt tricks and tuning that aren't applicable to production that indeed improve performance significantly on your eval.
54:00So, I've seen that happen that happens all the time.
54:02And it's um, it's not necessarily that Claude's like aware it's being evaled and saying like it's it's kind of some um, kind of unexpected behavior.
54:11It's more that like okay, you've given it a prompt and Claude can find ways to shape this prompt to like make number go up and it'll just do that.
54:18Um, that's why we use things like test train splits and try to regularize.
54:21So, I think overfitting is more of the practical concern that we're going to think about here.
54:25Um, but it is true that eval awareness is like a broader phenomenon that we we have discussed in in some in some of research blogs.
54:32Yeah, that's a good point.
54:33Um, kind of what's like the minimum kind of data set that's that's useful.
54:38What is like the split between test and train?
54:40Uh, I haven't looked at that in depth.
54:42Um, frankly, just ask Claude.
54:44Like um, when you set this up and you run this command, Claude has some heuristics to reason about the size of the eval set that's actually helpful given the signal or like what you're trying to measure.
54:54Claude can kind of reason about like how many examples you should have.
54:56And then there's some guidance in the skill for this.
54:59So, it's something you can kind of uh, work with Claude to kind of con- tune.
55:03But I haven't looked at like spoop or super specifically like uh, different like eval set sizes and like um I think the central concern I would have is more
55:13um it is probably overfitting and um and like variance.
55:18So, if you have a very very small number of samples, like okay, is that really representative of what you care about?
55:22Um uh it's a lot easier to overfit.
55:25So, like they may not be out of distribution for your production like what you actually care about.
55:30Um they may be easier to overfit.
55:32Uh they may be noisier.
55:34Um so, anyway, like I think those are the problems that arise when you're in like the very very small sample regime.
55:40Yeah, okay.
55:41So, the question is about hill climbing on tasks that are like very very long duration.
55:44So, kind of that's funny cuz that's actually why I mentioned I typically do like simple surfaces like prompts because they're like very very quick to change.
55:52Uh you are right though.
55:53There's some cases you're in a regime where like your changes could be very very long duration.
55:59That effectively makes hill climbing much harder cuz like it's a much longer duration task then.
56:02So, it's a lot longer time.
56:04That's actually where that gets into our final point about kind of harness design.
56:07If you're going to do that If you're doing like very very long duration hill climbing, you actually need harnesses that can like remain reliable for like many many hours to days.
56:15And so, we have we have that obviously internally.
56:17We use things like manage agents.
56:19Um I think it just becomes a bigger infrastructure lift when you're doing that.
56:23It can be done.
56:24Um it obviously is harder because of what I just mentioned, infrastructure issues, failures, um and just like a lot longer time to see your signal.
56:32Uh single turn versus multi-turn evals.
56:35Um I mean, yeah.
56:37When you're building agents, you do typically need multi-turn evals.
56:40Um Claude can help you do that.
56:43Particularly like the build eval command, it can look at traces.
56:46It can If you give it a bunch of traces, it can help you promote um it can help you promote particular traces to your eval set.
56:54Um in terms of grading, um I mean, yeah, there's a lot of different philosophies on this.
56:58You can have evaluators looking at the entire trace trajectory.
57:01You can have evaluators looking at like the final agent outcome.
57:04Uh there's a lot of different philosophies.
57:05The build eval command has a lot of guidance.
57:07So, like if you have a particular like agent you're interested in, if you basically give this command access to the traces, it should be pretty pretty effective at kind of
57:16shaping and designing the eval for you.
57:18And if you can tell it I want multi-turn and here's what I'm considering.
57:21Um cool.
57:23Interesting.
57:23So like for context engineering, what promotes of reliability a lot versus what's like overhyped as a technique?
57:29Um Uh let's see.
57:33Um So okay, the key thing I think and I've actually I've mentioned this in the past, we mentioned in the paper too.
57:41I think the recursive language model work in RLM idea is quite good.
57:45Um and we actually employ that idea here.
57:47I think a very important thing in terms of context engineering reliability is decoupling brain and hands.
57:52So basically designing your system such that it um the con- the key point is this, the context object is independent from the harness.
58:02So the harness is basically your cloud can effectively query this context object that lives independently.
58:08That's beneficial for all these different reasons.
58:10One, the context is always recoverable.
58:13So typically imagine alternatively you just have like, you know, a message history, you do a compaction event and everything you compacted is gone.
58:20You can never recover it if you haven't like backed it up in the way like this.
58:23That's why I actually like this architecture a lot for reliability in terms of context engineering.
58:27It does exactly what the RLM work advocated.
58:29Like it's basically splitting the context object from the model and the context object comes linked the model can interrogate through different mechanisms.
58:36And that's exactly what we do here.
58:38So interrogating that session log is basically done by the harness through different tools.
58:43Um and that's a very nice architecture I think.
58:45So I think that's like one of the tricks that I I found to be very nice for reliability, recoverability, um and in particular building like long duration agents, it's hard to know what will matter in the future.
58:56So when you do a compaction, um you don't necessarily know if you need to like go back and interrogate that context that you're summarizing.
59:04And when you have it backed up in an object like this, you can always do that.
59:08So I think that's a trick that I I I a lot.
59:11Uh and the RLM work was was very nice in that direction.
59:15Yep.
59:17Yeah.
59:17Well, I think [clears throat] that's why, you know, we spend a lot of time on um, good eval design.
59:26Um, yeah, these models, um, are non-deterministic.
59:32Um, in many cases the operating regimes where outcomes are not easily verifiable, like not everything is a coding problem.
59:39Many cases you want agents to do things that are more judgment-based, more free from open-ended outputs.
59:45In those regimes, you have to be very smart about your eval design so that you're giving Claude an objective that I can climb.
59:50And I think all the things we talked about in terms of like kind of verifier design, like good objectives, clear grading scales, multiple models can validate the result, low variance, all are like super important
60:03when designing like the problem that you want Claude to go and solve.
60:06So, I think, as kind of mentioned it in this talk, when it comes to unhambling frontier models, I think spending a lot of time up front in a very very good eval design is a way to make sure that you're getting the most out of the model because
60:20if you said tell the model, "Okay, optimize performance on this like very poorly structured evaluation."
60:25It won't get anywhere.
60:26And it's not the model's fault because you haven't thought through the eval, it's because cases are impossible or ambiguous.
60:31So, I think part of the message here is to just be thoughtful and spend a lot of time up front in the eval design such that you kind of set up frontier models for success and effectively unhamble them.
60:42That was kind of the the namesake of the talk.
60:45Maybe, um, I we're probably at time and everyone's probably tired.
60:48I was at a wedding last night, so I'm very tired.
60:51That was a funny timing, but it's very nice to be here, of course.
60:54And actually, it's been uh, it's been a lot of fun, very good questions.
60:57Um, maybe one more question over here.
61:00Mm.
61:00Yeah, so I think that's a good question.
61:02It's about like, yeah, models are getting better all the time, so like, how do I modify my evals to keep up?
61:08And I think simply it should be the case that as models get better, you should see performance on eval going up like we see in this diagram.
61:15If you don't see that, then I would go back and question the eval was structured.
61:18Um it is true like Yeah, we see this all the time.
61:22Models are saturating eval very quickly.
61:24So, we do need the development of new and harder evals that like keep up with the material.
61:29And actually, that's a good point.
61:31It's actually pretty hard.
61:32It's hard to come up with evals that really challenge like the best models today, like the Fables Opus 5.5 even.
61:39Um so, I do think it's a it's a it's a it's a recurring challenge um to design good evals that test the capabilities of the best models.
61:50It's absolutely a challenge.
61:51Um but I do think so, if you have evals that you built sometime ago, you should be seeing this trend on them.
61:58So, as models get better, you should be seeing performance going up and up and up.
62:01If you don't, I would question things about the eval itself.
62:04That's kind of the key point I would make.
62:06I mean, I've actually seen a lot of interesting work on this from the various trace and companies.
62:11So, like the the the like workflow I've seen quite a bit of is like you have agents that are deployed, they're like producing traces, you have an online eval you're looking at those traces and scoring them.
62:22Score Trace that score poorly are like then pulled into some offline eval set, you kind of then hill climb on that.
62:30And then that loop continues.
62:31That's a good That's like a very good pattern, I think.
62:34It's very reasonable.
62:35Uh and so, what's happening there is you're typically like hill climbing on these cases that you're failing on in production.
62:41And then you're like, you know, putting those in regression tests once they all pass, and that like continues until in production you're seeing very few failures.
62:48So, that actually is a good general way to structure your kind of eval loop and in many of the, you know, the LangChains, Rises, Braintrusts, all these companies kind of have product offerings that offer this kind of
63:04this like outer loop, which does seem quite quite effective.
63:08You do.
63:09So, the challenge there becomes okay, well, what are your online evaluators?
63:12So, like okay, how do you know what good is in production and what are you promoting to your offline eval sets?
63:16And Um, yeah, it's not easy at all.
63:21Um, I think that is Yeah, that's like there's like a lot of product work in that particular domain now.
63:28As far as I haven't followed that area too closely.
63:31Um, like what all the latest um, kind of updates that the various like tracing observability companies are working on, but my understanding is that it's like a very very active area of like product development among that class of companies.
63:43So, that's pretty interesting.
63:46Uh, okay, with that maybe we'll let everyone go.
63:48And um, very nice to see you all.
63:51So, feel free to reach out to me anywhere and thanks for the time.
63:55[applause]