WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ys6YQ2yfBpM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ys6YQ2yfBpM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.680 --> 00:00:13.520
We can kick it off.

s2
00:00:13.520 --> 00:00:14.920
Thank you all for coming.

s3
00:00:14.920 --> 00:00:15.760
I'm Lance.

s4
00:00:15.760 --> 00:00:18.320
I'm member of technical staff at Anthropic.

s5
00:00:18.320 --> 00:00:22.280
And I actually want to share some very simple things with you today.

s6
00:00:22.280 --> 00:00:26.480
A few simple ways to get the most out of frontier models.

s7
00:00:26.880 --> 00:00:31.400
So we typically think of cost and quality as a trade-off.

s8
00:00:31.400 --> 00:00:35.920
So in order to get, you know, better performance, you have to spend more money.

s9
00:00:35.920 --> 00:00:40.200
And in order to kind of reduce cost, you have to scratch sacrifice performance.

s10
00:00:40.200 --> 00:00:46.200
And so you can kind of think about it on any given evaluation, you have models that kind of create this Pareto frontier.

s11
00:00:46.200 --> 00:00:54.360
And you can imagine this case, Fable is kind of up on the far right of the curve, and then you have Opus, you have Sonnet, then you have Haiku.

s12
00:00:54.360 --> 00:01:01.160
Now what I want to share is actually that many times you're not actually on this frontier.

s13
00:01:01.160 --> 00:01:05.519
Your application is actually sitting inside this frontier that you could be on.

s14
00:01:05.519 --> 00:01:08.600
And it's because of minor misconfigurations.

s15
00:01:08.600 --> 00:01:11.395
Which I'm going to show you today how to fix.

s16
00:01:11.395 --> 00:01:11.640
[snorts]

s17
00:01:11.640 --> 00:01:14.760
And we've seen this repeatedly.

s18
00:01:15.440 --> 00:01:21.840
Improve performance and reduce cost for many customers using some view very simple things.

s19
00:01:22.280 --> 00:01:24.720
So first, I'm going to cover five themes.

s20
00:01:24.720 --> 00:01:26.400
And then we can make it interactive.

s21
00:01:26.400 --> 00:01:27.880
The slides won't take an hour.

s22
00:01:27.880 --> 00:01:28.600
Not even close.

s23
00:01:28.600 --> 00:01:32.480
So I question answering is actually much more fun.

s24
00:01:32.480 --> 00:01:34.880
First, audit your prompts.

s25
00:01:34.880 --> 00:01:41.040
This is a kind of important point that prompts accumulate debt just like code.

s26
00:01:41.040 --> 00:01:46.120
So if you recall several generations ago, we used to use things like think step-by-step.

s27
00:01:46.120 --> 00:01:49.560
We used to use verification rituals, double-check your work.

s28
00:01:49.560 --> 00:01:53.640
We used to use thought from this boosters, be maximally thorough.

s29
00:01:53.640 --> 00:02:00.492
Sometimes we would use kind of scratch pads, tell the agent how to think, give it places to think.

s30
00:02:00.492 --> 00:02:02.332
You don't need that anymore.

s31
00:02:02.332 --> 00:02:07.932
Frontier models are very good at at thinking, choosing when to think.

s32
00:02:07.932 --> 00:02:11.932
They don't need to be told what to do and like kind of how to verify their work.

s33
00:02:11.932 --> 00:02:14.012
They naturally verify their work.

s34
00:02:14.012 --> 00:02:15.932
Um they don't need told to be thoughtful.

s35
00:02:15.932 --> 00:02:17.772
They're already very thoughtful.

s36
00:02:17.772 --> 00:02:28.052
Um and worse, when you do these things, it actually increases cost and can reduce performance in some ways I'll talk about briefly.

s37
00:02:28.052 --> 00:02:37.812
Another interesting one with earlier models, if your prompt had contradictory rules in them, earlier models that weren't as good at instruction following would kind of ignore one of the two rules.

s38
00:02:37.812 --> 00:02:39.052
So it kind of still worked.

s39
00:02:39.052 --> 00:02:44.572
But with frontier models, they'll often get confused by contradictory rules.

s40
00:02:44.572 --> 00:02:50.852
So I'm going to give you one very simple thing to do right now if you have any code that you want to check.

s41
00:02:50.852 --> 00:02:54.732
In Claude code, you can run this command checkup prompt audit.

s42
00:02:54.732 --> 00:02:56.612
You can also run Claude API prompt audit.

s43
00:02:56.612 --> 00:02:58.732
These are two different skills in Claude code.

s44
00:02:58.732 --> 00:03:00.292
They link to the same code.

s45
00:03:00.292 --> 00:03:06.252
It's just checkup is typically how people do checkups on their Claude code configuration itself.

s46
00:03:06.252 --> 00:03:16.012
So your skills, your prompts within Claude code itself, the Claude API skill is a separate skill bundled in Claude code that's meant for a kind of code that uses our APIs.

s47
00:03:16.012 --> 00:03:21.532
But both of these commands use the same principle and they kind of link down to the same underlying skill itself.

s48
00:03:21.532 --> 00:03:23.707
It's just two different ways to call it.

s49
00:03:23.707 --> 00:03:24.612
[snorts]

s50
00:03:24.612 --> 00:03:29.932
These skills will audit your code, anything in the working directory of Claude code.

s51
00:03:29.932 --> 00:03:35.612
So that could be skills, that could be tools, that could be prompts for these anti-patterns and will remove them for you.

s52
00:03:35.612 --> 00:03:37.092
This has become kind of very popular.

s53
00:03:37.092 --> 00:03:39.412
Many people we all Many people use it within Anthropic.

s54
00:03:39.412 --> 00:03:42.572
It's kind of become somewhat popular, you know, on Twitter as well.

s55
00:03:42.572 --> 00:03:49.252
Um it's very simple and often it can improve cost and performance.

s56
00:03:49.252 --> 00:03:54.212
So here's an example uh where it This is an evaluation that I built.

s57
00:03:54.212 --> 00:03:57.652
It's a synthetic evaluation of customer support tickets.

s58
00:03:57.652 --> 00:03:58.772
There's like 50 tickets.

s59
00:03:58.772 --> 00:04:03.534
And what I do is I start with Opus 48.

s60
00:04:03.534 --> 00:04:09.534
And I basically simulate a migration to Opus 55, okay?

s61
00:04:09.534 --> 00:04:15.934
And in this process, I inject each anti-pattern individually and measure the result.

s62
00:04:15.934 --> 00:04:18.094
That's this middle point, okay?

s63
00:04:18.094 --> 00:04:23.374
So, what happens is it's taking the average of all the anti-patterns across the benchmark, basically.

s64
00:04:23.374 --> 00:04:25.534
The key point is this.

s65
00:04:25.534 --> 00:04:36.854
When I do the migration to Opus 55 from 48, initially with these kind of anti-patterns injected, we do see some cost savings and a little bit performance savings.

s66
00:04:36.854 --> 00:04:40.094
You can see it goes to the left and it goes up a bit.

s67
00:04:40.094 --> 00:04:48.894
If I run prompt audit on top of that, I get a further bump both in performance and cost reduction for a bunch of different reasons.

s68
00:04:48.894 --> 00:04:58.014
Because this command is going through and it's removing verification rituals, boosters that are actually causing Opus 55 to expend excessive tokens.

s69
00:04:58.014 --> 00:05:11.054
So, this little simple command often will basically reduce token use, saving cost, and it can improve performance in cases where things like contradictory prompts were confusing frontier models.

s70
00:05:11.054 --> 00:05:13.494
So, this is a very simple command that I recommend.

s71
00:05:13.494 --> 00:05:18.374
Um and we've used it very extensively within Anthropic and also many customers have enjoyed this quite a lot.

s72
00:05:18.374 --> 00:05:19.734
It's very simple to run.

s73
00:05:19.734 --> 00:05:28.534
You run it kind of in any code using the API or even on your cloud code configuration to make sure your prompts are kind of well-tuned for frontier models.

s74
00:05:28.534 --> 00:05:30.614
That's point one.

s75
00:05:30.614 --> 00:05:34.454
Now, point two is make sure you're using the prompt cache.

s76
00:05:34.454 --> 00:05:39.694
This is probably pretty familiar to many of users, but I do just want to make sure it's extremely clear.

s77
00:05:39.694 --> 00:05:43.974
So, agents, for example, operate across many turns.

s78
00:05:43.974 --> 00:05:45.974
Every turn adds just a little bit.

s79
00:05:45.974 --> 00:05:47.974
For example, one tool call.

s80
00:05:47.974 --> 00:05:51.054
Much of kind of context is repeated across turns.

s81
00:05:51.054 --> 00:05:52.934
You don't want to pay for it twice.

s82
00:05:52.934 --> 00:05:54.214
That's why you can use the prompt cache.

s83
00:05:54.214 --> 00:05:58.054
So, prompt cache is built at like a tenth the the rate of base tokens.

s84
00:05:58.054 --> 00:05:59.814
So, it's much cheaper.

s85
00:05:59.814 --> 00:06:00.894
Varies a bit per model.

s86
00:06:00.894 --> 00:06:05.455
I think 10 a a tenth is I think the current pricing, but you can check the pricing page.

s87
00:06:05.455 --> 00:06:07.575
The point is it's much cheaper.

s88
00:06:07.575 --> 00:06:09.015
It's also much faster.

s89
00:06:09.015 --> 00:06:14.535
So, it's very important in multi-turn applications to be making heavy use of the prompt cache.

s90
00:06:14.815 --> 00:06:20.215
Now, the problem is prompt cache are a strict prefix match.

s91
00:06:20.215 --> 00:06:25.335
So, any even a byte change in your prefix will invalidate the cache.

s92
00:06:25.335 --> 00:06:29.855
So, you have to be careful in building agents or multi-turn applications that you don't break the prompt cache.

s93
00:06:29.855 --> 00:06:32.335
That's the key point.

s94
00:06:32.695 --> 00:06:36.895
And Claude console, if you're using the API, has nice diagnostics for you.

s95
00:06:36.895 --> 00:06:43.175
So, if you're building applications with the Claude API, you can see for your application your prompt cache hit rate, which is very important to track.

s96
00:06:43.175 --> 00:06:48.455
For example, with Claude code, we track the prompt cache hit rate very very religiously.

s97
00:06:48.455 --> 00:06:51.495
It's very important for the economics of the application, obviously.

s98
00:06:51.495 --> 00:06:59.855
And it's also in your cases, if you're building apps with the Claude API, it's very very important to make sure you're using making heavy use of the prompt cache.

s99
00:07:00.215 --> 00:07:03.775
Now, in general, how to organize your prompts.

s100
00:07:03.775 --> 00:07:10.215
You know, you can think about basically, um system instructions, tools, things that are globally cached across all of your sessions.

s101
00:07:10.215 --> 00:07:11.935
You can think about Claude MDM memory.

s102
00:07:11.935 --> 00:07:14.175
That's kind of cached per project.

s103
00:07:14.175 --> 00:07:17.935
Session state is kind of per session, and then messages kind of incremented each turn.

s104
00:07:17.935 --> 00:07:20.495
So, that's kind of how you want to structure your prompts.

s105
00:07:20.495 --> 00:07:24.055
And the key point is this, run this in Claude code.

s106
00:07:24.055 --> 00:07:29.695
So, if you have a project using the API, you want to make check and make sure that you're using you're making proper use of the prompt cache.

s107
00:07:29.695 --> 00:07:30.335
You can run this.

s108
00:07:30.335 --> 00:07:34.375
It'll run diagnostics to make sure that you're using the prompt cache effectively.

s109
00:07:34.375 --> 00:07:40.095
It'll find opportunities to reorganize your prompts to make sure that they're organized such that you get a high prompt cache hit rate.

s110
00:07:40.095 --> 00:07:42.435
That's really the key point here.

s111
00:07:42.435 --> 00:07:43.015
[sighs]

s112
00:07:43.015 --> 00:07:45.215
And this is kind of a fun exercise.

s113
00:07:45.215 --> 00:07:53.935
So, myself and some folks on our Claude AI team ran um ran this command on a bunch of internal benchmarks that had kind of default harnesses.

s114
00:07:53.935 --> 00:08:05.175
We found opportunities to reduce costs by up to 72% with no meaningful change in performance, just by running this command, because some of the harnesses on these Evals were not using the prompt cache properly.

s115
00:08:05.175 --> 00:08:06.415
Really simple command.

s116
00:08:06.415 --> 00:08:13.457
You can run it in your API code and it'll just look for opportunities to make sure you're saving as much cost as you should be, i.e. by using the prompt cache.

s117
00:08:13.457 --> 00:08:14.617
It'll also check for other things.

s118
00:08:14.617 --> 00:08:19.897
It'll look for um I'll I'll talk about it a little bit later, but it'll also check your effort levels, it'll check your prompt.

s119
00:08:19.897 --> 00:08:26.817
So, it it'll do a number of other things, but in particular it'll look at your prompt cache to make sure you have a high prompt cache hit rate.

s120
00:08:26.817 --> 00:08:31.977
Again, this has, you know, optimized prompt cache has no change in performance.

s121
00:08:31.977 --> 00:08:34.177
It's like effectively just free cost savings.

s122
00:08:34.177 --> 00:08:38.737
So, it's something to be very serious about when you're building applications with the API in particular.

s123
00:08:38.737 --> 00:08:44.177
Um it's very something to very to very seriously make sure you're utilizing effectively.

s124
00:08:45.377 --> 00:08:49.337
So, I want to get into to some other fundamentals.

s125
00:08:49.337 --> 00:08:55.697
Um we talked about prompt caching, we talked about making sure your prompts are kind of well that are that are well-tuned.

s126
00:08:55.697 --> 00:08:58.297
You can audit them using prompt cache.

s127
00:08:58.297 --> 00:09:02.537
Now, let's talk about a few other things that are very important for working with frontier models.

s128
00:09:02.537 --> 00:09:09.097
So, Claude, frontier Claude in particular, is very good at self-correction.

s129
00:09:09.097 --> 00:09:16.457
So, if you give Claude a well-structured evaluation, it can basically just hill climb on that in that eval and I'll talk about that in a minute.

s130
00:09:16.457 --> 00:09:19.857
But important in that is good eval design.

s131
00:09:19.857 --> 00:09:23.177
And you have to give Claude an evaluation that is climbable.

s132
00:09:23.177 --> 00:09:33.497
And so, actually we spend a lot of time thinking about eval design and making sure Claude is set up for success when building an eval and setting up hill climbing.

s133
00:09:33.577 --> 00:09:36.097
So, some general features of good eval design.

s134
00:09:36.097 --> 00:09:43.337
This is This is basics, this is nothing [clears throat] related to, you know, Anthropic, this is just kind of fundamentals, but it's good to revisit briefly.

s135
00:09:43.697 --> 00:09:49.137
What you should see generally is that these are a number of different curves looking at basically tokens per attempt.

s136
00:09:49.137 --> 00:09:58.257
This is kind of like thinking on the X and then score on the Y. Frontier model at the top, mid-size model on in the middle curve, small model at the bottom.

s137
00:09:58.257 --> 00:10:01.817
So, think about that as like Opus, Sonnet, Haiku.

s138
00:10:01.817 --> 00:10:06.177
You should generally see on your evaluation as you step up model and performance goes up.

s139
00:10:06.177 --> 00:10:07.937
That's point one.

s140
00:10:07.937 --> 00:10:10.178
As you increase thinking, performance should go up.

s141
00:10:10.178 --> 00:10:11.898
That's point two.

s142
00:10:11.898 --> 00:10:14.778
There should be headroom to improve at the frontier.

s143
00:10:14.778 --> 00:10:15.938
So, that's point three.

s144
00:10:15.938 --> 00:10:18.578
So, basically shouldn't be saturated.

s145
00:10:18.578 --> 00:10:21.258
And there should be low run-run variance.

s146
00:10:21.258 --> 00:10:24.778
Very simple points, uh but actually very important to get right.

s147
00:10:24.778 --> 00:10:34.058
And actually even a lot of very popular evaluations that we see have many tasks that are ambiguous, effectively limiting that headroom at the frontier.

s148
00:10:34.058 --> 00:10:35.818
And this is a problem I see all the time.

s149
00:10:35.818 --> 00:10:39.738
You build an evaluation, it turns out you have a bunch of ambiguous cases that can't be solved anyway.

s150
00:10:39.738 --> 00:10:45.378
So, it looks like, you know, oh, I've 20% more that I can get on this, but no model can actually achieve it.

s151
00:10:45.378 --> 00:10:49.778
So, you could spend many tokens [clears throat] hill climbing on an evaluation that's effectively saturated.

s152
00:10:49.778 --> 00:10:52.387
It's very important to get this right.

s153
00:10:52.387 --> 00:10:53.458
[sighs]

s154
00:10:53.458 --> 00:10:54.618
Try this command.

s155
00:10:54.618 --> 00:10:57.018
Uh Claude API build eval.

s156
00:10:57.018 --> 00:11:03.378
This is actually a very nice command because it's not opinionated at all about where your data lives.

s157
00:11:03.378 --> 00:11:04.058
It doesn't care.

s158
00:11:04.058 --> 00:11:07.418
You can point this at whatever tracing solution you have.

s159
00:11:07.418 --> 00:11:10.298
It's just a skill in Claude code.

s160
00:11:10.298 --> 00:11:16.138
It's more employing the principles I talked about to test and sanity check the evaluation for you.

s161
00:11:16.138 --> 00:11:26.738
So, you can kind of give this skill, hey, here's a problem I want to solve, here's what I'm interested in, here is the raw data, for example, here's the traces that I want to work with, and this command can help you build an eval set.

s162
00:11:26.738 --> 00:11:30.178
It'll walk you through a process and step you through building it.

s163
00:11:30.178 --> 00:11:37.658
It'll it'll help you you'll show you the data, it'll show you the evaluation, it'll sanity it'll sanity check the grader the criteria with you.

s164
00:11:37.658 --> 00:11:44.738
So, it'll kind of walk you through the process of building eval based upon some of our, you know, some of our best practice and we have a nice blog post on this as well that

s165
00:11:44.738 --> 00:11:46.698
that I'll share.

s166
00:11:46.698 --> 00:11:50.218
Um and it pairs very well with hill climbing.

s167
00:11:50.218 --> 00:11:58.698
So, once you've built a good evaluation that has headroom at the frontier, low variance, scales as expected with thinking and model, you can then climb on it.

s168
00:11:58.698 --> 00:12:01.818
Frontier Claude models are very, very effective at hill climbing.

s169
00:12:01.818 --> 00:12:08.458
So, autonomously looking at an evaluation and autonomously kind of improving on it without you in the loop.

s170
00:12:08.458 --> 00:12:13.138
So, you build a good eval, Claude can hill climb on it.

s171
00:12:13.219 --> 00:12:17.259
The problem is there's many ways that this can kind of go wrong.

s172
00:12:17.259 --> 00:12:22.379
And in particular, you can end up overfitting on the evaluation itself.

s173
00:12:22.379 --> 00:12:32.259
So like some classic examples would be um like the task in your benchmark have particular needs that are bespoke or specific to those tasks.

s174
00:12:32.259 --> 00:12:39.059
Like maybe one task needs OCR, so it happens your harness to solve the eval adds like an OCR tool.

s175
00:12:39.059 --> 00:12:40.899
But in production, you don't see that problems.

s176
00:12:40.899 --> 00:12:44.939
So you end up overfitting on the eval distribution, it doesn't apply to production.

s177
00:12:44.939 --> 00:12:46.979
This is very common.

s178
00:12:46.979 --> 00:12:52.859
Um yeah, like distinctive phrasing specific to your eval examples.

s179
00:12:52.979 --> 00:13:05.019
For example, another classic is like it just kind of it like the prompt when you're basically hill climbing overfits the specifics of your eval distribution, it doesn't apply to to your to your test distribution.

s180
00:13:05.019 --> 00:13:08.459
So overfitting is the major challenge concern when you're hill climbing.

s181
00:13:08.459 --> 00:13:11.779
Imagine you spent a lot of tokens hill climbing an evaluation.

s182
00:13:11.779 --> 00:13:13.539
It looks really good after hill climbing.

s183
00:13:13.539 --> 00:13:16.419
You run it in production and performance is not better at all.

s184
00:13:16.419 --> 00:13:18.779
That's frustrating because you spent a lot of money.

s185
00:13:18.779 --> 00:13:27.539
You hill climbed and like, oh, it's not applicable to production because often you'll you've just kind of overfit to the eval distribution.

s186
00:13:27.619 --> 00:13:30.019
So try this command.

s187
00:13:30.019 --> 00:13:31.859
In Claude code is Claude API hill climb.

s188
00:13:31.859 --> 00:13:34.699
So if you build an eval, it'll hill climb for you.

s189
00:13:34.699 --> 00:13:37.979
And it tries to combat this with a few tricks.

s190
00:13:37.979 --> 00:13:39.299
Now it's of course not perfect.

s191
00:13:39.299 --> 00:13:44.859
You have to audit this carefully and and and indeed be in the loop and and make sure you really understand what's happening.

s192
00:13:44.859 --> 00:13:46.659
But it can help you.

s193
00:13:46.659 --> 00:13:51.339
And in particular, what it does is a process like this.

s194
00:13:51.339 --> 00:13:56.019
It will look at your evaluation and it will split it into test and train sets.

s195
00:13:56.019 --> 00:14:00.219
This is kind of very classic um kind of classic principles from statistics.

s196
00:14:00.219 --> 00:14:02.419
Create a test and train split.

s197
00:14:02.419 --> 00:14:05.979
What happens is when hill climbing, this is just on Claude code by the way.

s198
00:14:05.979 --> 00:14:08.299
So Claude code basically has this skill.

s199
00:14:08.299 --> 00:14:10.619
You run it on an eval.

s200
00:14:10.619 --> 00:14:18.261
It will only ever look at the train split and reflect on those failures when it's making decisions about what to change to improve on the eval.

s201
00:14:18.261 --> 00:14:24.421
So, what's nice about that is at every round, it never sees a test split.

s202
00:14:24.421 --> 00:14:28.901
And it will basically check performance per round against test and train.

s203
00:14:28.901 --> 00:14:32.741
So, what happens is a little it's a little early signal for overfitting.

s204
00:14:32.741 --> 00:14:38.541
If you see train improving but test flat, it's a signal of "Ooh, you might be overfitting."

s205
00:14:38.541 --> 00:14:48.461
So, basically it applies this check and a number of other more subtle things check to make sure while hill climbing's happening, you aren't just overfitting to your eval distribution, you're actually

s206
00:14:48.461 --> 00:14:51.541
hill climbing in a way that will apply to production.

s207
00:14:51.541 --> 00:14:57.861
Assuming you built your eval distribution well, and that's always something that we can address in that build eval phase.

s208
00:14:58.021 --> 00:15:01.021
Now, I'll share some concrete examples that I've used here.

s209
00:15:01.021 --> 00:15:04.901
So, this is actually the Claude API skill itself.

s210
00:15:04.901 --> 00:15:10.061
So, if I back up a little bit, this is a skill in Claude code.

s211
00:15:10.061 --> 00:15:15.741
And it is a skill that gives Claude full awareness of our APIs.

s212
00:15:15.741 --> 00:15:18.181
Uh so, basically our SDKs, APIs.

s213
00:15:18.181 --> 00:15:22.421
And it has to be grounded in our documentation.

s214
00:15:22.421 --> 00:15:26.221
So, I built an eval set based upon our documentation.

s215
00:15:26.221 --> 00:15:28.181
And I tested the skill.

s216
00:15:28.181 --> 00:15:32.861
The skill initially was not very good at the eval questions in my set, 66%.

s217
00:15:32.861 --> 00:15:45.181
So, I set up a hill climbing loop with giving Claude code access to the skill text and the documentation itself, and let it basically just iteratively look at the failures in the eval set, and try to figure out why

s218
00:15:45.181 --> 00:15:47.741
the skill was misrepresenting what's in the documentation.

s219
00:15:47.741 --> 00:15:51.781
And there's all sorts of little tricks, and I could, you know, talk about that more in more detail.

s220
00:15:51.781 --> 00:16:00.501
But the point is, Claude can autonomously find, in this particular case, the right skill text to improve significantly on the evaluation.

s221
00:16:00.501 --> 00:16:03.101
And I talk about this in the blog link below.

s222
00:16:03.101 --> 00:16:16.901
Um but again, this is a good example of a kind of a well-scoped evaluation, a simple service I'm modifying a skill, and an autonomous hill climbing loop that resulted in a much better performance just when I set the system up this way with a carefully scoped eval,

s223
00:16:16.901 --> 00:16:22.502
a surface that's easy to modify, claw can just rip through it and like significantly improve performance.

s224
00:16:22.502 --> 00:16:25.782
This works well for prompts, it works well for skills.

s225
00:16:25.782 --> 00:16:30.542
Simple text-based surfaces are very very good for this kind of hill climbing process.

s226
00:16:30.862 --> 00:16:36.182
Now, this is another kind of interesting trick that I've seen a lot of customers like.

s227
00:16:36.182 --> 00:16:39.742
You can also apply hill climbing to cost reduction.

s228
00:16:39.742 --> 00:16:44.302
So, what's kind of nice about this is I've added guidance to this hill climb command.

s229
00:16:44.302 --> 00:16:55.222
If you tell it to reduce cost, what it'll do is it will basically try to find a configuration that's that is at is at least at parity in performance and for sure reduces cost.

s230
00:16:55.222 --> 00:16:58.382
And it'll search things like effort, model, and your prompts.

s231
00:16:58.382 --> 00:17:02.702
Here's a simple example on that same um customer support benchmark.

s232
00:17:02.702 --> 00:17:06.262
I start with Opus Opus 48 high with some baseline prompt.

s233
00:17:06.262 --> 00:17:09.142
And I basically tell the hill climber reduce cost.

s234
00:17:09.142 --> 00:17:21.262
So, what it does, it basically will search through model effort combinations and basically pick those that continuously reduces cost over the rounds without degrading performance.

s235
00:17:21.262 --> 00:17:23.102
In this case, it moves to Opus 5 low.

s236
00:17:23.102 --> 00:17:27.622
We see a big bump in performance and minor reduction in cost.

s237
00:17:27.622 --> 00:17:30.622
The hill climber then autonomously tries flying at 5 low.

s238
00:17:30.622 --> 00:17:35.102
We see again degrades performance a bit, but big cost reduction.

s239
00:17:35.102 --> 00:17:39.982
And then it it then decide tune the prompt and we kind of risk rescue the performance.

s240
00:17:39.982 --> 00:17:46.302
So, we basically got a 90% reduction in cost with notable improvement in performance autonomously using the hill climber.

s241
00:17:46.302 --> 00:17:56.102
This is another very nice way if you have an evaluation, you can use the hill climber to basically say, "Okay, keep performance fixed but reduce my cost materially by tuning things like effort,

s242
00:17:56.102 --> 00:17:59.022
things like model selection, and my prompts."

s243
00:17:59.022 --> 00:18:00.542
Another very nice use of the hill climber.

s244
00:18:00.542 --> 00:18:05.902
And actually, what's nice about this is you can do this if you have a saturated eval.

s245
00:18:05.902 --> 00:18:08.462
So, you can basically say, "Okay, say my eval saturated, fine.

s246
00:18:08.462 --> 00:18:09.062
It doesn't matter.

s247
00:18:09.062 --> 00:18:10.222
I'm not I don't care about performance.

s248
00:18:10.222 --> 00:18:12.022
I just want to reduce cost."

s249
00:18:12.022 --> 00:18:16.782
So, then you can just tell the hill climber, "Okay, keep performance fixed and just try to burn down cost."

s250
00:18:16.782 --> 00:18:21.143
And it can basically search for configurations and often it'll find very interesting things.

s251
00:18:21.143 --> 00:18:30.623
Like in this particular case, there's generally an intuition that moving to higher models at lower effort is often cheaper and more performant than lower models at high effort.

s252
00:18:30.623 --> 00:18:38.623
It's like giving someone who's smarter can like try less to actually get a better objective or or or to kind of solve the problem more effectively.

s253
00:18:38.623 --> 00:18:43.703
Um and so it ends up actually using fewer tokens and as a result ends up being cheaper.

s254
00:18:43.703 --> 00:18:46.583
That's kind of a broad intuition here that we've seen quite a bit.

s255
00:18:46.583 --> 00:18:49.423
And that's baked into this hill climber skill.

s256
00:18:49.423 --> 00:18:53.463
So, um another nice trick.

s257
00:18:53.463 --> 00:18:55.503
Now, I'm going to open up the questions pretty soon.

s258
00:18:55.503 --> 00:18:57.503
I'll just kind of close with some other comments.

s259
00:18:57.503 --> 00:19:01.263
And this is a bit of a nod to the magic agent stuff you heard this morning.

s260
00:19:01.263 --> 00:19:02.783
I just want to re-emphasize this a little bit.

s261
00:19:02.783 --> 00:19:12.983
I do think um as part of unhobbling the frontier, it's very important to design harnesses that can take most advantage of the latest Claude models.

s262
00:19:12.983 --> 00:19:23.023
And in particular, it's interesting if you look back the last few years, time is on the X, and this is task duration for human on the Y. You've seen these curves.

s263
00:19:23.023 --> 00:19:25.663
They're just meter charts.

s264
00:19:25.703 --> 00:19:32.303
It's interesting to think back, you know, Claude Opus 3 was capable of only like 4 minutes of autonomous work.

s265
00:19:32.303 --> 00:19:36.023
And in that kind of regime, there were very limited products you could build.

s266
00:19:36.023 --> 00:19:38.503
It's really auto completion or chat.

s267
00:19:38.503 --> 00:19:41.183
Now, we saw over the course of 2025, models become more capable.

s268
00:19:41.183 --> 00:19:42.663
They can do like hours of work.

s269
00:19:42.663 --> 00:19:45.223
Like maybe an hour to 2 hours of human equivalent work.

s270
00:19:45.223 --> 00:19:51.703
Claude code obviously kind of um gained significant over the past over 2025.

s271
00:19:51.703 --> 00:19:59.583
And we're kind of entering a new regime now where these um task durations for models are are exceeding like a day to 2 days.

s272
00:19:59.583 --> 00:20:03.663
I think this is kind of outside the scope of what meter can even measure now.

s273
00:20:03.663 --> 00:20:10.343
But in this regime where you have models that can do the equivalent of like, say, days of work, what harnesses do you need?

s274
00:20:10.343 --> 00:20:16.463
Like the considerations change quite a bit because you want models that can just do things autonomously over very, very long periods of time.

s275
00:20:16.463 --> 00:20:18.783
Now, how do you make sure they're secure and reliable?

s276
00:20:18.783 --> 00:20:21.503
The considerations change a lot.

s277
00:20:21.503 --> 00:20:28.625
And I hear you heard about that morning, but this is actually the motivation behind one of our newer APIs called Cloud Manage Agents.

s278
00:20:28.625 --> 00:20:35.545
Um and the principle here, I wrote this engine blog back in April on it, is decoupling the brain from the hands.

s279
00:20:35.545 --> 00:20:37.105
And the idea is pretty simple here.

s280
00:20:37.105 --> 00:20:39.065
You have an agent harness, which is effectively the brain.

s281
00:20:39.065 --> 00:20:40.385
You have hands, which are sandboxes.

s282
00:20:40.385 --> 00:20:43.865
That's where the agent harness writes code and performs actions.

s283
00:20:43.865 --> 00:20:46.225
And you want those to be separate things.

s284
00:20:46.225 --> 00:20:53.305
In particular, you want that you want it to be the case that if the harness dies or the sand box dies, the system can recover.

s285
00:20:53.305 --> 00:20:58.545
Imagine you have an agent that you spin up overnight to just run and the sandbox dies midway through the night.

s286
00:20:58.545 --> 00:20:59.545
It's very frustrating.

s287
00:20:59.545 --> 00:21:02.745
You lose state, you lose your session log, and everything.

s288
00:21:02.745 --> 00:21:07.185
So when we design this, we designed this for a world where Cloud would be doing work for very, very long durations.

s289
00:21:07.185 --> 00:21:08.985
And so it's recoverable.

s290
00:21:08.985 --> 00:21:16.505
So that means the session log is an independent object separate from the harness, separate from the sandbox, that is append-only, that is recoverable.

s291
00:21:16.505 --> 00:21:21.025
So if the harness dies, the sandbox dies, you can always recover from the session log.

s292
00:21:21.025 --> 00:21:22.905
Sandboxes are also recoverable.

s293
00:21:22.905 --> 00:21:25.025
So if a sandbox dies, you can boot up a new one.

s294
00:21:25.025 --> 00:21:28.425
If the harness dies, you can boot up a new one through this orchestrator.

s295
00:21:28.425 --> 00:21:30.665
Another point here is safety.

s296
00:21:30.665 --> 00:21:36.145
So one of the important things that Manage Agents does is it actually never gives Cloud direct access to secrets.

s297
00:21:36.145 --> 00:21:38.025
Secrets are held in vaults.

s298
00:21:38.025 --> 00:21:42.065
Um and that's actually stored in that kind of tools box you see at the top.

s299
00:21:42.065 --> 00:21:46.225
So vaults ensure that Cloud can reference API keys or secrets, but it can't actually see them.

s300
00:21:46.225 --> 00:21:54.185
Because obviously, if you have an agent that's working for, say, days on end, it's a little bit spooky to give it direct access to very kind of sensitive secrets.

s301
00:21:54.185 --> 00:22:02.025
And so we ensure the security is is kind of very is is a very important consideration when we build Manage Agents.

s302
00:22:02.025 --> 00:22:09.225
So reliability and security become much more important as agents can operate for hours, days, and maybe soon weeks.

s303
00:22:09.425 --> 00:22:19.345
And this morning you probably heard a little more about this, but we have a nice uh recent blog that I put out with CJ that walks through uh an example

s304
00:22:19.345 --> 00:22:26.385
of basically building an agent that can serve as like kind of a an an agent automation uh for regular context updates.

s305
00:22:26.385 --> 00:22:30.866
It can reach into different sources that you specify, it can write updates to Slack.

s306
00:22:30.866 --> 00:22:35.986
Very simple automation example, it's highly customizable, but it shows how all these different primitives come together.

s307
00:22:35.986 --> 00:22:49.666
It brings together a scheduled deployment, all this all runs on Anthropic infrastructure, it secure- it securely stores secrets, it has memory, and um it's a very nice kind of end-end example.

s308
00:22:49.666 --> 00:22:53.586
Now again, the final point is you can just run this in Cloud Code.

s309
00:22:53.586 --> 00:23:03.146
So, run Cloud API magic sandboard and then give it the link to the blog, and it'll just boot up this tutorial for you, it'll spin up the example, and you can just play with it.

s310
00:23:03.146 --> 00:23:13.946
Um so, the really the key point here is um there's a number of ways you can kind of harness and uh make sure that Frontier is effectively unhobbled.

s311
00:23:13.946 --> 00:23:17.786
Uh it is very important to audit your prompts.

s312
00:23:17.786 --> 00:23:23.946
It's very important to make sure that you're effectively using prompt cache, especially building agents for long duration, to make sure that cost is managed.

s313
00:23:23.946 --> 00:23:29.746
It's important to think about harnesses that are designed for long duration in terms of reliability and security.

s314
00:23:29.746 --> 00:23:32.346
And it's also important to think about evaluations and hill climbing.

s315
00:23:32.346 --> 00:23:43.946
And when this all comes together, you can have agents that you can give a task, for example, an evaluation, and let it run for long period of time and hill climb on that evaluation, come back, for example, in the morning, and it's it's significantly improved.

s316
00:23:43.946 --> 00:23:46.746
And that's really the way that I see many of my colleagues working now.

s317
00:23:46.746 --> 00:23:58.146
People are spending much more time on eval design, and basically just specifying tasks that Claude can effectively hill climb on, and doing this in a way such that agent is highly secure when doing this,

s318
00:23:58.146 --> 00:24:02.226
and is highly reliable for long duration jobs.

s319
00:24:02.226 --> 00:24:03.946
And let's see where we are on time.

s320
00:24:03.946 --> 00:24:08.306
With that, I think I'll just take it to questions, because it's more fun when it's interactive anyway.

s321
00:24:08.306 --> 00:24:11.986
You can follow me on all the the socials here.

s322
00:24:11.986 --> 00:24:15.706
And uh thank you for your time and attention.

s323
00:24:16.736 --> 00:24:18.736
[applause]

s324
00:24:25.026 --> 00:24:27.216
Sure, start there.

s325
00:24:27.216 --> 00:24:27.786
[laughter]

s326
00:24:27.786 --> 00:24:29.148
This is a very good question.

s327
00:24:29.148 --> 00:24:32.388
The question was how automatable is eval design?

s328
00:24:32.388 --> 00:24:35.987
So actually I was engaging a bit with with Hamel on Twitter.

s329
00:24:35.987 --> 00:24:38.148
Extremely thoughtful with respect to evals.

s330
00:24:38.148 --> 00:24:44.908
And some feedback he had on our skill was that it should indeed walk the user through the data first, which I actually agree with.

s331
00:24:44.908 --> 00:24:48.388
I'll make it some minor modifications to the skill to do that.

s332
00:24:48.388 --> 00:24:56.227
So I think that uh eval design, I'll go kind of go back to it.

s333
00:24:56.227 --> 00:25:00.268
Claude can assist you when building evals significantly.

s334
00:25:00.268 --> 00:25:08.067
So for example, in the example he gave when his walk-through, um he was looking at like a it was like a customer support app or something.

s335
00:25:08.067 --> 00:25:13.547
And the first question was like, okay, like it's a bunch of traces.

s336
00:25:13.547 --> 00:25:15.148
What are the failure cases here?

s337
00:25:15.148 --> 00:25:17.668
What should I even like promote to an eval set?

s338
00:25:17.668 --> 00:25:21.747
So Claude's really good at kind of identifying, ooh, okay, here's case where there might be errors.

s339
00:25:21.747 --> 00:25:29.307
It's effectively like acting as like an online evaluator, but looking at your offline logs, scanning them saying, okay, here's some problematic examples.

s340
00:25:29.307 --> 00:25:32.107
And Claude was very good at that in his example.

s341
00:25:32.107 --> 00:25:40.467
Then it is actually important and quite useful if Claude then shows those raw like the raw data to you in a viewer.

s342
00:25:40.467 --> 00:25:42.028
So I'm going to add that to the skill.

s343
00:25:42.028 --> 00:25:44.187
I have a PR up and I'll add it.

s344
00:25:44.187 --> 00:25:49.587
Currently, Claude goes directly to analyzing the data and like building the eval and showing it to you.

s345
00:25:49.587 --> 00:25:54.627
I do think it's important and helpful for there to be an intermediate stage where you look at the data first.

s346
00:25:54.627 --> 00:26:00.268
But I think having data annotated by Claude where Claude's giving you some ideas about what to look at is very important.

s347
00:26:00.268 --> 00:26:06.587
So kind of like Claude assisted evaluation and scanning of the data is like I think like I I think a nice marriage.

s348
00:26:06.587 --> 00:26:10.707
Um and then once that's done, you can kind of promote like kind of examples to your eval set.

s349
00:26:10.707 --> 00:26:17.028
So I think there is still quite a role um for the human in the loop there, but Claude can assist quite a bit.

s350
00:26:17.028 --> 00:26:19.427
Hill climb with human feedback.

s351
00:26:19.427 --> 00:26:29.347
Um so hill climbing only requires that you have an eval and you have some signal that Claude can use to judge good versus bad.

s352
00:26:29.347 --> 00:26:37.589
So if you've taken human feedback and you've rolled that into an eval signal, for example, then it should be perfectly feasible to hill climb on an evaluation structure that way.

s353
00:26:37.589 --> 00:26:39.349
Yeah, this is a good one.

s354
00:26:39.349 --> 00:26:42.469
So, for hill climbing, what's like the best surface to start with?

s355
00:26:42.469 --> 00:26:44.989
So, I kind of have that here as like the thing we edit.

s356
00:26:44.989 --> 00:26:48.669
Yeah, so so I've actually played with this quite a bit.

s357
00:26:48.669 --> 00:27:01.149
Text is nice, in particular if things like prompts and skills are very nice because it's a very it is a very clear place that we know sign that we know significantly alters and affects model behavior.

s358
00:27:01.149 --> 00:27:08.629
So, it's like a very constrained surface to modify, and we know it's highly coupled to the output, typically in terms of like what you're evaluating.

s359
00:27:08.629 --> 00:27:10.429
Uh like here's a good example.

s360
00:27:10.429 --> 00:27:16.989
Um like when I'm doing skill hill climbing, I'm testing something that is directly reading from the skill, so I know there's clear coupling.

s361
00:27:16.989 --> 00:27:21.029
Now, I'm going to contrast that to I've also done hill climbing with agent harnesses.

s362
00:27:21.029 --> 00:27:23.829
But, what happens there is it can be a bit more hard to interpret.

s363
00:27:23.829 --> 00:27:30.509
So, you're modifying the harness code, and so the the the the changes per round are much larger diffs.

s364
00:27:30.509 --> 00:27:33.909
And um there's just more surface area for confusion.

s365
00:27:33.909 --> 00:27:38.029
You can be making changes that are actually decoupled from the output signal you're measuring.

s366
00:27:38.029 --> 00:27:41.989
So, it's certainly possible, and indeed people have done quite a bit of work hill climbing on agent harnesses.

s367
00:27:41.989 --> 00:27:53.469
I just think it's a little bit trickier, and text-based services are often a good starting point, prompts, skills, um because they're so clearly coupled to model behavior, and they're easy to modify.

s368
00:27:53.469 --> 00:27:57.309
The catch actually is that is that things like prompts are very easy to overfit.

s369
00:27:57.309 --> 00:28:00.709
I've seen that in in many of my uh hill climbing efforts as well.

s370
00:28:00.709 --> 00:28:12.869
So, like um I don't know if I have a good diagram, but um you know, you have some eval distribution you're hill climbing on, even with these regularizations in place, like you you the eval test train split and so forth.

s371
00:28:12.869 --> 00:28:22.069
Um prompts are very easy to like wiggle Cloud's very good at wiggling the prompt into a shape that does well in your eval that isn't really representative of what's in production.

s372
00:28:22.069 --> 00:28:25.869
So, you have to be very careful with prompts, but they are very easy to to modify.

s373
00:28:25.869 --> 00:28:29.709
So, it's a good starting point, you do need to be careful with overfitting.

s374
00:28:29.709 --> 00:28:31.389
Uh let's see.

s375
00:28:31.389 --> 00:28:32.829
Turning a prompt into a skill.

s376
00:28:32.829 --> 00:28:41.550
So, they're kind of compliment like um maybe do you mean if you if you're like going through some standard operating procedure, when to like promote that to a skill?

s377
00:28:41.550 --> 00:28:42.790
Okay, yeah.

s378
00:28:42.790 --> 00:28:45.390
Um that's actually kind of interesting.

s379
00:28:45.390 --> 00:28:48.390
I think often I and Claude is quite good at judging that.

s380
00:28:48.390 --> 00:28:53.390
So, basically you can basically go through a workflow and then you can ask Claude, okay, turn that into a skill.

s381
00:28:53.390 --> 00:28:54.390
I do that all the time.

s382
00:28:54.390 --> 00:28:58.670
So, Claude is pretty good at like catching, okay, you're doing some repetitive thing, make it a skill.

s383
00:28:58.670 --> 00:29:01.590
Uh I actually think you should just have Claude do that.

s384
00:29:01.590 --> 00:29:04.750
Um just tell it, turn this into a skill, and it'll do it.

s385
00:29:04.750 --> 00:29:18.950
So, usually in the in the case of evals, the setup will be I'll have an existing skill that I've kind of defined and maybe it has some boilerplate starting text and I'll say, okay, I want you to make this skill improved.

s386
00:29:18.950 --> 00:29:23.270
I want to improve this skill text to basically perform better on this eval.

s387
00:29:23.270 --> 00:29:24.670
So, that's usually the way I set it up.

s388
00:29:24.670 --> 00:29:28.030
Like I have a skill boilerplate as a starting point.

s389
00:29:28.030 --> 00:29:30.350
I know I want this thing to be a skill.

s390
00:29:30.350 --> 00:29:31.910
Like in this in this case, right?

s391
00:29:31.910 --> 00:29:34.950
Like I had this skill for the Claude API.

s392
00:29:34.950 --> 00:29:37.990
So, and it was basically you can imagine it's basically distillation of our docs.

s393
00:29:37.990 --> 00:29:42.510
It's like go to this page if you want to learn about this, go to this page you want to learn about this.

s394
00:29:42.510 --> 00:29:47.590
Um it had a bunch of little gotchas and issues, and so my eval performance was pretty low to start.

s395
00:29:47.590 --> 00:29:51.910
But I had a starting point, and that's what Claude was able to then modify.

s396
00:29:51.910 --> 00:29:56.150
So, I think usually with skills you're kind of starting with some baseline.

s397
00:29:56.150 --> 00:30:02.190
And it could be like a very crude default, but then give Claude something to like start from.

s398
00:30:02.190 --> 00:30:03.590
That's usually the way I would do it.

s399
00:30:03.590 --> 00:30:04.070
Right.

s400
00:30:04.070 --> 00:30:11.590
So, verification in um domains that that have kind of open-ended criteria.

s401
00:30:11.590 --> 00:30:15.590
So, like yeah, I think in general heuristic evaluators are preferable, of course.

s402
00:30:15.590 --> 00:30:22.270
So, if you can have define like a a a deterministic check for good or bad, that's obviously preferable.

s403
00:30:22.270 --> 00:30:23.950
So, in the maze that doesn't work.

s404
00:30:23.950 --> 00:30:33.790
So, what I found is um well-structured rubrics that produce similar uh grades.

s405
00:30:33.790 --> 00:30:36.350
So, okay, a few different things.

s406
00:30:36.350 --> 00:30:41.552
Build a rubric that you as a human can reason about first.

s407
00:30:41.552 --> 00:30:45.432
So, it's like let's say we're grading a piece of like text.

s408
00:30:45.432 --> 00:30:47.752
Is this good or bad?

s409
00:30:47.752 --> 00:30:52.312
Build a rubric that like I could look at as a human to like look at some text and say, "Okay, I can grade that."

s410
00:30:52.312 --> 00:30:53.272
That's point one.

s411
00:30:53.272 --> 00:30:56.752
Point two, don't use arbitrary scales.

s412
00:30:56.752 --> 00:31:00.032
I see this all the time like grade it from zero to five.

s413
00:31:00.032 --> 00:31:01.072
It's very ambiguous.

s414
00:31:01.072 --> 00:31:06.351
Make it crisp and simple like ideally a binary classification good versus bad.

s415
00:31:06.351 --> 00:31:15.672
You if you want to evaluate multiple things, you can set up like multiple binary classifiers rather than the worst thing is give Claude like from one to 10 rate this.

s416
00:31:15.672 --> 00:31:17.351
That's extremely noisy.

s417
00:31:17.351 --> 00:31:19.912
Another sanity check, test across different models.

s418
00:31:19.912 --> 00:31:22.952
Have Sonnet, Opus, and Fable all grade using your rubric.

s419
00:31:22.952 --> 00:31:25.592
If they all disagree, that's like a bad sign.

s420
00:31:25.592 --> 00:31:28.072
Another test, run replicates.

s421
00:31:28.072 --> 00:31:35.351
So, basically um like I show here how you want low variance, run replicates with your rubric and make sure it's tight.

s422
00:31:35.351 --> 00:31:42.712
So, if you do all those things, you can definitely make it work pretty well um for yeah, like text-based answers that are squishy.

s423
00:31:42.712 --> 00:31:47.192
Like there is no very definitive like way to heuristically check that.

s424
00:31:47.192 --> 00:31:51.912
Uh but LLM as judge with a good well-structured rubric can be quite effective.

s425
00:31:51.912 --> 00:31:53.152
Yeah, that's actually a good one.

s426
00:31:53.152 --> 00:31:55.112
So, actually I I I I should have mentioned that.

s427
00:31:55.112 --> 00:31:57.952
Uh pairwise comparisons can actually be quite good.

s428
00:31:57.952 --> 00:32:03.471
It's much easier for humans even models to kind of compare two artifacts than it is to grade an artifact.

s429
00:32:03.471 --> 00:32:06.832
Like you can typically answer, "Oh, I like this webpage better than this one."

s430
00:32:06.832 --> 00:32:10.112
versus "I'm sure this webpage grade it between zero and five."

s431
00:32:10.112 --> 00:32:11.072
It's much harder.

s432
00:32:11.072 --> 00:32:20.912
So, pairwise comparisons are pretty effective and actually I think that is a good that is a good way uh if if your problem allows for that, that actually can be a effective way to deal

s433
00:32:20.912 --> 00:32:23.992
with um more ambiguous gradings.

s434
00:32:23.992 --> 00:32:27.632
Well, I think it goes a bit back to just looking at the data carefully.

s435
00:32:27.632 --> 00:32:30.512
Kind of relate to the question asked over here.

s436
00:32:30.512 --> 00:32:42.113
Um when I say the distribution, um I more mean ensuring that um you've sampled some set of cases for your eval set.

s437
00:32:42.113 --> 00:32:45.113
And making sure those cases represent what you actually care about in production.

s438
00:32:45.113 --> 00:32:47.193
That's really what I mean by distribution.

s439
00:32:47.193 --> 00:32:49.833
And so, there are some interesting tricks here.

s440
00:32:49.833 --> 00:32:58.433
Like one problem that happens is adversarial sampling, where you select like this is maybe a little bit confusing, but I'll try to explain it clearly.

s441
00:32:58.433 --> 00:33:04.713
You select cases that are specific failures from a from a particular model class.

s442
00:33:04.713 --> 00:33:06.513
So, models are jagged.

s443
00:33:06.513 --> 00:33:09.753
And so, you just look at, okay, like Sonnet 5 can't do these examples.

s444
00:33:09.753 --> 00:33:11.233
I promote them to my eval.

s445
00:33:11.233 --> 00:33:15.953
Opus 5 may maybe it'll solve them all just because intelligence is jagged.

s446
00:33:15.953 --> 00:33:20.793
And so, like a particular pocket of one model might just get solved by another.

s447
00:33:20.793 --> 00:33:33.073
So, typically, a good recommendation is promote eval cases promote cases in eval set that are hard or interesting from first principles, from your own intuition, from looking at production data, not necessarily

s448
00:33:33.073 --> 00:33:35.473
completely sampled from the failures of any given model.

s449
00:33:35.473 --> 00:33:44.513
Cuz that can result in kind of this odd distribution that's shaped to the valleys of one particular like jagged model's intelligence that don't apply when you upgrade models.

s450
00:33:44.513 --> 00:33:47.953
So, like can you use bucket-based evals or do you have to use binary classification?

s451
00:33:47.953 --> 00:33:50.033
I'm being a little bit overly prescriptive.

s452
00:33:50.033 --> 00:33:55.033
You obviously can, but just as a general rule, I see people going too far in that direction.

s453
00:33:55.033 --> 00:34:01.633
So, there'll be some very arbitrary scale between 0 and 10 with unclear grading and rules that even a very small model can't resolve.

s454
00:34:01.633 --> 00:34:08.273
So, you obviously, if you have a well thought through scale, you obviously can do that as long as it satisfies we talked about.

s455
00:34:08.273 --> 00:34:15.753
It has like low variance, it's reproduced across models, a human grader can kind of distinguish between these two grades, that's totally fine.

s456
00:34:15.753 --> 00:34:20.033
I'm just saying binary is like a shortcut of like if to simplify the problem, usually.

s457
00:34:20.033 --> 00:34:22.873
And also for grading, you often want to use smaller, cheaper models.

s458
00:34:22.873 --> 00:34:27.073
So, you really don't want Fable doing your grading and depending on your cost sensitivity.

s459
00:34:27.073 --> 00:34:28.913
So, um, those are just the recommendations.

s460
00:34:28.913 --> 00:34:31.033
But obviously, if it's well thought out, that's totally fine.

s461
00:34:31.033 --> 00:34:37.593
You can think about using, um, yeah, like a multi uh, like a multi multi-category classification.

s462
00:34:37.593 --> 00:34:38.713
Yeah, that's a good point.

s463
00:34:38.713 --> 00:34:42.234
Like when when to build evals for what types of problems.

s464
00:34:42.234 --> 00:34:49.754
Like, is it worth building it for my like personal like use cases or we only doing it for things that are kind of production grade?

s465
00:34:49.754 --> 00:34:56.234
Um I think it depends on um I let's see.

s466
00:34:56.234 --> 00:34:58.834
It probably depends a bit on the importance of the task.

s467
00:34:58.834 --> 00:35:03.834
So, if it's even it's a personal task, you feel that it's quite important, then it would be perfectly reasonable to need eval for it.

s468
00:35:03.834 --> 00:35:04.674
I think that's totally fine.

s469
00:35:04.674 --> 00:35:06.114
I think people do that.

s470
00:35:06.114 --> 00:35:07.554
Kind of varies by person.

s471
00:35:07.554 --> 00:35:21.554
I personally haven't necessarily built many personal evals for like my personal automations, but um I think if it's a task that's highly important, it's in particular um I okay, I'll give a good example.

s472
00:35:21.554 --> 00:35:28.794
Like if say people work in finance and like you've automated parts of your your personal workflow, but quality is extremely important.

s473
00:35:28.794 --> 00:35:32.714
Like, this is a work output that's going to share externally or something, that would absolutely need evals.

s474
00:35:32.714 --> 00:35:34.674
Even though it's something you've built personally.

s475
00:35:34.674 --> 00:35:42.874
Um if it's my personal assistant, that's like, you know, it's like a a meeting it's like an app that I built to help me manage my calendar.

s476
00:35:42.874 --> 00:35:44.034
Uh I may or may not build evals.

s477
00:35:44.034 --> 00:35:45.634
It depends upon the context.

s478
00:35:45.634 --> 00:35:54.514
But I do think obviously if it's something that's important for your work, it's like, you know, producing deliverables that are, you know, externally viewed or doing something very sensitive task, you obviously want evals.

s479
00:35:54.514 --> 00:35:57.194
Even if it's a personal agent.

s480
00:35:57.194 --> 00:36:03.394
I actually worked at LangChain before Anthropic, but I I I I can't advocate for any one or the other.

s481
00:36:03.394 --> 00:36:06.434
Um Yeah, I I I plead the fifth on that.

s482
00:36:06.434 --> 00:36:09.514
Um I mean, look, I've I've used many of them.

s483
00:36:09.514 --> 00:36:14.634
Um I think actually if you're if you're working at relatively small scale, you obviously could roll your own.

s484
00:36:14.634 --> 00:36:16.714
Like, you don't necessarily need a fancy observability system.

s485
00:36:16.714 --> 00:36:24.994
In fact, we're going to be adding to the skill the ability to to um to to uh just like a simple data viewer or we're actually adding to the skill.

s486
00:36:24.994 --> 00:36:30.194
So, Claude can kind of Claude's very good at kind of building you UIs and visualization surfaces.

s487
00:36:30.194 --> 00:36:37.594
So, they can kind of roll If you have like a very bespoke problem you want to look at the data for, it can kind of roll you whatever app you want very easily.

s488
00:36:37.594 --> 00:36:39.834
And and not very expensively, either.

s489
00:36:39.834 --> 00:36:40.994
No, this is a key point.

s490
00:36:40.994 --> 00:36:42.754
It's not guaranteed to converge at all.

s491
00:36:42.754 --> 00:36:45.516
So, I've seen many cases in which the hill climber gets stuck.

s492
00:36:45.516 --> 00:36:57.196
And actually, we built You can read the blog, but we built a fallback in the hill climber where if it gets stuck for multiple rounds, it'll actually go and basically try to cluster the the remaining failure cases, and it'll try to look for.

s493
00:36:57.196 --> 00:36:59.276
So, for example, here's the problem.

s494
00:36:59.276 --> 00:37:07.396
Oftentimes, you'll run the hill climber without having run like a build eval command or with without having built a very like carefully constructed eval set.

s495
00:37:07.396 --> 00:37:14.156
So, the hill climber will go and look at your evals and try to find are these cases ambiguous like we talked about here.

s496
00:37:14.156 --> 00:37:21.396
Maybe that last 20% are all faulty grader examples, ambiguous ambiguous instructions.

s497
00:37:21.396 --> 00:37:22.396
This happens all the time.

s498
00:37:22.396 --> 00:37:23.956
This is like very common.

s499
00:37:23.956 --> 00:37:28.276
So, climber is just stuck not because Claude can't do better, it's because the eval is poorly structured.

s500
00:37:28.276 --> 00:37:31.116
So, the hill climbing skill will try to find that for you.

s501
00:37:31.116 --> 00:37:32.396
Uh so, we added that.

s502
00:37:32.396 --> 00:37:43.956
In fact, when I did this hill climbing I talked about here, um this second stage actually went it kind of got stuck, and it went and basically clustered the eval cases and looked at, "Okay, why the failure cases?

s503
00:37:43.956 --> 00:37:44.756
Why are they failing?

s504
00:37:44.756 --> 00:37:45.476
What's going on?"

s505
00:37:45.476 --> 00:37:47.076
It found a bunch of ambiguous cases.

s506
00:37:47.076 --> 00:37:47.876
It rewrote them.

s507
00:37:47.876 --> 00:37:51.996
And And anyway, so it'll actually do that for you, but that happens all the time.

s508
00:37:51.996 --> 00:37:54.316
It's absolutely not guaranteed to converge.

s509
00:37:54.316 --> 00:37:58.516
And usually it's because of failures in your eval or like kind of misconfigurations in eval.

s510
00:37:58.516 --> 00:38:00.156
Uh so, that that's really interesting.

s511
00:38:00.156 --> 00:38:05.596
I You have a lot of respect for the DSPy team and and Omar and and and and and um and colleagues.

s512
00:38:05.596 --> 00:38:09.716
Uh this this actually just is a skill built like bundled with Claude code.

s513
00:38:09.716 --> 00:38:11.396
The skill is also open source.

s514
00:38:11.396 --> 00:38:15.276
So, you could consider taking that skill and and like vendoring it in any other surface.

s515
00:38:15.276 --> 00:38:18.916
See if you want to could probably work with DSPy.

s516
00:38:18.916 --> 00:38:26.196
It's just some very simple general instructions for setting up a hill climbing loop following basically these principles.

s517
00:38:26.196 --> 00:38:32.236
Um so, um that [clears throat] it could be compatible with DSPy.

s518
00:38:32.236 --> 00:38:40.516
I haven't used DSPy, so I don't know like the details and how it's structured, but um it's possible this skill guidance here could be compatible with it.

s519
00:38:40.516 --> 00:38:46.637
If you want to like effectively implement this or run it in DSPy, the skill is open source.

s520
00:38:46.637 --> 00:38:49.357
Uh you said choosing effort level.

s521
00:38:49.357 --> 00:38:50.957
Yeah, this is a classic one.

s522
00:38:50.957 --> 00:38:52.677
Effort's confusing, isn't it?

s523
00:38:52.677 --> 00:38:53.597
Yeah.

s524
00:38:53.597 --> 00:38:55.437
I'm also confused by effort.

s525
00:38:55.437 --> 00:38:57.277
It's Yeah.

s526
00:38:57.277 --> 00:38:59.837
It it is it's non-obvious.

s527
00:38:59.837 --> 00:39:13.357
Um so um this is a so what's nice is if you have an eval uh the hill climber can absolutely try to configure effort for you.

s528
00:39:13.357 --> 00:39:14.037
Now, I get it.

s529
00:39:14.037 --> 00:39:16.477
It's like not that easy to have an eval.

s530
00:39:16.477 --> 00:39:22.877
So, like it's not a it's not applicable to every problem, but if you happen to have an eval then this hill climber can search through your model effort configurations

s531
00:39:22.877 --> 00:39:24.597
and it follows a very simple principle.

s532
00:39:24.597 --> 00:39:36.077
The principle is just simply that typically the stronger model at lower effort is is like more effective uh or it sorry is more token efficient and therefore cheaper than like a weaker model at high effort.

s533
00:39:36.077 --> 00:39:37.237
That's like the general principle.

s534
00:39:37.237 --> 00:39:38.997
That actually turns out to be true a lot.

s535
00:39:38.997 --> 00:39:47.837
So, it'll kind of search through the different models at low effort first and basically continue that search until you see a degradation performance below the baseline.

s536
00:39:47.837 --> 00:39:54.797
So, for here I don't show it, but like when I tried Haiku for this was 4.5. Haiku we now have 5.5, which is quite a bit better.

s537
00:39:54.797 --> 00:39:59.397
Uh 4.5 at low effort was a significant regression, so we admitted that that data point.

s538
00:39:59.397 --> 00:40:05.637
But basically if you have an eval hill climbing is a very good way to try search for a good model effort combination.

s539
00:40:05.637 --> 00:40:18.717
More general effort guidance if you don't have an eval is obviously tricky and subtle and like I think um people typically um I I will say the guidance I recommend generally on effort is try stronger model low effort

s540
00:40:18.717 --> 00:40:19.877
in general.

s541
00:40:19.877 --> 00:40:22.237
Opus 5.5 low effort.

s542
00:40:22.237 --> 00:40:23.157
Try that.

s543
00:40:23.157 --> 00:40:24.077
Basically.

s544
00:40:24.077 --> 00:40:29.957
Uh better models low effort are often more token efficient and therefore cheaper than than weaker models at high effort.

s545
00:40:29.957 --> 00:40:32.357
That's like the key point I'll I'll leave you with.

s546
00:40:32.357 --> 00:40:33.557
So, not medium?

s547
00:40:33.557 --> 00:40:36.437
Uh try low to start.

s548
00:40:36.437 --> 00:40:38.757
Um you can always dial back up.

s549
00:40:38.757 --> 00:40:43.477
But basically I like very smart model not trying too hard.

s550
00:40:43.477 --> 00:40:50.598
That's a very that's often a sweet spot for for um kind of kind of cost performance.

s551
00:40:50.598 --> 00:40:55.998
Obviously, you can dial up effort, but I think the the counter is people often start with something like max effort.

s552
00:40:55.998 --> 00:40:59.838
This extremely token and like token hungry.

s553
00:40:59.838 --> 00:41:07.318
So, it is actually wise often start like small smarter model low effort and kind of kind of go from there if you need it.

s554
00:41:07.318 --> 00:41:12.918
Yeah, actually, we talked about this in the blog here.

s555
00:41:12.918 --> 00:41:14.678
I was open that up.

s556
00:41:14.678 --> 00:41:15.518
I don't have internet.

s557
00:41:15.518 --> 00:41:16.438
That's fine.

s558
00:41:16.438 --> 00:41:18.278
I'll just tell you then.

s559
00:41:18.278 --> 00:41:21.198
Yeah, so it's basically telling Claude how hard to think.

s560
00:41:21.198 --> 00:41:27.478
So, low effort saying like don't think as hard as you scale it up it's saying think harder.

s561
00:41:27.478 --> 00:41:40.838
So, and that's kind of why more fundamentally intelligent model at low effort can actually be more token efficient and therefore cheaper than a cheaper model in terms of base tokens at high effort cuz that

s562
00:41:40.838 --> 00:41:51.038
that for example, Sonic 5 at high effort is spending lots and lots of tokens thinking where smarter model low effort will spend fewer tokens thinking at a higher baseline of intelligence.

s563
00:41:51.038 --> 00:41:52.718
That's kind of the intuition.

s564
00:41:52.718 --> 00:41:55.638
Yeah, so the question is about setting budgets in hill climbing.

s565
00:41:55.638 --> 00:41:59.878
So, basically when you when you run this skill and set this up, it'll ask you for a budget.

s566
00:41:59.878 --> 00:42:03.398
So, um it will manage that for you.

s567
00:42:03.398 --> 00:42:06.358
It'll track the budget you allocate for the hill climbing round.

s568
00:42:06.358 --> 00:42:07.758
That can definitely happen though.

s569
00:42:07.758 --> 00:42:10.158
And that's actually why this is a classic example.

s570
00:42:10.158 --> 00:42:12.558
That's why designing a good evaluation is really important.

s571
00:42:12.558 --> 00:42:19.318
If you have if you have a poorly specified evaluation, imagine for example, you have an eval where that final 20% is impossible.

s572
00:42:19.318 --> 00:42:25.798
You burn a huge number of tokens trying to solve those final you know, 20% of problems that are effectively not solvable.

s573
00:42:25.798 --> 00:42:32.078
Kind of an eval building, the question is about kind of human loop kind of human Claude collaboration.

s574
00:42:32.078 --> 00:42:41.958
So, I think I think that um humans still have the role to kind of steer the goal.

s575
00:42:41.958 --> 00:42:44.478
Like what are we actually trying to evaluate and why is it important?

s576
00:42:44.478 --> 00:42:45.878
So, it's kind of taste is important.

s577
00:42:45.878 --> 00:42:48.158
So, like what are you actually trying to see answer?

s578
00:42:48.158 --> 00:42:57.880
For example, I'll give you I'll take an example from like Hamel's public kind of ant or a public dog fooding of our um build eval skill.

s579
00:42:57.880 --> 00:43:00.000
So, it's like okay, it's a customer support app.

s580
00:43:00.000 --> 00:43:01.440
So, let's say that's your application.

s581
00:43:01.440 --> 00:43:02.000
What matters?

s582
00:43:02.000 --> 00:43:02.600
What's important?

s583
00:43:02.600 --> 00:43:03.320
What do you care about?

s584
00:43:03.320 --> 00:43:05.760
Those are things that a human can guide Claude.

s585
00:43:05.760 --> 00:43:18.400
Now, once those things are clearly specified, Claude um can look at like raw traces and determine effectively like okay, where is the app likely falling short of my goals?

s586
00:43:18.400 --> 00:43:22.240
So, that's I think kind of nice collaboration you can have in place.

s587
00:43:22.240 --> 00:43:25.640
And and and that's actually when like kind of looking at the data is very helpful.

s588
00:43:25.640 --> 00:43:28.600
So, Claude can kind of help to steer you okay, I've looked at the data.

s589
00:43:28.600 --> 00:43:30.160
Here's a bunch of cases that look problematic.

s590
00:43:30.160 --> 00:43:32.720
And that's that's just like having an online evaluator.

s591
00:43:32.720 --> 00:43:36.000
So, you like imagine you have a tracing system, you have an online evaluator.

s592
00:43:36.000 --> 00:43:39.880
It has some like grading for like customer satisfaction.

s593
00:43:39.880 --> 00:43:42.640
It's flagging like here's problematic traces for you.

s594
00:43:42.640 --> 00:43:43.440
The same kind of thing.

s595
00:43:43.440 --> 00:43:48.000
Like you're telling Claude, you're telling online evaluator, here's what I care about, here's what I want you check.

s596
00:43:48.000 --> 00:43:51.800
It can look at the data, it can elevate a bunch of cases that look like these look problematic.

s597
00:43:51.800 --> 00:43:57.880
Then given those cases, you can like review them, understand them, and then say okay, yes, build me an eval based upon these.

s598
00:43:57.880 --> 00:44:03.120
That's typically a very nice marriage between like Claude and human when it comes to building evals.

s599
00:44:03.120 --> 00:44:05.360
Yeah, question is like who's responsible for building evals?

s600
00:44:05.360 --> 00:44:10.680
I mean, it depends on the problem, it depends upon I mean again throw back at everyone ever everyone is empowered to build evals.

s601
00:44:10.680 --> 00:44:13.000
So, it's not really delineated per team.

s602
00:44:13.000 --> 00:44:24.040
It's typically based on ownership like of the product and um I think in particular when you have skills like this, it make it easier to build evals to help climb, it really opens up the number of people that can

s603
00:44:24.040 --> 00:44:27.760
feasibly kind of build and design evals, which is actually quite nice.

s604
00:44:27.760 --> 00:44:33.000
So, we've seen like Anthropic many many many different types of roles build evaluations.

s605
00:44:33.000 --> 00:44:36.920
Um because Claude's so involved and then has can help so much.

s606
00:44:36.920 --> 00:44:42.960
Um much of the tedium of like um you know, building an eval runner and stuff, that's like Claude can handle all that.

s607
00:44:42.960 --> 00:44:47.120
So, really it's about what we're talking about before, the human taste guiding the process.

s608
00:44:47.120 --> 00:44:47.800
What do I care about?

s609
00:44:47.800 --> 00:44:53.840
And that's very that many different types of roles can can kind of serve that guidance or provide that steering.

s610
00:44:53.840 --> 00:44:56.841
Um let's see if I have a good one.

s611
00:44:56.841 --> 00:45:00.681
Uh Right.

s612
00:45:00.681 --> 00:45:04.521
Um Uh, off top of my head.

s613
00:45:04.521 --> 00:45:09.961
Um I've seen Okay, I have a good one.

s614
00:45:09.961 --> 00:45:14.881
I have seen um evals for generative UIs.

s615
00:45:14.881 --> 00:45:18.241
Um, so that's kind of a good example.

s616
00:45:18.241 --> 00:45:23.041
Uh, actually I think in that case it was a pairwise judge back to the point that you were making.

s617
00:45:23.041 --> 00:45:34.601
Like so it's basically comparing like configuration A B, um what is better um between like two different generative UIs and like the greater would would do kind of pairwise comparison.

s618
00:45:34.601 --> 00:45:39.921
I think then those results were rolled across many trials and scored with something like an Elo score.

s619
00:45:39.921 --> 00:45:45.601
So that's an example of like generative UI with a pairwise evaluator I I've seen before.

s620
00:45:45.601 --> 00:45:48.561
There's many other examples I can't think of at the top of my head.

s621
00:45:48.561 --> 00:45:51.121
Um, but people use LM as judge all the time.

s622
00:45:51.121 --> 00:45:55.001
Um, for like retrieval tasks or like writing tasks.

s623
00:45:55.001 --> 00:46:00.881
Um Uh Look, categorizing traces, looking at traces and saying oh this is bad or good.

s624
00:46:00.881 --> 00:46:03.081
So like it's extremely common.

s625
00:46:03.081 --> 00:46:03.841
Yeah.

s626
00:46:03.841 --> 00:46:11.241
So okay, that a lot of the motivation for the hill climbing work is kind of it is kind of flowing from auto research.

s627
00:46:11.241 --> 00:46:16.361
Auto research is motivated heavily by the fact that models are getting very very good.

s628
00:46:16.361 --> 00:46:17.601
They're very good at self-correction.

s629
00:46:17.601 --> 00:46:21.881
If you give them a well-structured problem uh they can hill climb on.

s630
00:46:21.881 --> 00:46:26.041
So I think Karpathy's case is a good example of I think I can't remember the precise problem.

s631
00:46:26.041 --> 00:46:28.921
I think that general setup work with many different formulations.

s632
00:46:28.921 --> 00:46:32.721
Like it could be um you know, like drive down loss or something.

s633
00:46:32.721 --> 00:46:39.441
Um, so again there's like a very clear verifiable objective and clock can look at the objective and use that as a guide to hill climb.

s634
00:46:39.441 --> 00:46:49.361
Um, I have Yeah, I didn't show it here but I have some other examples doing auto research um in this kind of hill climbing process and it would it would this kind of

s635
00:46:49.361 --> 00:46:56.681
um This command in particular is actually designed more for eval-based hill climbing where you have like an eval set with a with a split.

s636
00:46:56.681 --> 00:47:08.083
Auto research was looking more like you build an artifact and like grade it, like look at the loss or something, but the the setup could probably very be very easily kind of modified to work with auto research with the very same intuition.

s637
00:47:08.083 --> 00:47:24.043
The point is that models are good at autonomous work over long horizons when given a clear um a clear objective or verifiable that that they can reward that they can kind of measure and use to inform what their next move is, they're very very good at autonomously solving problems.

s638
00:47:24.043 --> 00:47:28.003
So, it's driven by the same intuition as auto research.

s639
00:47:28.003 --> 00:47:31.483
hill climb and then extend into auto research.

s640
00:47:31.483 --> 00:47:37.243
Uh the the methodology for hill climb would So, it's just a it's just a question of a problem formulation.

s641
00:47:37.243 --> 00:47:38.362
This just to be clear.

s642
00:47:38.362 --> 00:47:53.083
So, like the auto research from formulation was um was initially focused more on like on like designing, for example, an a like a network with, for example, um solving some problem where you have like a a verifiable objective like loss.

s643
00:47:53.083 --> 00:47:54.003
That's really it.

s644
00:47:54.003 --> 00:47:58.043
The the hill climbing process though was like very similar where like Claude runs some experiments.

s645
00:47:58.043 --> 00:47:58.843
It measures loss.

s646
00:47:58.843 --> 00:48:01.803
It looks at the results and then adapts, does something else.

s647
00:48:01.803 --> 00:48:07.523
So, it's the same kind of hill climbing loop, but like in that particular case it's just applied to a slightly different domain.

s648
00:48:07.523 --> 00:48:16.323
So, it's like you could probably take this skill and basically say, "Okay, set this up for if you had like like a the nano GPT repo, you could use this skill with it, I'm sure."

s649
00:48:16.323 --> 00:48:18.243
It would Claude would figure out how to adapt things.

s650
00:48:18.243 --> 00:48:20.483
Just If you're interested in that, just just like try it.

s651
00:48:20.483 --> 00:48:22.523
But the the core process is the same.

s652
00:48:22.523 --> 00:48:30.323
The core intuition is also the same that like models are very very effective at using um at self-correction and hill climbing.

s653
00:48:30.323 --> 00:48:32.283
That's the key intuition.

s654
00:48:32.283 --> 00:48:33.803
Yep.

s655
00:48:33.803 --> 00:48:39.922
Yeah, I kind of try to do that in like this example where I'm looking at cost and um performance.

s656
00:48:39.922 --> 00:48:42.483
So, you're like kind of jointly optimizing across those two.

s657
00:48:42.483 --> 00:48:46.523
And yeah, you can just tell Claude the multi-objective.

s658
00:48:46.523 --> 00:48:51.803
And usually it will just kind of measure multiple objectives and and take that into account.

s659
00:48:51.803 --> 00:48:53.323
Um that's certainly feasible.

s660
00:48:53.323 --> 00:48:55.203
And this is a good example of that.

s661
00:48:55.203 --> 00:48:58.162
Have you checked like what's like the max you like

s662
00:48:58.162 --> 00:49:01.084
Uh I mean, yeah, I haven't pushed that far.

s663
00:49:01.084 --> 00:49:04.244
I think but I've typically looked at as cost and performance.

s664
00:49:04.244 --> 00:49:05.764
And like speed's another dimension.

s665
00:49:05.764 --> 00:49:07.044
So, you like cost performance speed.

s666
00:49:07.044 --> 00:49:11.284
That feels like a pretty reasonable like trifecta of things that you would like kind of jointly measure.

s667
00:49:11.284 --> 00:49:21.804
Um, yeah, I mean, I think that um, as you kind of scale up, it just hobbles the process of hill climbing a lot cuz there's lots of different objectives that Claus is trying to balance when making these changes.

s668
00:49:21.804 --> 00:49:26.244
So, I think it'll just it'll make the problem harder, but like it is doable.

s669
00:49:26.244 --> 00:49:29.324
Maybe you could think about kind of changing one thing at once.

s670
00:49:29.324 --> 00:49:31.524
Doing doing them serially.

s671
00:49:31.524 --> 00:49:33.564
Yeah, that's it that's a very good question.

s672
00:49:33.564 --> 00:49:41.244
Uh, so the prompt on this stuff, which actually of all the things I talked about is the simplest thing to try and often is like extremely it's like do the simple thing first.

s673
00:49:41.244 --> 00:49:43.684
It's very simple and often very effective.

s674
00:49:43.684 --> 00:49:54.124
Uh, this general the general guidance in the skill applies to um, the the model delineation's a little bit on like it's not precise.

s675
00:49:54.124 --> 00:50:00.484
These are well-known anti-patterns that definitively frontier models like the 55 series don't need anymore.

s676
00:50:00.484 --> 00:50:04.444
How far back do you need to go until these patterns are still like appropriate?

s677
00:50:04.444 --> 00:50:05.484
It's like not quite clear.

s678
00:50:05.484 --> 00:50:06.764
I've tried to do some testing.

s679
00:50:06.764 --> 00:50:13.284
It's maybe like Opus like 45 is the point at which these things start to like become less relevant.

s680
00:50:13.284 --> 00:50:14.084
So, it's a little bit squishy.

s681
00:50:14.084 --> 00:50:22.124
I don't know like precisely the model at which like you need to remove these, but my view is basically when you're doing a migration to the latest to like

s682
00:50:22.124 --> 00:50:24.044
to like Opus 55, just do this.

s683
00:50:24.044 --> 00:50:28.564
Like definitively like the latest frontier models don't need these patterns.

s684
00:50:28.564 --> 00:50:30.124
Yeah, yeah, okay, good point.

s685
00:50:30.124 --> 00:50:33.404
So, the skill does have some model specific caveats.

s686
00:50:33.404 --> 00:50:34.964
So, the skill takes that into account a little bit.

s687
00:50:34.964 --> 00:50:37.364
If you're for example going to like Haiku, I get it.

s688
00:50:37.364 --> 00:50:40.764
Like maybe Haiku like um, isn't quite as autonomous.

s689
00:50:40.764 --> 00:50:42.524
Maybe it does need slightly more boosters.

s690
00:50:42.524 --> 00:50:43.284
I don't know if that's true.

s691
00:50:43.284 --> 00:50:47.164
I'm just kind of yeah, kind of kind of suggesting something.

s692
00:50:47.164 --> 00:50:49.084
Um, it it'll take that into account.

s693
00:50:49.084 --> 00:50:52.124
So, it does have some model specific guidance built in.

s694
00:50:52.124 --> 00:50:58.164
It's like smart to figure out like okay, here's the model you're working with and here's the anti-patterns that we know are are not needed anymore.

s695
00:50:58.164 --> 00:51:00.604
Um, but it's it's a good call out.

s696
00:51:00.604 --> 00:51:04.725
Yeah, uh, uh,

s697
00:51:04.725 --> 00:51:13.045
Yeah, um it it is not a Yeah, I I should I should get back to to like to be really clear about this.

s698
00:51:13.045 --> 00:51:16.485
Um it is not a hard cap.

s699
00:51:16.485 --> 00:51:22.085
It is basically telling Claude emphasizing how much thinking to do to solve the problem.

s700
00:51:22.085 --> 00:51:29.725
So, like for example, even if you're on very high effort, you give it a simple problem that won't just arbitrarily spend huge amounts of tokens verifying it.

s701
00:51:29.725 --> 00:51:36.805
There's some Claude has some thoughtfulness about like what problems require, um you know, all this extra thinking.

s702
00:51:36.805 --> 00:51:41.365
So, like um And and I don't know if I have any evals that show that.

s703
00:51:41.365 --> 00:51:42.765
Uh yeah, maybe not.

s704
00:51:42.765 --> 00:51:47.005
Um but it is suggested thinking.

s705
00:51:47.005 --> 00:51:50.725
So, that's the kind of way I think it's basically telling Claude how hard to think about a problem.

s706
00:51:50.725 --> 00:51:54.605
The problem comes up when you have like a harder problem, like a frontier coding problem.

s707
00:51:54.605 --> 00:52:05.805
That's where at for example hot extra high effort or max effort, you can see huge differences in token usages from like medium or low without much difference in actual outcome because Claude will do more work to verify the result for example.

s708
00:52:05.805 --> 00:52:13.605
So, I think out in the frontier regime of problems, that's where these effort differences can really result in like big big token spend differences.

s709
00:52:13.605 --> 00:52:15.725
And that's where we need to be more careful.

s710
00:52:15.725 --> 00:52:25.885
Yeah, I mean, this um this is So, if you kind of think about what's happening, um it's not dissimilar to to training.

s711
00:52:25.885 --> 00:52:34.845
And basically you're giving a model some some reward or objective, and it it is indeed trying to make modifications to the system to improve the objective.

s712
00:52:34.845 --> 00:52:37.445
Um so, yeah, philosophically that is similar.

s713
00:52:37.445 --> 00:52:44.205
And in fact, even where we're used, you know, we employ some simple principles like test train splits that are often employed like at training time.

s714
00:52:44.205 --> 00:52:47.405
But the nice thing is these are kind of run time modifications you can make.

s715
00:52:47.405 --> 00:52:56.125
So, this is basically a way to employ some of those principles, but basically like once a model has been deployed and like you're modifying services like prompts or skills,

s716
00:52:56.125 --> 00:53:02.525
um you can employ some of those same principles that we use in training, um but like in your particular domain.

s717
00:53:02.525 --> 00:53:04.565
Uh that's kind of the the big idea.

s718
00:53:04.565 --> 00:53:07.527
Yeah, so like is Claude aware this being evaled?

s719
00:53:07.527 --> 00:53:09.007
Um, right, exactly.

s720
00:53:09.007 --> 00:53:12.447
That I think that's something we've published on a fair amount.

s721
00:53:12.447 --> 00:53:22.887
That is a concern um, in the that that is certainly a concern kind of in in training uh, in different stages of training, absolutely.

s722
00:53:22.887 --> 00:53:32.847
What we're talking about here um, I I do think um, it could be somewhat of a concern.

s723
00:53:32.847 --> 00:53:36.687
I think the bigger concern than what we're talking about here is typically more overfitting.

s724
00:53:36.687 --> 00:53:40.287
So, Claude may know it's kind of being evaluated.

s725
00:53:40.287 --> 00:53:49.767
Um, but it will find ways to hill climb the objective based upon what's been provided in kind of this eval distribution that has.

s726
00:53:49.767 --> 00:54:00.367
The classic example being it'll find um, kind of prompt tricks and tuning that aren't applicable to production that indeed improve performance significantly on your eval.

s727
00:54:00.367 --> 00:54:02.927
So, I've seen that happen that happens all the time.

s728
00:54:02.927 --> 00:54:11.487
And it's um, it's not necessarily that Claude's like aware it's being evaled and saying like it's it's kind of some um, kind of unexpected behavior.

s729
00:54:11.487 --> 00:54:18.687
It's more that like okay, you've given it a prompt and Claude can find ways to shape this prompt to like make number go up and it'll just do that.

s730
00:54:18.687 --> 00:54:21.287
Um, that's why we use things like test train splits and try to regularize.

s731
00:54:21.287 --> 00:54:25.567
So, I think overfitting is more of the practical concern that we're going to think about here.

s732
00:54:25.567 --> 00:54:32.367
Um, but it is true that eval awareness is like a broader phenomenon that we we have discussed in in some in some of research blogs.

s733
00:54:32.367 --> 00:54:33.647
Yeah, that's a good point.

s734
00:54:33.647 --> 00:54:38.167
Um, kind of what's like the minimum kind of data set that's that's useful.

s735
00:54:38.167 --> 00:54:40.487
What is like the split between test and train?

s736
00:54:40.487 --> 00:54:42.407
Uh, I haven't looked at that in depth.

s737
00:54:42.407 --> 00:54:44.807
Um, frankly, just ask Claude.

s738
00:54:44.807 --> 00:54:54.367
Like um, when you set this up and you run this command, Claude has some heuristics to reason about the size of the eval set that's actually helpful given the signal or like what you're trying to measure.

s739
00:54:54.367 --> 00:54:56.967
Claude can kind of reason about like how many examples you should have.

s740
00:54:56.967 --> 00:54:59.367
And then there's some guidance in the skill for this.

s741
00:54:59.367 --> 00:55:03.607
So, it's something you can kind of uh, work with Claude to kind of con- tune.

s742
00:55:03.607 --> 00:55:13.728
But I haven't looked at like spoop or super specifically like uh, different like eval set sizes and like um I think the central concern I would have is more

s743
00:55:13.728 --> 00:55:18.088
um it is probably overfitting and um and like variance.

s744
00:55:18.088 --> 00:55:22.888
So, if you have a very very small number of samples, like okay, is that really representative of what you care about?

s745
00:55:22.888 --> 00:55:25.888
Um uh it's a lot easier to overfit.

s746
00:55:25.888 --> 00:55:30.368
So, like they may not be out of distribution for your production like what you actually care about.

s747
00:55:30.368 --> 00:55:32.408
Um they may be easier to overfit.

s748
00:55:32.408 --> 00:55:34.008
Uh they may be noisier.

s749
00:55:34.008 --> 00:55:40.608
Um so, anyway, like I think those are the problems that arise when you're in like the very very small sample regime.

s750
00:55:40.608 --> 00:55:41.208
Yeah, okay.

s751
00:55:41.208 --> 00:55:44.968
So, the question is about hill climbing on tasks that are like very very long duration.

s752
00:55:44.968 --> 00:55:52.968
So, kind of that's funny cuz that's actually why I mentioned I typically do like simple surfaces like prompts because they're like very very quick to change.

s753
00:55:52.968 --> 00:55:53.968
Uh you are right though.

s754
00:55:53.968 --> 00:55:59.408
There's some cases you're in a regime where like your changes could be very very long duration.

s755
00:55:59.408 --> 00:56:02.928
That effectively makes hill climbing much harder cuz like it's a much longer duration task then.

s756
00:56:02.928 --> 00:56:04.248
So, it's a lot longer time.

s757
00:56:04.248 --> 00:56:07.608
That's actually where that gets into our final point about kind of harness design.

s758
00:56:07.608 --> 00:56:15.928
If you're going to do that If you're doing like very very long duration hill climbing, you actually need harnesses that can like remain reliable for like many many hours to days.

s759
00:56:15.928 --> 00:56:17.928
And so, we have we have that obviously internally.

s760
00:56:17.928 --> 00:56:19.888
We use things like manage agents.

s761
00:56:19.888 --> 00:56:23.168
Um I think it just becomes a bigger infrastructure lift when you're doing that.

s762
00:56:23.168 --> 00:56:24.528
It can be done.

s763
00:56:24.528 --> 00:56:32.568
Um it obviously is harder because of what I just mentioned, infrastructure issues, failures, um and just like a lot longer time to see your signal.

s764
00:56:32.568 --> 00:56:35.008
Uh single turn versus multi-turn evals.

s765
00:56:35.008 --> 00:56:37.008
Um I mean, yeah.

s766
00:56:37.008 --> 00:56:40.688
When you're building agents, you do typically need multi-turn evals.

s767
00:56:40.688 --> 00:56:43.248
Um Claude can help you do that.

s768
00:56:43.248 --> 00:56:46.448
Particularly like the build eval command, it can look at traces.

s769
00:56:46.448 --> 00:56:54.448
It can If you give it a bunch of traces, it can help you promote um it can help you promote particular traces to your eval set.

s770
00:56:54.448 --> 00:56:58.768
Um in terms of grading, um I mean, yeah, there's a lot of different philosophies on this.

s771
00:56:58.768 --> 00:57:01.768
You can have evaluators looking at the entire trace trajectory.

s772
00:57:01.768 --> 00:57:04.648
You can have evaluators looking at like the final agent outcome.

s773
00:57:04.648 --> 00:57:05.848
Uh there's a lot of different philosophies.

s774
00:57:05.848 --> 00:57:07.568
The build eval command has a lot of guidance.

s775
00:57:07.568 --> 00:57:16.209
So, like if you have a particular like agent you're interested in, if you basically give this command access to the traces, it should be pretty pretty effective at kind of

s776
00:57:16.209 --> 00:57:18.369
shaping and designing the eval for you.

s777
00:57:18.369 --> 00:57:21.769
And if you can tell it I want multi-turn and here's what I'm considering.

s778
00:57:21.769 --> 00:57:23.249
Um cool.

s779
00:57:23.249 --> 00:57:23.729
Interesting.

s780
00:57:23.729 --> 00:57:29.449
So like for context engineering, what promotes of reliability a lot versus what's like overhyped as a technique?

s781
00:57:29.449 --> 00:57:33.369
Um Uh let's see.

s782
00:57:33.369 --> 00:57:41.129
Um So okay, the key thing I think and I've actually I've mentioned this in the past, we mentioned in the paper too.

s783
00:57:41.129 --> 00:57:45.369
I think the recursive language model work in RLM idea is quite good.

s784
00:57:45.369 --> 00:57:47.529
Um and we actually employ that idea here.

s785
00:57:47.529 --> 00:57:52.929
I think a very important thing in terms of context engineering reliability is decoupling brain and hands.

s786
00:57:52.929 --> 00:58:02.449
So basically designing your system such that it um the con- the key point is this, the context object is independent from the harness.

s787
00:58:02.449 --> 00:58:08.129
So the harness is basically your cloud can effectively query this context object that lives independently.

s788
00:58:08.129 --> 00:58:10.289
That's beneficial for all these different reasons.

s789
00:58:10.289 --> 00:58:13.049
One, the context is always recoverable.

s790
00:58:13.049 --> 00:58:20.169
So typically imagine alternatively you just have like, you know, a message history, you do a compaction event and everything you compacted is gone.

s791
00:58:20.169 --> 00:58:23.489
You can never recover it if you haven't like backed it up in the way like this.

s792
00:58:23.489 --> 00:58:27.569
That's why I actually like this architecture a lot for reliability in terms of context engineering.

s793
00:58:27.569 --> 00:58:29.849
It does exactly what the RLM work advocated.

s794
00:58:29.849 --> 00:58:36.849
Like it's basically splitting the context object from the model and the context object comes linked the model can interrogate through different mechanisms.

s795
00:58:36.849 --> 00:58:38.969
And that's exactly what we do here.

s796
00:58:38.969 --> 00:58:43.769
So interrogating that session log is basically done by the harness through different tools.

s797
00:58:43.769 --> 00:58:45.689
Um and that's a very nice architecture I think.

s798
00:58:45.689 --> 00:58:56.809
So I think that's like one of the tricks that I I found to be very nice for reliability, recoverability, um and in particular building like long duration agents, it's hard to know what will matter in the future.

s799
00:58:56.809 --> 00:59:04.849
So when you do a compaction, um you don't necessarily know if you need to like go back and interrogate that context that you're summarizing.

s800
00:59:04.849 --> 00:59:08.529
And when you have it backed up in an object like this, you can always do that.

s801
00:59:08.529 --> 00:59:11.529
So I think that's a trick that I I I a lot.

s802
00:59:11.529 --> 00:59:15.571
Uh and the RLM work was was very nice in that direction.

s803
00:59:15.571 --> 00:59:17.011
Yep.

s804
00:59:17.011 --> 00:59:17.731
Yeah.

s805
00:59:17.731 --> 00:59:26.771
Well, I think [clears throat] that's why, you know, we spend a lot of time on um, good eval design.

s806
00:59:26.771 --> 00:59:32.811
Um, yeah, these models, um, are non-deterministic.

s807
00:59:32.811 --> 00:59:39.691
Um, in many cases the operating regimes where outcomes are not easily verifiable, like not everything is a coding problem.

s808
00:59:39.691 --> 00:59:45.131
Many cases you want agents to do things that are more judgment-based, more free from open-ended outputs.

s809
00:59:45.131 --> 00:59:50.971
In those regimes, you have to be very smart about your eval design so that you're giving Claude an objective that I can climb.

s810
00:59:50.971 --> 01:00:03.091
And I think all the things we talked about in terms of like kind of verifier design, like good objectives, clear grading scales, multiple models can validate the result, low variance, all are like super important

s811
01:00:03.091 --> 01:00:06.731
when designing like the problem that you want Claude to go and solve.

s812
01:00:06.731 --> 01:00:20.411
So, I think, as kind of mentioned it in this talk, when it comes to unhambling frontier models, I think spending a lot of time up front in a very very good eval design is a way to make sure that you're getting the most out of the model because

s813
01:00:20.411 --> 01:00:25.451
if you said tell the model, "Okay, optimize performance on this like very poorly structured evaluation."

s814
01:00:25.451 --> 01:00:26.771
It won't get anywhere.

s815
01:00:26.771 --> 01:00:31.771
And it's not the model's fault because you haven't thought through the eval, it's because cases are impossible or ambiguous.

s816
01:00:31.771 --> 01:00:42.411
So, I think part of the message here is to just be thoughtful and spend a lot of time up front in the eval design such that you kind of set up frontier models for success and effectively unhamble them.

s817
01:00:42.411 --> 01:00:45.571
That was kind of the the namesake of the talk.

s818
01:00:45.571 --> 01:00:48.891
Maybe, um, I we're probably at time and everyone's probably tired.

s819
01:00:48.891 --> 01:00:51.611
I was at a wedding last night, so I'm very tired.

s820
01:00:51.611 --> 01:00:54.491
That was a funny timing, but it's very nice to be here, of course.

s821
01:00:54.491 --> 01:00:57.491
And actually, it's been uh, it's been a lot of fun, very good questions.

s822
01:00:57.491 --> 01:01:00.051
Um, maybe one more question over here.

s823
01:01:00.051 --> 01:01:00.971
Mm.

s824
01:01:00.971 --> 01:01:02.451
Yeah, so I think that's a good question.

s825
01:01:02.451 --> 01:01:08.731
It's about like, yeah, models are getting better all the time, so like, how do I modify my evals to keep up?

s826
01:01:08.731 --> 01:01:15.211
And I think simply it should be the case that as models get better, you should see performance on eval going up like we see in this diagram.

s827
01:01:15.211 --> 01:01:18.972
If you don't see that, then I would go back and question the eval was structured.

s828
01:01:18.972 --> 01:01:22.252
Um it is true like Yeah, we see this all the time.

s829
01:01:22.252 --> 01:01:24.532
Models are saturating eval very quickly.

s830
01:01:24.532 --> 01:01:29.972
So, we do need the development of new and harder evals that like keep up with the material.

s831
01:01:29.972 --> 01:01:31.532
And actually, that's a good point.

s832
01:01:31.532 --> 01:01:32.692
It's actually pretty hard.

s833
01:01:32.692 --> 01:01:39.532
It's hard to come up with evals that really challenge like the best models today, like the Fables Opus 5.5 even.

s834
01:01:39.532 --> 01:01:50.372
Um so, I do think it's a it's a it's a it's a recurring challenge um to design good evals that test the capabilities of the best models.

s835
01:01:50.372 --> 01:01:51.692
It's absolutely a challenge.

s836
01:01:51.692 --> 01:01:58.292
Um but I do think so, if you have evals that you built sometime ago, you should be seeing this trend on them.

s837
01:01:58.292 --> 01:02:01.492
So, as models get better, you should be seeing performance going up and up and up.

s838
01:02:01.492 --> 01:02:04.452
If you don't, I would question things about the eval itself.

s839
01:02:04.452 --> 01:02:06.692
That's kind of the key point I would make.

s840
01:02:06.692 --> 01:02:11.212
I mean, I've actually seen a lot of interesting work on this from the various trace and companies.

s841
01:02:11.212 --> 01:02:22.972
So, like the the the like workflow I've seen quite a bit of is like you have agents that are deployed, they're like producing traces, you have an online eval you're looking at those traces and scoring them.

s842
01:02:22.972 --> 01:02:30.292
Score Trace that score poorly are like then pulled into some offline eval set, you kind of then hill climb on that.

s843
01:02:30.292 --> 01:02:31.892
And then that loop continues.

s844
01:02:31.892 --> 01:02:34.492
That's a good That's like a very good pattern, I think.

s845
01:02:34.492 --> 01:02:35.812
It's very reasonable.

s846
01:02:35.812 --> 01:02:41.572
Uh and so, what's happening there is you're typically like hill climbing on these cases that you're failing on in production.

s847
01:02:41.572 --> 01:02:48.732
And then you're like, you know, putting those in regression tests once they all pass, and that like continues until in production you're seeing very few failures.

s848
01:02:48.732 --> 01:03:04.172
So, that actually is a good general way to structure your kind of eval loop and in many of the, you know, the LangChains, Rises, Braintrusts, all these companies kind of have product offerings that offer this kind of

s849
01:03:04.172 --> 01:03:08.852
this like outer loop, which does seem quite quite effective.

s850
01:03:08.852 --> 01:03:09.532
You do.

s851
01:03:09.532 --> 01:03:12.692
So, the challenge there becomes okay, well, what are your online evaluators?

s852
01:03:12.692 --> 01:03:16.652
So, like okay, how do you know what good is in production and what are you promoting to your offline eval sets?

s853
01:03:16.652 --> 01:03:21.693
And Um, yeah, it's not easy at all.

s854
01:03:21.693 --> 01:03:28.253
Um, I think that is Yeah, that's like there's like a lot of product work in that particular domain now.

s855
01:03:28.253 --> 01:03:31.893
As far as I haven't followed that area too closely.

s856
01:03:31.893 --> 01:03:43.613
Um, like what all the latest um, kind of updates that the various like tracing observability companies are working on, but my understanding is that it's like a very very active area of like product development among that class of companies.

s857
01:03:43.613 --> 01:03:46.213
So, that's pretty interesting.

s858
01:03:46.213 --> 01:03:48.893
Uh, okay, with that maybe we'll let everyone go.

s859
01:03:48.893 --> 01:03:51.373
And um, very nice to see you all.

s860
01:03:51.373 --> 01:03:55.069
So, feel free to reach out to me anywhere and thanks for the time.

s861
01:03:55.069 --> 01:03:57.069
[applause]