WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=9QebvrrY3KY

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/9QebvrrY3KY.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:14.440
Good to go.

s3
00:00:14.440 --> 00:00:14.760
All right.

s4
00:00:14.760 --> 00:00:19.520
Well, take a quick sip and then let's start.

s5
00:00:19.520 --> 00:00:20.840
It is great to be here.

s6
00:00:20.840 --> 00:00:25.120
Um this is like like my third year coming to this conference and I always really enjoy it.

s7
00:00:25.120 --> 00:00:26.360
And thank you for coming to this workshop.

s8
00:00:26.360 --> 00:00:29.280
I know there's many interesting talks.

s9
00:00:29.280 --> 00:00:36.280
Let me talk a little bit about um our view of async agents at Anthropic and some things we've been up to lately.

s10
00:00:36.280 --> 00:00:39.600
So, this is kind of a way I think about models in product.

s11
00:00:39.600 --> 00:00:46.800
So, you can think about Claude as a light source and you can think about products as windows that allow the light to pass through.

s12
00:00:46.800 --> 00:00:54.320
And what's kind of interesting is over time, the window that you need to actually kind of see the light of the model kind of shifts.

s13
00:00:54.720 --> 00:00:57.240
And we've seen this over the past few years.

s14
00:00:57.240 --> 00:01:00.920
So, I'm plotting here different Claude models and their task horizon.

s15
00:01:00.920 --> 00:01:04.960
So, how much autonomous work can they do over time?

s16
00:01:04.960 --> 00:01:12.920
And you might recall back in like the Opus 3 days, this was kind of like 2024, models could only do, you know, maybe 10 to 20 minutes of autonomous work.

s17
00:01:12.920 --> 00:01:15.320
This is measured by meter.

s18
00:01:15.320 --> 00:01:18.360
And in that regime, only certain product surfaces made sense.

s19
00:01:18.360 --> 00:01:26.600
Things like autocomplete, things like chat, where your human is very in the loop cuz the model's really only doing a very short amount of work before you're steering it.

s20
00:01:26.600 --> 00:01:30.120
Now, the past year we saw the rise of synchronous coding agents like Claude code.

s21
00:01:30.120 --> 00:01:35.280
And this is, you know, a kind of a shift because then models could do maybe an hour of work.

s22
00:01:35.280 --> 00:01:41.560
So, it made sense to have them run, but typically locally, where you could still steer them easily.

s23
00:01:41.560 --> 00:01:48.520
And it's kind of interesting because during this regime, I remember efforts and I was involved in some efforts to build kind of async agents.

s24
00:01:48.520 --> 00:01:54.080
But when models can only do like an hour of work, async as an experience is kind of bad.

s25
00:01:54.080 --> 00:01:59.080
Um the model goes off and it like hits an error and it comes back to you over a short period of time.

s26
00:01:59.080 --> 00:02:03.160
In order to really unlock async, we needed longer task horizons.

s27
00:02:03.160 --> 00:02:06.240
And so we're starting to see that now.

s28
00:02:06.479 --> 00:02:12.080
And kind of with this shift in capability and time horizon came a shift in the API surfaces.

s29
00:02:12.080 --> 00:02:17.480
So if you look at the lower left, Messages API came out like 2 years ago.

s30
00:02:17.480 --> 00:02:19.600
It's basically prompt response.

s31
00:02:19.600 --> 00:02:24.240
It's great for building harnesses, but it's a very simple API.

s32
00:02:24.240 --> 00:02:27.520
Again, you're just passing in a message and you get a response out.

s33
00:02:27.520 --> 00:02:29.360
There's no sense of deployment with that.

s34
00:02:29.360 --> 00:02:34.720
So you basically take Messages API and you can roll your own harness, you can deploy that harness, and you have an agent.

s35
00:02:34.720 --> 00:02:38.959
Now over the past year, we saw the rise of, you know, coding agents in particular.

s36
00:02:38.959 --> 00:02:40.480
So we released Agent SDK.

s37
00:02:40.480 --> 00:02:43.800
So that's basically a way to programmatically call Claude code.

s38
00:02:43.800 --> 00:02:47.240
And that's like basically us giving you a harness.

s39
00:02:47.280 --> 00:03:01.519
But over the past few months, since April, as we've seen longer longer task horizons, we released a new API called Managed Agents, which basically packages both the harness as well as all the managed deployment infrastructure for you.

s40
00:03:01.519 --> 00:03:13.320
And I'll talk about some of the themes that underpin this new uh surface, Claude Managed Agents, and and some of the themes that kind of extend beyond just Managed Agents broadly to think about this kind of new type of asynchronous agents,

s41
00:03:13.320 --> 00:03:19.320
uh which can apply of course to Claude and other types of kind of longer running long horizon agents.

s42
00:03:19.600 --> 00:03:23.040
So theme one is decoupling the brain from the hands.

s43
00:03:23.040 --> 00:03:28.640
Um so when we first set out to build Managed Agents, we started with the container.

s44
00:03:28.640 --> 00:03:32.680
We put the harness in the sandbox in the same container.

s45
00:03:32.680 --> 00:03:39.320
Now the problem here is, what happens if the harness dies or the container dies?

s46
00:03:39.320 --> 00:03:41.959
What we saw is you actually lose the session.

s47
00:03:41.959 --> 00:03:54.280
So basically, this architecture is kind of tricky for long horizon agents because what can happen is your agent's running, and And container dies, you lose everything with it.

s48
00:03:54.280 --> 00:04:02.520
Also, as models get more capable, putting the credentials in the same container with the agent itself can be problematic.

s49
00:04:02.520 --> 00:04:13.120
So, for example, giving Claude access to a bunch of your secrets and letting it run for 10 hours and not watching it can be a little bit spooky and have some security concerns, especially as models get extremely capable.

s50
00:04:13.400 --> 00:04:21.920
So, for this reason, we kind of decouple what we call the brain, that's the harness, from the hands, the execution environments, and manage agents to set up like this.

s51
00:04:21.920 --> 00:04:29.880
So, the story here is that the harness becomes a stateless process that talks to a session.

s52
00:04:29.880 --> 00:04:36.720
The session is an append-only event log and that can reach out to hands, which are just containers.

s53
00:04:36.720 --> 00:04:39.919
So, that's just Those are sandboxes where work is done.

s54
00:04:39.919 --> 00:04:44.800
And one thing that's interesting is Claude is increasingly capable of managing many hands.

s55
00:04:44.800 --> 00:04:50.480
So, that is you can give one harness access to many different containers to perform to execution.

s56
00:04:50.480 --> 00:04:54.919
And Claude can manage this very easily and and and and effectively.

s57
00:04:55.000 --> 00:05:08.520
If the session, uh sorry, if the harness dies or sandbox dies, it's completely fine because the session is always backed up in this append-only log and credentials are never actually added to the sandbox.

s58
00:05:08.520 --> 00:05:10.560
They're stored in a separate vault.

s59
00:05:10.560 --> 00:05:15.200
So, this decoupling actually makes it quite reliable and safe, particularly for long-horizon tasks.

s60
00:05:15.200 --> 00:05:20.440
And this is kind of one of the core ideas that underpins manage agents architecturally.

s61
00:05:20.760 --> 00:05:28.200
And I think an interesting thing that falls out of this is related to you guys may have kind of seen or come across the recursive language models work.

s62
00:05:28.200 --> 00:05:33.520
The session becomes an external context object that the model interrogates.

s63
00:05:33.520 --> 00:05:36.480
And this has all sorts of benefits for context management.

s64
00:05:36.480 --> 00:05:46.520
So, you think about it, when you're doing something like compaction, you're choosing some logic to retain some amount of context, and naively in a typical in a kind of a typical step, you're discarding

s65
00:05:46.520 --> 00:05:49.240
all the context that you didn't compact.

s66
00:05:49.240 --> 00:05:57.080
In this architecture, and also more broadly with recursive language models, the idea is that the context object is persistent and is unadulterated.

s67
00:05:57.080 --> 00:05:58.880
So, it's append-only.

s68
00:05:58.880 --> 00:06:02.000
And the model can always go back and fetch old context.

s69
00:06:02.000 --> 00:06:14.840
So, basically creates a very very nice architecture for context engineering because the core context object is immutable in the sense that it's it's it's non-destructive and you can only and you only append to it over time.

s70
00:06:14.840 --> 00:06:20.320
So, we've seen this be to be quite nice in terms of long horizon context engineering as well.

s71
00:06:20.840 --> 00:06:24.560
So, the second theme is use verifiers.

s72
00:06:25.000 --> 00:06:35.160
And one of the problems that we've seen with Claude and other models in general is that when you ask them to do a bunch of work and then say, "Okay,

s73
00:06:35.160 --> 00:06:37.120
grade your work."

s74
00:06:37.120 --> 00:06:47.280
If that same context is being used to both do the work and grade, you can get lots of odd artifacts and confabulation and and basically odd behavior.

s75
00:06:47.280 --> 00:06:55.840
For example, this is just an image showing you can think about that that context window is filled with lots of different information and the model is grading itself, often

s76
00:06:55.840 --> 00:07:00.440
it's not properly tuned to do kind of critical verification.

s77
00:07:00.440 --> 00:07:06.560
And so, what we found is it's quite effective to separate verification into a separate context window.

s78
00:07:06.560 --> 00:07:08.440
This is a very general trend.

s79
00:07:08.440 --> 00:07:11.120
We talked about it in a number of different engineering blogs.

s80
00:07:11.120 --> 00:07:18.199
Um and the reason is the verifier context can be tuned very specifically for the critical verification task.

s81
00:07:18.199 --> 00:07:18.600
[snorts]

s82
00:07:18.600 --> 00:07:25.480
And so, the way this works in practice is when you build loops, you can have a loop of a build context and a verifier context.

s83
00:07:25.480 --> 00:07:27.320
And this can be a build agent verifier agent.

s84
00:07:27.320 --> 00:07:36.400
And what happens is the verifier has some goal or rubric and it's verifying the result or work of the build agent.

s85
00:07:36.400 --> 00:07:40.480
And this continues in a loop until verification is complete.

s86
00:07:40.480 --> 00:07:50.200
And this is really the big idea behind this whole loops trend that you might have heard about and we found it to be a kind of a very powerful paradigm especially for working with some of the higher capacity models.

s87
00:07:50.400 --> 00:07:51.720
And here are some of the primitives.

s88
00:07:51.720 --> 00:07:58.240
So in Claude code you have goal and manage agents you have outcomes and the principles are really the same.

s89
00:07:58.240 --> 00:08:01.440
You're setting up a measurable end state in both cases.

s90
00:08:01.440 --> 00:08:03.720
You're using an independent context model.

s91
00:08:03.720 --> 00:08:09.320
You're using independent context to grade over the course of this loop.

s92
00:08:09.320 --> 00:08:15.960
The loop can run and you only exit the loop until this independent verifier has verified that it has the outcomes or outputs that you want.

s93
00:08:15.960 --> 00:08:18.440
That's the key idea.

s94
00:08:18.720 --> 00:08:20.800
Now let me tell you a story about how I've used this.

s95
00:08:20.800 --> 00:08:24.680
So this is a kind of a fun and interesting challenge called parameter golf.

s96
00:08:24.680 --> 00:08:28.680
It's a benchmark that set up that was put up by Open AI.

s97
00:08:28.680 --> 00:08:33.200
And it tests models ability to effectively do kind of ML research.

s98
00:08:33.200 --> 00:08:45.560
So it asks the model to basically take a small um kind of model and train it in with eight with eight 8100 GPUs in less than 10 minutes.

s99
00:08:45.560 --> 00:08:48.839
And what you see on the Y is basically you can think about it as loss.

s100
00:08:48.839 --> 00:08:51.360
So lower is better, okay?

s101
00:08:51.360 --> 00:09:03.400
And what I did was I set up a kind of a verifier loop using manage agents and outcomes to test the ability for Opus 4.7 and one of our frontier models like mythos class models

s102
00:09:03.400 --> 00:09:04.240
on this task.

s103
00:09:04.240 --> 00:09:16.640
And what you see is basically I allow the model to continue to iterate until the outcome that I specify is is satisfied which is it finished exactly 20 iterations

s104
00:09:16.640 --> 00:09:21.440
and kind of it kind of met all the experimental criteria as defined by the benchmark.

s105
00:09:21.440 --> 00:09:33.880
What you see is the frontier capability models are extremely good with this pattern of kind of loops in software and and kind of verification because what happens is instead

s106
00:09:33.880 --> 00:09:41.760
of encoding steering me and into like me as the human, you're encoding the signal into the environment.

s107
00:09:41.760 --> 00:09:47.040
So, the model can self-correct when it receives feedback from, for example, the verifier.

s108
00:09:47.040 --> 00:09:52.280
And using this kind of paradigm with very high-capacity models, you can get very strong results.

s109
00:09:52.280 --> 00:10:02.000
So, the main point I'm trying to make here is that this paradigm of loops, which a lot of people been talking about today, paired with very capacity models is a very

s110
00:10:02.000 --> 00:10:05.720
good general primitive for long-running asynchronous work.

s111
00:10:05.720 --> 00:10:07.846
That's really the key point here.

s112
00:10:07.846 --> 00:10:09.240
[snorts]

s113
00:10:09.240 --> 00:10:14.840
Now, let me talk another about another theme of self-learning.

s114
00:10:14.960 --> 00:10:21.000
So, the human brain has two kind of interesting systems for memory.

s115
00:10:21.000 --> 00:10:27.680
So, one is as you go about your day, the hippocampus is kind of writing traces of kind of short-term, very fast kind of experiential memory.

s116
00:10:27.680 --> 00:10:29.920
Like, you might remember what you had for lunch today.

s117
00:10:29.920 --> 00:10:33.760
You had lunch an hour ago, you kind of remember, that's kind of written to short-term memory.

s118
00:10:33.760 --> 00:10:38.960
When you go to bed at night, though, an offline process or out-of-band process, dreams.

s119
00:10:38.960 --> 00:10:43.400
And dreaming stores certain important details to long-term memory in the cortex.

s120
00:10:43.400 --> 00:10:46.040
So, for example, tomorrow, you might not remember what you ate for lunch today.

s121
00:10:46.040 --> 00:10:47.440
That's kind of a local trace.

s122
00:10:47.440 --> 00:10:55.560
But, like, if you had a very important experience today, maybe this talk, maybe not this talk, but if you had an interesting experience today, that might be written to long-term memory.

s123
00:10:55.560 --> 00:10:56.320
That's kind of the point.

s124
00:10:56.320 --> 00:10:59.080
These two subsystems in human work like this.

s125
00:10:59.080 --> 00:11:04.200
And we've actually found memory systems with Claude actually can employ these same two principles.

s126
00:11:04.200 --> 00:11:10.040
So, this is showing Claude's capacity as an in-band memory writer.

s127
00:11:10.040 --> 00:11:11.960
So, you basically give Claude memory tools.

s128
00:11:11.960 --> 00:11:17.520
And when I say memory tools, I mean it's basically the ability to write to a file system, that is basically a memory directory.

s129
00:11:17.520 --> 00:11:19.480
That's really it.

s130
00:11:19.480 --> 00:11:25.400
Now, this is showing some work on Claude plays Pokémon with Claude's on it 3.5. And here's the key point.

s131
00:11:25.400 --> 00:11:35.000
When Sonnet 3.5 is given this access to a memory directory, and it can write memory, {quote} in-band, as it progresses through this game, it's not very good.

s132
00:11:35.000 --> 00:11:36.960
So, the memories it writes are pretty crappy.

s133
00:11:36.960 --> 00:11:41.440
It's kind of um it's kind of uh tactical notes.

s134
00:11:41.440 --> 00:11:43.640
It's it's not very strategic.

s135
00:11:43.640 --> 00:11:46.040
And the game progress is quite limited.

s136
00:11:46.040 --> 00:11:53.720
But with more recent models like this is looking at 4 6, the notes are much more strategic and game progress is much further.

s137
00:11:53.720 --> 00:12:01.720
So, the key point I'm making here is that Claude has gotten much better at this in-band memory writing across model generations.

s138
00:12:02.600 --> 00:12:05.520
And this is another way to show that same result.

s139
00:12:05.520 --> 00:12:08.600
So, this is a benchmark that I ran called Continual Learning Bench.

s140
00:12:08.600 --> 00:12:09.520
It's an open-source benchmark.

s141
00:12:09.520 --> 00:12:10.480
I took one of the tasks.

s142
00:12:10.480 --> 00:12:19.320
This is a task that basically asked the model to perform a sequential question answering with a SQL database and it can write memory in between each step.

s143
00:12:19.320 --> 00:12:25.760
And what you see is basically the performance improves across models.

s144
00:12:25.760 --> 00:12:35.878
Um and so what this is kind of showing is that models get natively better at this in-band memory writing with respect to model capability.

s145
00:12:35.878 --> 00:12:36.120
[snorts]

s146
00:12:36.120 --> 00:12:43.320
And some of the most interesting things I found from this are that the main differentiation between There we go.

s147
00:12:43.320 --> 00:12:52.240
Um the main differentiation between like a lower capacity model and a high capacity model is kind of this distillation step.

s148
00:12:52.240 --> 00:13:00.520
And so, basically, higher capacity models have a better sense of like what abstraction to save to memory that'll be useful later.

s149
00:13:00.520 --> 00:13:02.720
Like they're not just writing a specific fact.

s150
00:13:02.720 --> 00:13:05.360
They're writing how do I how does this generalize to future sessions?

s151
00:13:05.360 --> 00:13:11.040
That's kind of the key difference that I found that higher capacity models kind of have when they're writing memory.

s152
00:13:11.040 --> 00:13:18.400
So, this is a very important thing to keep in mind that models are getting better better and better at this kind of in-band memory writing across model generations.

s153
00:13:19.200 --> 00:13:22.000
Now, there's a little trick here which is very important.

s154
00:13:22.000 --> 00:13:25.120
So, we talked about kind of in-band memory and we talked about dreaming.

s155
00:13:25.120 --> 00:13:28.920
So, at night I dream and I write things to like long-term memory.

s156
00:13:28.920 --> 00:13:37.440
Dreaming is very important because when I'm writing memory in band over the course of a day over the course of a session, sometimes you can write incorrect memories.

s157
00:13:37.440 --> 00:13:42.960
And or you're writing things that are locally optimal, but not globally optimal.

s158
00:13:42.960 --> 00:13:51.840
So, you're writing over the course of a task to like kind of help you solve that task, but not necessarily looking forward to future tasks.

s159
00:13:51.840 --> 00:14:00.240
This is a very important nuance in this process of dreaming is is kind of an offline or out of band process that we've used to consolidate and improve memory.

s160
00:14:00.240 --> 00:14:04.440
And I want to show you a fun example that I've used dreaming for.

s161
00:14:04.440 --> 00:14:06.880
So, this is again Pokémon.

s162
00:14:06.880 --> 00:14:12.200
And this is I played a lot of games of Pokémon with Claude to find this.

s163
00:14:12.200 --> 00:14:15.520
So, this was a very hard one lesson, so I hope you appreciate it.

s164
00:14:15.520 --> 00:14:17.120
Okay, here's the point.

s165
00:14:17.120 --> 00:14:23.440
So, basically what happened is Claude wrote an incorrect memory, okay?

s166
00:14:23.440 --> 00:14:29.640
And what happened is this incorrect memory was related to the location of The details don't necessarily matter.

s167
00:14:29.640 --> 00:14:40.640
Uh the point is that this incorrect memory causes Claude to mislocalize itself or Pokémon to mislocalize itself, and it falls through this trapdoor, okay?

s168
00:14:40.640 --> 00:14:42.120
That's the key point.

s169
00:14:42.120 --> 00:14:43.520
So, it writes this incorrect memory.

s170
00:14:43.520 --> 00:14:47.800
This incorrect memory causes it to mislocalize in the game, and it falls down this trap.

s171
00:14:47.800 --> 00:14:49.040
This is very consistent.

s172
00:14:49.040 --> 00:14:50.720
So, I saw this in five replicates.

s173
00:14:50.720 --> 00:14:54.800
Five out of five replicates with raw memory store fell down this trap.

s174
00:14:54.800 --> 00:15:03.000
With the dreaming, this error is corrected, and it's able to properly localize itself and not fall fall down this trap.

s175
00:15:03.000 --> 00:15:05.560
And I'll show you kind of a fun visualization of this.

s176
00:15:05.560 --> 00:15:09.960
This is looking at kind of memory traces or basically traces of game progress.

s177
00:15:09.960 --> 00:15:12.760
So, going upwards on the Y axis is improvement.

s178
00:15:12.760 --> 00:15:14.440
That's like moving to the next level.

s179
00:15:14.440 --> 00:15:16.760
Going down is backtracking.

s180
00:15:16.760 --> 00:15:23.760
So, what's interesting here, the no memory baseline, which is that gray bar, kind of doesn't make much progress at all.

s181
00:15:23.760 --> 00:15:25.280
It just kind of like is stuck.

s182
00:15:25.280 --> 00:15:28.200
It's like a particularly hard level, okay?

s183
00:15:28.200 --> 00:15:33.080
The memory, which is the orange, actually keeps falling down this trapdoor and falls back.

s184
00:15:33.080 --> 00:15:34.240
So, it backtracks.

s185
00:15:34.240 --> 00:15:41.080
The dreaming traces though consistently kind of fix this error in its memory and proceed to the next level.

s186
00:15:41.080 --> 00:15:48.120
So, this is a very practical example of how dreaming can kind of work out of band on your memory store to fix corrections.

s187
00:15:48.120 --> 00:15:54.800
Because what it does is it looks at your memory store and it looks at all your prior traces or sessions and kind of can find and correct errors.

s188
00:15:54.800 --> 00:16:02.400
That's the key point and that's why the dreaming process can be very helpful because in band, while Claude is writing to memory, it can make mistakes.

s189
00:16:02.400 --> 00:16:05.720
And those mistakes get stuck in memory unless you have an offline process to kind of correct them.

s190
00:16:05.720 --> 00:16:08.160
That's the key intuition.

s191
00:16:08.560 --> 00:16:19.840
Um and theme four, and I'll open up for questions after this, is what I think um is kind of this trend that we're going to see moving towards org-level harnesses with async agents.

s192
00:16:19.840 --> 00:16:28.200
And so, we released Claude Tag and a lot of the reaction was like, "Ah, Slack bot."

s193
00:16:28.200 --> 00:16:30.800
And like, look, I have actually created a lot of Slack bots myself.

s194
00:16:30.800 --> 00:16:33.240
I understand not every Slack bot is particularly interesting or great.

s195
00:16:33.240 --> 00:16:36.680
In fact, I've created many Slack bots that are quite bad.

s196
00:16:36.680 --> 00:16:41.760
But what's interesting about Claude Tag is not the fact that it's accessible through Slack.

s197
00:16:41.760 --> 00:16:49.080
What's interesting about it is the fact that it has a very very rich kind of system underneath it, which I want to just cut touch on briefly.

s198
00:16:49.080 --> 00:16:54.080
And in particular, what's interesting about it is it represents what I consider an org-level harness.

s199
00:16:54.080 --> 00:16:56.839
So, agents historically have been kind of single player.

s200
00:16:56.839 --> 00:17:03.600
So, you have an an agent like Claude code on your machine with your local context that you've tuned and configured for yourself.

s201
00:17:03.600 --> 00:17:09.400
What's interesting about Claude Tag is it is a harness that everyone in the organization has access to and can use.

s202
00:17:09.400 --> 00:17:11.520
So, it is a multiplayer harness.

s203
00:17:11.520 --> 00:17:14.000
And what's nice about that is it has its own identity.

s204
00:17:14.000 --> 00:17:20.959
Its identity and credentials are not tied to a given user and has access to organizational level context, not just my local context.

s205
00:17:20.959 --> 00:17:24.680
This has many interesting and useful implications.

s206
00:17:24.680 --> 00:17:40.640
Including the ability to like check others work before you do an experiment, the ability to kind of deduplicate findings, the ability to like do internal research, the ability to give everyone access to kind of a a very well developed harness on day one, whereas when you your own personal harness,

s207
00:17:40.640 --> 00:17:47.560
often new employees takes them weeks or maybe even months to kind of ramp up fully to configure all the right connectors and so forth.

s208
00:17:47.560 --> 00:17:56.880
So, org level harnesses are real leveler of the playing field and I think it was people kind of saw the Slack bot piece, but they didn't really appreciate the depth of kind of

s209
00:17:56.880 --> 00:18:00.120
that the depth of benefit you get from building out org level harnesses.

s210
00:18:00.120 --> 00:18:03.160
So, I do think that was kind of important thing to note.

s211
00:18:03.160 --> 00:18:14.240
And I think we're going to see the rise of kind of harnesses that operate across orgs, across many different users that can operate increasingly on longer async async um

s212
00:18:14.240 --> 00:18:16.840
async agents that can operate on longer time frames.

s213
00:18:16.840 --> 00:18:20.800
That's kind of one clear kind of follow-up that we're I think we're going to see from this.

s214
00:18:20.800 --> 00:18:28.680
And another thing where I think we're going to see is that asynchronous agents um are going to be increasingly proactive.

s215
00:18:28.680 --> 00:18:33.920
Uh so, typically with for example like local locally scoped agents, they tend to be reactive.

s216
00:18:33.920 --> 00:18:36.240
They're responsive to how you steer it.

s217
00:18:36.240 --> 00:18:40.600
versus async agents increasingly have the ability to steer proactivity.

s218
00:18:40.600 --> 00:18:43.120
And that's one very nice thing about Cloud Tag.

s219
00:18:43.120 --> 00:18:53.720
Um where basically you can configure it to tell you things when like looking at this org level context, alert me with things I might need to know about.

s220
00:18:53.720 --> 00:19:01.640
Um and this is a very important kind of new kind of UX that I think is going to be more and more common with async agents that kind of access to organizational context.

s221
00:19:01.640 --> 00:19:02.800
And of course multiplayer.

s222
00:19:02.800 --> 00:19:11.680
So, the the ability for a single harness to be steered by many many different people kind of concurrently is an important shift in agent UX uh that I think

s223
00:19:11.680 --> 00:19:13.960
uh will be quite interesting going forward.

s224
00:19:13.960 --> 00:19:21.920
So, um, yeah, let me let me just open up for questions and um, thank you for listening.

s225
00:19:21.983 --> 00:19:23.983
[applause]

s226
00:19:26.568 --> 00:19:28.568
[applause]

s227
00:19:30.600 --> 00:19:31.720
Sure.

s228
00:19:31.720 --> 00:19:41.680
Uh, one thing that we see sort of empirically and also in the benchmarks is that the frontier models perform better on these long horizon tasks than on the ones that have stacking.

s229
00:19:41.680 --> 00:19:49.200
What they need to keep in mind that this is not What is your view on and like the guidance on it and like the right away model?

s230
00:19:49.200 --> 00:19:51.200
Yeah.

s231
00:19:53.840 --> 00:20:00.480
Um, what's your view on why that is and then how long the frontier will be able to maintain that gap?

s232
00:20:00.480 --> 00:20:01.280
I see.

s233
00:20:01.280 --> 00:20:09.280
So, the question was kind of on the the gap between the frontier models kind of on for example, a benchmark like meter on like long horizon tasks.

s234
00:20:09.280 --> 00:20:09.960
Yeah.

s235
00:20:09.960 --> 00:20:11.040
Okay.

s236
00:20:11.040 --> 00:20:21.080
Um, so, in the latest results that I saw from like for example, Codex like 56, I think I'd also kind of is in that 12 plus hour regime on meter.

s237
00:20:21.080 --> 00:20:29.760
So, I think you're right that like the frontier models like the, you know, mythos class models, strong models from Open AI kind of are in this like 12 plus hour regime.

s238
00:20:29.760 --> 00:20:36.880
Um, why is it that non-frontier models are not kind of in that regime?

s239
00:20:36.880 --> 00:20:38.760
I am actually not necessarily sure.

s240
00:20:38.760 --> 00:20:50.640
I do think I do think um, in order to build agents that can effectively operate in this regime, it's important to note that it's not just the model capability.

s241
00:20:50.640 --> 00:20:56.960
Like for example, with a Claude tag product, it's actually a combination of improvement in memory because memory is very important.

s242
00:20:56.960 --> 00:21:03.320
If you have agents working for for example, 12 hours, you want to make sure that your productivity preferences are well encoded in memory.

s243
00:21:03.320 --> 00:21:06.120
So, it knows when to reach out to you if it gets stuck for example.

s244
00:21:06.120 --> 00:21:10.760
So, memory is very important, security is very important, so resistance to prompt injection.

s245
00:21:10.760 --> 00:21:18.720
Um also like model architecture, like kind of the agent architecture is very important, that decoupling of brain and hand, so it's secure and safe and like resistant to failure.

s246
00:21:18.720 --> 00:21:28.720
So, actually I think to build real agents that can operate in these long time horizons, a bunch of things need to come together in terms of like architecture, infrastructure,

s247
00:21:28.720 --> 00:21:30.960
security, memory.

s248
00:21:30.960 --> 00:21:37.800
And that might be why and and but Frontier Labs invested in all these areas, so that might be why it's become a gap in terms of like the agent products that we've released.

s249
00:21:37.800 --> 00:21:44.800
And so we spent a lot of time, for example, building managed agents to kind of have these kind of considerations baked in.

s250
00:21:44.800 --> 00:21:45.840
Yeah.

s251
00:21:45.840 --> 00:21:47.840
Sure.

s252
00:22:05.200 --> 00:22:07.480
Yeah, okay, this is interesting.

s253
00:22:07.480 --> 00:22:15.480
Um the question was about um kind of like the the best memory substrate, so like why file systems versus, for example, databases.

s254
00:22:15.480 --> 00:22:22.960
Um this is kind of a subtle point that actually um I want to think about carefully.

s255
00:22:22.960 --> 00:22:31.080
So, I don't necessarily think that it has to be the case that you use a file system for memory.

s256
00:22:31.080 --> 00:22:42.240
I think what's quite important that we've seen is that it's you want something that is highly programmable with simple primitives that the model can manipulate to like write, manage its own memory.

s257
00:22:42.240 --> 00:22:47.720
So, for example, a database could work fine relative to the file system.

s258
00:22:47.720 --> 00:22:57.080
But what I've seen doesn't work is when you specify the structure of memory for the model very explicitly, whether that's in a file system or database or whatever.

s259
00:22:57.080 --> 00:23:02.040
Like a memory schema, I pre I kind of pre populate, here's the types of memories you need to save.

s260
00:23:02.040 --> 00:23:08.800
Cuz I think that ends up being not very bitter lesson pill in the sense that models can learn to manage their own memory much better than you can intuit

s261
00:23:08.800 --> 00:23:11.160
these memory types for the model ahead of time.

s262
00:23:11.160 --> 00:23:20.880
So, I think what we've seen is that very general substrates for memory, be it just your database or file system, are good because the model can manage them freely versus

s263
00:23:20.880 --> 00:23:26.960
a very very kind of like prescriptive memory schema they are trying to pigeonhole the model into.

s264
00:23:26.960 --> 00:23:29.120
That's when you see performance drop.

s265
00:23:29.120 --> 00:23:31.560
That's the key differentiation.

s266
00:23:35.280 --> 00:23:37.120
Right.

s267
00:23:37.120 --> 00:23:38.440
That's the key point.

s268
00:23:38.440 --> 00:23:40.920
Let the model structure and maintain its own memory.

s269
00:23:40.920 --> 00:23:43.800
Don't give it a prescribed memory schema.

s270
00:23:43.800 --> 00:23:53.000
And that's like a common failure because models are getting good enough that they can manage their own memory much more effectively than you can reason about types of This is like very classically bitter lesson pill, but like

s271
00:23:53.000 --> 00:24:00.640
you can Models can reason about their own memory and context structure much better than you can prescribe for them a way to structure their own memories.

s272
00:24:00.640 --> 00:24:02.960
That's the key observation.

s273
00:24:02.960 --> 00:24:04.960
Yep.

s274
00:24:07.040 --> 00:24:09.040
Yes.

s275
00:24:10.120 --> 00:24:10.880
That's right.

s276
00:24:10.880 --> 00:24:12.120
Exactly.

s277
00:24:12.120 --> 00:24:15.880
General substrates for for memory management.

s278
00:24:15.880 --> 00:24:17.880
Yep.

s279
00:24:19.560 --> 00:24:21.560
Yes.

s280
00:24:22.800 --> 00:24:23.800
Okay.

s281
00:24:23.800 --> 00:24:26.240
So, that's a good point.

s282
00:24:26.240 --> 00:24:30.000
Basically, the question is so, you do this dreaming thing.

s283
00:24:30.000 --> 00:24:32.920
You look at the sessions, you look at the memory store, you update the memory store.

s284
00:24:32.920 --> 00:24:34.920
How do you know those are correct?

s285
00:24:34.920 --> 00:24:38.640
Um so, evaluations obviously are one way to do it.

s286
00:24:38.640 --> 00:24:45.240
Um this is kind of a fun anecdotal example from Pokémon showing that like you can do perform corrections via dreaming.

s287
00:24:45.240 --> 00:24:52.720
The key point is that we've actually run a lot of different evals showing that dreaming can indeed improve performance for very intuitive reasons as you see here.

s288
00:24:52.720 --> 00:24:58.600
But of course, evals are important in like your own context to confirm it's actually worth the offline compute.

s289
00:24:58.600 --> 00:24:59.240
Yep.

s290
00:24:59.240 --> 00:25:00.240
I guess we're done.

s291
00:25:00.240 --> 00:25:02.560
Thank you all.

s292
00:25:16.342 --> 00:25:18.342
[music]