WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7Y4sbZgBdnc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7Y4sbZgBdnc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:30.360
My name is Jedidiah and I joined Cursor early in 2025 as the longest tenured engineer working on our agent harness and recently I came along to SpaceX as part of the acquisition of Cursor.

s3
00:00:30.360 --> 00:00:39.920
So, today I want to talk to you all about how the agent harness has evolved to where we are today and where I think things are going.

s4
00:00:39.920 --> 00:00:49.360
And there's a motivating question that I have for us to answer today, which is is the agent harness dead?

s5
00:00:49.360 --> 00:00:52.520
So, let's get started.

s6
00:00:54.365 --> 00:00:55.400
[clears throat]

s7
00:00:55.400 --> 00:01:02.440
Here's a more natural and flowing version of that script which reads like a cohesive conference talk rather than a series of short points.

s8
00:01:02.440 --> 00:01:03.760
Oops.

s9
00:01:03.760 --> 00:01:07.480
Just making sure everyone's still got their soft detectors on before lunch.

s10
00:01:07.480 --> 00:01:09.200
Um no, for real.

s11
00:01:09.200 --> 00:01:17.280
So, the story starts in early 2023 when prompt engineer was the hottest title out there.

s12
00:01:17.280 --> 00:01:27.920
By mid-2025, that had evolved to context engineer, someone who programmatically puts the best context for the model together on the fly.

s13
00:01:27.920 --> 00:01:39.520
And then only at the beginning of this year did the industry land on the term agent harness and harness engineering to describe work that my team and I have been doing for a while.

s14
00:01:40.440 --> 00:01:43.560
So, what's in an agent harness?

s15
00:01:43.560 --> 00:01:48.040
The harness is like the eyes and the hands for the model.

s16
00:01:48.040 --> 00:01:53.040
It wraps around the model, steers it, and connects it to the world.

s17
00:01:53.040 --> 00:02:02.560
Users connect to our harness through apps and then the harness connects the model to computers and also to third-party services like MCPs.

s18
00:02:02.560 --> 00:02:10.240
I like to divide the harness into two layers, the runtime and the context and orchestration layers.

s19
00:02:10.640 --> 00:02:14.040
The runtime is kind of like the foundation of the harness.

s20
00:02:14.040 --> 00:02:21.280
It controls where the core agent loop runs and how state is stored so that work can resume.

s21
00:02:21.280 --> 00:02:25.959
Building this foundation is a lot like traditional software engineering.

s22
00:02:26.959 --> 00:02:31.760
And then, the context and orchestration layers are kind of like the wild west.

s23
00:02:31.760 --> 00:02:36.800
This is what a lot of people refer to when they talk about the harness.

s24
00:02:36.840 --> 00:02:44.120
These parts of the harness control which models are called when and what context they see.

s25
00:02:44.760 --> 00:02:55.080
That includes things like prompts and tools, of course, but also memory compaction and a bunch of tricks you might not know that we do to make the agent behave in the way that we want.

s26
00:02:55.080 --> 00:02:58.760
So, this is the area that I primarily work in.

s27
00:02:59.320 --> 00:03:03.000
In the new year, we kind of hit peak harness.

s28
00:03:03.000 --> 00:03:12.040
Cursor announced that we had created a browser entirely from scratch uh using a new long-running harness called Grind Swarm.

s29
00:03:12.040 --> 00:03:21.800
This was a multimodal harness that orchestrated a lot of a lot of agents and a lot of machines and it was very expensive to run.

s30
00:03:21.800 --> 00:03:26.760
Uh it actually took down Cursor maybe once or twice to to get this demo.

s31
00:03:26.760 --> 00:03:40.519
Um now, like anything in the world of agents, it only took a few months of peak harness for people online to start saying that the harness was dead.

s32
00:03:40.519 --> 00:03:46.320
Tariq from Anthropic talked about how they had killed much of the system prompt from Claude Code.

s33
00:03:46.320 --> 00:03:54.280
Garry Tan from YC thought that the harness would fade away and only skills would matter.

s34
00:03:54.280 --> 00:04:04.720
Even myself and my colleague Stefan, we wrote about how we had removed all these guardrails that we had in cursor to patch over gaps in the model's intelligence.

s35
00:04:04.720 --> 00:04:11.840
We used to have things like a small model just to apply code edits to a file because the frontier models weren't good at this.

s36
00:04:11.840 --> 00:04:18.760
We also had all sorts of coding specific instructions like how to explore the codebase and how to check your work.

s37
00:04:19.079 --> 00:04:28.320
So, if all these people are saying the harness is dead, they were saying that smarter models are what killed it.

s38
00:04:28.320 --> 00:04:31.800
And there's a lot of evidence for that.

s39
00:04:31.800 --> 00:04:39.480
I want you all to think about a very simple harness where the model only has access to the shell and to all of my secrets.

s40
00:04:39.480 --> 00:04:40.680
Please don't screenshot this slide by the way.

s41
00:04:40.680 --> 00:04:43.080
This one's off limits.

s42
00:04:43.640 --> 00:04:53.000
So, who here thinks that this really simple harness would do just as well on coding evals as a more sophisticated one?

s43
00:04:53.000 --> 00:04:54.160
Maybe some hands who thinks it would?

s44
00:04:54.160 --> 00:04:58.000
Who thinks so Who thinks it would if you Raise your hand if you think it would.

s45
00:04:58.080 --> 00:05:00.200
A few hands, but most of you know.

s46
00:05:00.200 --> 00:05:05.919
So, a few recent studies tried to investigate exactly that question.

s47
00:05:05.919 --> 00:05:14.800
The first one out of UC Berkeley tried seven different frontier models on a series of harnesses including Pi, Codex, and Cloud Code.

s48
00:05:14.800 --> 00:05:26.200
And what they found, particularly on the newer smarter models, is that the choice of harness did not actually make a difference significantly in the score on SweetBench light.

s49
00:05:26.200 --> 00:05:36.600
And then another another study from from Apple and EPFL found similarly that a dedicated MLE harness did not make a big difference on MLEBench.

s50
00:05:36.600 --> 00:05:39.000
So, what is going on here?

s51
00:05:39.000 --> 00:05:40.880
Is it Is it actually true that the harness is dead?

s52
00:05:40.880 --> 00:05:42.480
Are they right?

s53
00:05:42.480 --> 00:05:45.120
Here's what I think that the evidence shows.

s54
00:05:45.120 --> 00:05:49.520
A fancy harness in this day and age, no longer adds coding intelligence.

s55
00:05:49.520 --> 00:05:51.960
I think that is true.

s56
00:05:51.960 --> 00:05:54.120
I know that's surprising.

s57
00:05:54.120 --> 00:06:03.160
But, these things that we used to do in the harness to patch over gaps in the raw coding muscle memory of the model, like teaching it how to explore the code base,

s58
00:06:03.160 --> 00:06:10.440
uh how to work persistently for very long time with things like slash goal, uh how to run tests to check your work.

s59
00:06:10.440 --> 00:06:14.560
The model has natively absorbed these skills.

s60
00:06:16.520 --> 00:06:19.880
So, why is this happening?

s61
00:06:20.360 --> 00:06:27.440
It turns out the harness is actually making itself obsolete through what I call the data flywheel.

s62
00:06:27.440 --> 00:06:34.120
So, in this process, the harness brings harder and harder coding tasks within reach.

s63
00:06:34.120 --> 00:06:38.560
At that point, more users can productively attempt those tasks with agents.

s64
00:06:38.560 --> 00:06:41.600
That means more coding data is shared with the labs.

s65
00:06:41.600 --> 00:06:47.160
And eventually the models learn natively how to do these skills as well.

s66
00:06:47.680 --> 00:06:57.320
Eventually, the models get so good that the harness can actually delete the things it needed to do to make those tasks work in the first place.

s67
00:06:57.880 --> 00:07:00.720
It's a very interesting process.

s68
00:07:00.720 --> 00:07:04.120
So, that's how you get people saying that the harness is dead.

s69
00:07:04.120 --> 00:07:08.600
And if the models are good enough to code well in any harness, then why should you care?

s70
00:07:08.600 --> 00:07:11.960
Does it really matter which harness you use?

s71
00:07:12.520 --> 00:07:14.760
Is the harness really dead?

s72
00:07:14.760 --> 00:07:18.440
I say no, and there's two reasons why.

s73
00:07:18.440 --> 00:07:26.120
First of all, and I'm going to go into both these points, the harness actually opens the new frontier of tasks for the model to solve.

s74
00:07:26.120 --> 00:07:34.600
And then, second, there are some tasks or some parts of the harness, especially outside the context layer, which the model cannot absorb.

s75
00:07:34.600 --> 00:07:41.000
So, to first talk about how the harness unlocks the next frontier, I want to talk about Grokbot.

s76
00:07:41.000 --> 00:07:42.720
Who here has tried Grockbot?

s77
00:07:42.720 --> 00:07:44.400
Can I see some hands?

s78
00:07:44.400 --> 00:07:44.960
Awesome.

s79
00:07:44.960 --> 00:07:45.960
Let's go.

s80
00:07:45.960 --> 00:07:47.000
That's great.

s81
00:07:47.000 --> 00:07:52.920
Um so, for those of you who haven't, Grockbot is our autonomous colleague who gets work done for you.

s82
00:07:52.920 --> 00:07:56.120
Not just coding, but all kinds of knowledge work.

s83
00:07:56.120 --> 00:07:58.240
And I really recommend you try it.

s84
00:07:58.240 --> 00:08:00.560
I'm not going to sell too much, but it's a groundbreaking new product.

s85
00:08:00.560 --> 00:08:02.880
It's already created a new category.

s86
00:08:02.880 --> 00:08:05.040
Let's just talk about the harness, though.

s87
00:08:05.040 --> 00:08:16.960
So, Grockbot has a new kind of harness that operates not based off of one-off chats, but long-lived coordinator bots that orchestrate a fleet of worker agents.

s88
00:08:16.960 --> 00:08:29.200
You can kind of think of it like hiring a high-powered lawyer who deals with you, you know, directly and owns strategy, but they farm out their tasks to a team of paralegals.

s89
00:08:29.200 --> 00:08:31.520
It's like that.

s90
00:08:32.200 --> 00:08:43.800
Now, something interesting about the Grockbot harness is I mentioned how in Cursor we found ourselves removing things from the harness, removing prompting, removing guardrails as the models got smarter.

s91
00:08:43.800 --> 00:08:50.480
But, curiously in Grockbot, we've actually found ourselves putting back a lot of these things, adding new prompting, adding new guardrails.

s92
00:08:50.480 --> 00:09:01.520
And that's because there's a lot of new tasks in this new way of working, in the new domains that Grockbot is meant to solve, that the models have not nailed yet.

s93
00:09:01.839 --> 00:09:11.400
To give a few examples of those kind of things, just a few things that we're focused on right now when it comes to Grockbot, there's key website flows like making slide decks and booking travel,

s94
00:09:11.400 --> 00:09:12.720
communication.

s95
00:09:12.720 --> 00:09:17.480
An autonomous colleague needs to keep you up to date on what it's been doing without overwhelming you.

s96
00:09:17.480 --> 00:09:19.880
Acting versus asking is another big one.

s97
00:09:19.880 --> 00:09:24.880
Grockbot can do a lot of things which are kind of risky in some sense, like emailing your colleagues.

s98
00:09:24.880 --> 00:09:28.560
You want us to be able to do that, but you only want us to do it in the way that you intended.

s99
00:09:28.560 --> 00:09:33.720
And then, of course, learning over time the way that you work through memories is another big focus.

s100
00:09:36.680 --> 00:09:43.440
So, those of you who might have pushed the boundaries with coding agents might have noticed that none of these tasks are new.

s101
00:09:43.440 --> 00:09:49.600
But, for GrokBot, they're much more important because everybody is needing them in some way.

s102
00:09:49.600 --> 00:09:54.040
And right now in GrokBot, it is the harness that is making these tasks feasible.

s103
00:09:54.040 --> 00:09:59.040
Of course, the model is a big part of that, but the harness is helping bring them into its own reach.

s104
00:09:59.600 --> 00:10:13.160
So, going back to our data flywheel, GrokBot, that example, lets us fill in the last the last arrow in our cycle, which is when a capability like coding gets commoditized by the model,

s105
00:10:13.160 --> 00:10:22.520
it's actually the harness which unlocks a new frontier for the model to then solve and kicks off that flywheel again.

s106
00:10:22.520 --> 00:10:35.120
So, that might look like, you know, taking on new tasks outside of the coding domain, um as we just saw with GrokBot, or it could just mean taking on tasks in the software engineering job beyond

s107
00:10:35.120 --> 00:10:36.120
pure coding.

s108
00:10:36.120 --> 00:10:42.000
I'm sure some of you here are, you know, pretty senior in your software careers, and you might know the more senior you get, the less coding you're actually doing.

s109
00:10:42.000 --> 00:10:44.280
So, there's a lot of other things in the job that matter.

s110
00:10:44.280 --> 00:10:53.200
Um this is the core idea that I would like you to understand, this flywheel and how the model and the harness together drive it.

s111
00:10:53.480 --> 00:11:05.160
One more example of how the harness can unlock new frontiers is some of you may have heard of this idea of the software factory, where you have multiple agents coordinating together

s112
00:11:05.160 --> 00:11:09.240
to solve the full software development life cycle.

s113
00:11:09.240 --> 00:11:18.800
Now, this is far from a solved vision, but there's a lot of industry or there's a lot of innovation happening across the industry at the harness.

s114
00:11:18.800 --> 00:11:26.560
To give an example of how this might manifest, you know, internally, we have a lot of channels where someone might report a bug to that channel, and then an automation

s115
00:11:26.560 --> 00:11:28.520
immediately auto triages that bug.

s116
00:11:28.520 --> 00:11:36.440
Then it might kick off a cloud agent to try to fix the bug, and that agent will use our tooling to show before and after videos to prove that it's fixed.

s117
00:11:36.440 --> 00:11:46.960
And then, once that fix is deployed and or is merged in and starts deploying, we have our rollouts product, which kicks off another agent to watch in production for any issues.

s118
00:11:50.440 --> 00:11:56.800
So, you can think of all this as almost like a meta harness, which is orchestrating many agents together.

s119
00:11:56.800 --> 00:11:59.800
That's how I'd like you to think about this.

s120
00:12:00.000 --> 00:12:07.640
So, that's the first reason you should care about the harness because a frontier harness is needed to be at the frontier of agentic tasks.

s121
00:12:07.640 --> 00:12:12.480
And then the next reason is there are some parts of the harness that the model can't absorb.

s122
00:12:12.480 --> 00:12:15.280
So, let's talk about those.

s123
00:12:16.480 --> 00:12:21.920
First of all, there is the infrastructure and runtime needed to make an autonomous colleague reliable.

s124
00:12:21.920 --> 00:12:30.120
Over the past year plus, we've gone from agents that work on a typical task for minutes to hours to days and even longer.

s125
00:12:30.120 --> 00:12:36.160
And there's just no way that this can work on one person's laptop, which might go to sleep.

s126
00:12:36.160 --> 00:12:40.960
It has to be running in the cloud if you really want an autonomous colleague.

s127
00:12:43.320 --> 00:12:45.760
And then next, we have trust and security.

s128
00:12:45.760 --> 00:12:56.720
Yes, like we saw before, uh a simple harness with just bash and all your, you know, all your API keys in a file can solve coding tasks, but you probably don't want to give all your API keys in a file

s129
00:12:56.720 --> 00:12:57.839
to the model.

s130
00:12:57.839 --> 00:13:02.400
Um or if you're okay with that, I'm sure that your head of security would have something to say.

s131
00:13:02.400 --> 00:13:20.160
Um so, this is obviously really important companies where we want to have permissions that are scoped down to specific agents and specific people, but it's also important to have guardrails for anyone who doesn't want to wake up one morning to a really catastrophic action or to a $10,000 API fee, right?

s132
00:13:21.360 --> 00:13:24.320
And then finally, there's efficiency.

s133
00:13:24.320 --> 00:13:33.640
If you think back to to two studies I showed, where the choice of harness did not make a significant difference in eval scores, both of those studies actually found

s134
00:13:33.640 --> 00:13:38.000
that there was a significant difference in cost per task.

s135
00:13:38.000 --> 00:13:43.880
I have personally been focused over the past month on optimizing the efficiency of the Grok Bot harness.

s136
00:13:43.880 --> 00:13:55.800
Now, through tweaking variables like uh cache hit rate, improving that, token efficiency, um model mix, we were able to bring down the costs and the speed of Grok Bots significantly

s137
00:13:55.800 --> 00:13:58.200
for users without any hit to quality.

s138
00:13:58.200 --> 00:13:59.960
Users have really, really been appreciating this.

s139
00:13:59.960 --> 00:14:01.320
It's been awesome to see.

s140
00:14:01.320 --> 00:14:16.640
Um now, it could be an entire different talk about how we did this, but the key takeaway here is that the more and more that you use agents to do your work, the more it matters, the more valuable it will be for you to have the most efficient harness.

s141
00:14:18.160 --> 00:14:21.480
So, by now you might be convinced that the harness still matters.

s142
00:14:21.480 --> 00:14:24.360
Then you might be asking, "What's next?"

s143
00:14:24.360 --> 00:14:27.440
Here's a few trends that I'm anticipating.

s144
00:14:27.520 --> 00:14:35.560
First of all, that flywheel is speeding up, and right now we've entered a new age where the product is once again moving faster than the model.

s145
00:14:35.560 --> 00:14:39.640
That means that the harness is going to get more messy before it gets cleaned up.

s146
00:14:39.640 --> 00:14:41.480
There's going to be more things shipped.

s147
00:14:41.480 --> 00:14:50.640
So, because of this, we're expecting people to um build their harnesses, design them so it's easier to ship and unship new features.

s148
00:14:50.640 --> 00:14:58.120
One way we're already seeing that manifest is with both Grok Bot and other similar products, less and less of the harness inner workings are being exposed to the user.

s149
00:14:58.120 --> 00:15:05.080
Some of you may have heard of Hyrum's Law, which says that anything that you expose to users, they will come to depend on.

s150
00:15:05.080 --> 00:15:17.960
Now, obviously there are tradeoffs to giving users less, you know, less visibility into what's happening, but one advantage is that there's less change management to deal with, and you can iterate much faster.

s151
00:15:18.280 --> 00:15:26.120
Now, another way that we've been seeing um this trend manifest is a bigger focus on designing primitives for agents.

s152
00:15:26.120 --> 00:15:32.600
Primitives are like building block interfaces that can be composed and extended into powerful features.

s153
00:15:32.600 --> 00:15:41.520
When you have primitives, you can add new features without making things more complicated for the agent, and you can remove features without having to do a complicated refactor in your code base.

s154
00:15:41.520 --> 00:15:43.480
It's just good engineering.

s155
00:15:43.480 --> 00:15:48.920
One example across our agent harnesses of a primitive we've added is called subscriptions.

s156
00:15:48.920 --> 00:15:51.760
Agents need a way to wake up in response to events.

s157
00:15:51.760 --> 00:16:00.200
Those might be someone posting in Slack, CI going red on a PR that the agent cares about, or just a certain time of day.

s158
00:16:00.200 --> 00:16:12.520
Now, rather than having one separate integration, you know, having separate integrations for each of these things, we have one framework and one tool that the model can use or the agent can use to subscribe to an event.

s159
00:16:12.520 --> 00:16:20.120
And then another cool thing that we do, I mentioned earlier that we have these deterministic guardrails to patch over bad behaviors in the model.

s160
00:16:20.120 --> 00:16:28.360
That might be, for example, if GrokBot doesn't respond to the user fast enough, and we might give it give it a little nudge and say, "Hey, user's waiting on you."

s161
00:16:28.360 --> 00:16:31.040
Now, we'd rather not do these things.

s162
00:16:31.040 --> 00:16:36.760
They're kind of like hacks, but we would rather fix them now than, you know, wait for the next model checkpoint to to get released.

s163
00:16:36.760 --> 00:16:46.120
So, in order to make our code base, you know, be less messy and and have less of a worse time iterating the future, we track all these things, which we call shims centrally,

s164
00:16:46.120 --> 00:16:48.640
and every time the model uses one, it gets logged.

s165
00:16:48.640 --> 00:16:58.040
So, now we have a bot who watches this dashboard, and whenever a shim stops firing, that model or that agent will put up a PR and say this shim can be cleaned up.

s166
00:16:58.640 --> 00:17:04.480
Another thing that I think is next for the harness is it is moving up to a higher level of abstraction.

s167
00:17:04.480 --> 00:17:08.680
Yesterday's generation of harnesses were built from composing a bunch of LLM calls.

s168
00:17:08.680 --> 00:17:15.800
Now, the new harnesses we're seeing, like GrokBot and the software factory, are using agents as the core building block.

s169
00:17:15.800 --> 00:17:22.199
And then finally, I predict that within 6 months, 90% of agent work is going to run off the laptop.

s170
00:17:22.199 --> 00:17:35.720
This is already where things are going, and I think there's just no way that, you know, this is this is not necessary in order to have truly autonomous agents who can, you know, really work with us in in the way that we want.

s171
00:17:36.720 --> 00:17:39.760
So, is the harness dead?

s172
00:17:39.760 --> 00:17:45.440
The parts that made it smarter at coding, yes, that is the flywheel at work.

s173
00:17:46.120 --> 00:17:47.680
That is the flywheel at work.

s174
00:17:47.680 --> 00:17:53.760
But, the harness, as we saw, is how the agents can unlock the next frontier of work.

s175
00:17:53.760 --> 00:18:00.400
And then also the trust, the efficiency, the infrastructure that the harness brings, those things are not going anywhere.

s176
00:18:00.400 --> 00:18:07.000
So, yesterday's harness may be dead, but long live the agent harness of tomorrow.

s177
00:18:07.000 --> 00:18:08.622
Thank you.

s178
00:18:08.622 --> 00:18:10.622
[applause]