WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=pz74r-4dGfQ

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/pz74r-4dGfQ.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.520 --> 00:00:24.680
I'm Ben Holmes and I'm a DevEx engineering lead at Warp and I'm going to be talking about building self-improving agents, agents that look through their past work and figure out how to improve their own setup.

s2
00:00:24.680 --> 00:00:36.160
Uh so for a bit of context, I work on Warp Factories, which is open flexible infrastructure to help you all build software factories of your own and crucially ones that improve themselves

s3
00:00:36.160 --> 00:00:37.800
as part of the stack.

s4
00:00:37.800 --> 00:00:46.400
Uh I'm not going to be talking too much about software factories as a whole today, but I'm going to be talking about one crucial part of it, which is having agents introspect

s5
00:00:46.400 --> 00:00:54.720
all of the work that they do inside of one of those systems and suggest improvements so that the agent performs better and better over time.

s6
00:00:54.720 --> 00:01:02.400
Uh so I'm going to start by just presenting what you might think of when you hear the word software factory for a bit of definition setting.

s7
00:01:02.400 --> 00:01:05.920
It's the idea of automating the software development life cycle.

s8
00:01:05.920 --> 00:01:10.240
You may have seen this as an assembly line of different agents doing different things and handing off.

s9
00:01:10.240 --> 00:01:19.640
You may have also seen it as a wheel as you see in this case where you start from a task list, a board, Slack messages, wherever you're tracking work

s10
00:01:19.640 --> 00:01:35.880
and you have an assembly line of agents go through the process of researching it or triaging, maybe working with a human to write a spec, implementing that change, and bringing computer use into the picture to take screenshots and recordings so that it's actually a good pull request that you get on the other side.

s11
00:01:35.880 --> 00:01:43.640
And also maybe doing rounds of adversarial code review in the mix uh to make sure that it's something that is ready for human eyes.

s12
00:01:43.640 --> 00:01:48.360
Um and on the other hand you have agents monitor the system and around and around we go.

s13
00:01:48.360 --> 00:01:59.200
Uh but the more interesting part I would say is all of the sort of meta-engineering that goes around making that wheel run as efficiently as it can.

s14
00:01:59.200 --> 00:02:09.452
Um so a few terms have been maybe floating around right now like meta engineering, gardening, or something that we've been using which is factory engineering.

s15
00:02:09.452 --> 00:02:16.452
It's the idea of actually putting in engineering work to improve the system that is driving the agents themselves.

s16
00:02:16.452 --> 00:02:20.132
And there are a lot of things that you might want to optimize inside of that system.

s17
00:02:20.132 --> 00:02:29.332
It may be the skills that are driving the agents, it may be improving the code architecture and doing static analysis to make sure the agents are working on the right parts of the code base.

s18
00:02:29.332 --> 00:02:34.932
I've been hearing a lot about like SonarQube and other sponsors at this event that are doing things in this vein.

s19
00:02:34.932 --> 00:02:40.412
The improvement loop that we're going to be talking about is specifically around agent skills and configuration.

s20
00:02:40.412 --> 00:02:42.612
And the loop is going to look something like this.

s21
00:02:42.612 --> 00:02:47.132
It's the idea that you can have agents review their past conversations.

s22
00:02:47.132 --> 00:02:51.972
You can set up some scoring agents to score the quality of how those conversations went.

s23
00:02:51.972 --> 00:03:00.412
So, think of this like an LLM as a judge reviewing against rubrics that you define across tricks like efficiency or code quality or test coverage.

s24
00:03:00.412 --> 00:03:08.972
And then once you graded a whole lot of conversations across all the work your team is doing, you could find all the failures, figure out reasons that those failures are happening,

s25
00:03:08.972 --> 00:03:12.292
and then maybe an agent wakes up on a cron to suggest improvements.

s26
00:03:12.292 --> 00:03:16.812
And then you can continue evaluating, seeing how the scores perform, and so on.

s27
00:03:16.812 --> 00:03:24.092
So, I'm going to break down that wheel into the individual segments that we can talk about each of them individually.

s28
00:03:24.092 --> 00:03:35.772
So, the first part of that loop that I showed you is getting all of your agent conversations in one place so that the agent can actually reason about all the work that your team is doing.

s29
00:03:35.772 --> 00:03:45.012
If you're working with you know local agents like Claude code or codex or pie or any of those, those are going to be files on your system.

s30
00:03:45.012 --> 00:03:54.452
So, Claude you're going to have a Claude folder world that lives, codex you're going to have a codex folder, and that works well enough if you want to sort of optimize your single player setup.

s31
00:03:54.452 --> 00:04:03.854
I've actually talked to teams that set up an S3 bucket and an automation to like take everyone's dot Claude folders and zip them up and put them in a shared S3 instance

s32
00:04:03.854 --> 00:04:06.894
just so we have everyone's conversations in one place.

s33
00:04:06.894 --> 00:04:09.534
It It's a way of doing things.

s34
00:04:09.534 --> 00:04:15.054
Um but there are other ways that you can sort of centralize this information.

s35
00:04:15.054 --> 00:04:19.214
Uh if you're using a software factory system, it kind of comes for free.

s36
00:04:19.214 --> 00:04:23.494
This is the idea that agents are running inside of maybe a shared computer.

s37
00:04:23.494 --> 00:04:29.374
So not on everyone's individual laptops, but like a shared computer that everyone's going to send tasks into.

s38
00:04:29.374 --> 00:04:37.934
That might be like a long-running compute instance on Open Claw or it might be like isolated sandboxes that spin up for every run that you do.

s39
00:04:37.934 --> 00:04:44.494
Uh if you're using Warp Factory setup, it would be isolated Docker sandboxes for each agent working on tasks.

s40
00:04:44.494 --> 00:04:47.814
Using other cloud runners, you'll probably end up in the same sort of system.

s41
00:04:47.814 --> 00:04:49.414
But it's all the same idea.

s42
00:04:49.414 --> 00:04:58.334
We want to find a way to get everyone's team your whole team's conversations in one place so that they're easy to introspect and figure out how to improve your setup.

s43
00:04:58.334 --> 00:05:06.134
And if you're doing it on one cloud runner where all the tasks flow through, you kind of get that for free cuz all the agent traces belong to a single instance.

s44
00:05:06.134 --> 00:05:08.454
Um so that's something that's baked in.

s45
00:05:08.454 --> 00:05:13.334
And inside of that, you're going to want to make sure you have all the information up front.

s46
00:05:13.334 --> 00:05:20.054
So not just the like transcript talking to the agent, what did it send back, what did the human send back.

s47
00:05:20.054 --> 00:05:22.734
You also want all of the metadata around it.

s48
00:05:22.734 --> 00:05:35.414
So that means the tool calls, the file system calls, the MCPs that it's connecting to, everything that connects the agent to the outside world should also be inside of that agent trace when you start doing this analysis.

s49
00:05:35.414 --> 00:05:37.494
And you also want the outputs to be in there.

s50
00:05:37.494 --> 00:05:41.814
That can just be the code diffs that live inside of all of your conversations.

s51
00:05:41.814 --> 00:05:47.134
It can also be the actual pull request on the other side if you like pin the commit and look at the diff.

s52
00:05:47.134 --> 00:05:49.614
In any case, you just want to make sure that you have three things.

s53
00:05:49.614 --> 00:06:00.254
You got the transcript of what was said, you've got the tool calls that show what the agent did, and then you have the output which is something that you may want to grade for quality purposes.

s54
00:06:00.254 --> 00:06:07.135
Uh and once you have all of that, you can start actually scoring the quality or rather you can have an agent start scoring the quality.

s55
00:06:07.135 --> 00:06:11.055
Um so, I like doing a a show of hands for this little part.

s56
00:06:11.055 --> 00:06:15.735
Um how many people in here feel like their agents write too many tests?

s57
00:06:15.935 --> 00:06:16.895
Too many tests.

s58
00:06:16.895 --> 00:06:24.215
Okay, I expected more, but for those at home, it's about like 40% of people, which I think is a pretty big number.

s59
00:06:24.215 --> 00:06:26.975
On our team, it was about 100% of hands.

s60
00:06:26.975 --> 00:06:30.855
Uh it can depend on your model, and it can depend on your instructions.

s61
00:06:30.855 --> 00:06:35.255
But, let's say that your team is noticing your agents are writing a lot of tests.

s62
00:06:35.255 --> 00:06:43.495
Maybe extraneous ones, maybe you have your own opinions about do you need end-to-end test coverage versus integration coverage versus unit test coverage.

s63
00:06:43.495 --> 00:06:45.775
People are experimenting with all sorts of things there.

s64
00:06:45.775 --> 00:06:55.095
So, you might want to set up a scorer for that called redundant test that evaluates how often is the agent writing too many tests across each conversation that it has.

s65
00:06:55.095 --> 00:07:01.535
Uh another thing that you may want to score is um the way the agent communicates with you.

s66
00:07:01.535 --> 00:07:06.535
So, maybe another show of hands, how many people feel like their agents write too many comments?

s67
00:07:06.895 --> 00:07:08.895
Curious about this one.

s68
00:07:08.895 --> 00:07:12.735
Yeah, I'm not going to call out the model name, but I I have an idea.

s69
00:07:12.735 --> 00:07:12.959
Uh

s70
00:07:12.959 --> 00:07:13.775
[laughter]

s71
00:07:13.775 --> 00:07:15.655
It it could depend.

s72
00:07:15.655 --> 00:07:24.815
Uh but, this is also a common issue with agents, and it's not just for noise or for visuals, because we all know that code review is kind of the biggest bottleneck right now.

s73
00:07:24.815 --> 00:07:36.295
It does matter how well the agent communicates changes, both summarizing them, and also in the comments that it leaves within the code itself, because they can be barriers to actually get a code change reviewed quickly.

s74
00:07:36.295 --> 00:07:44.615
Um so, you may have another metric like verbosity, which is what we call it, which is is the agent overly verbose when it's talking to you or when it's writing code?

s75
00:07:44.615 --> 00:07:49.015
Both of those are examples of kind of the same thing, which is a handle on language.

s76
00:07:49.015 --> 00:07:52.535
Um so, you could set up a scorer that sort of defines that.

s77
00:07:52.535 --> 00:08:00.935
And a third one I like to bring up is just efficiency of the agent, and this speaks to the need for tool calls and MCP calls in your traces.

s78
00:08:00.935 --> 00:08:13.337
Because when you're running things a little bit uh without a human in the loop, so you're sending them off on longer and longer runs, maybe they're running in a cloud sandbox, so you're not looking at the conversation itself.

s79
00:08:13.337 --> 00:08:18.297
It becomes easier and easier to miss all of the failed tool calls that an agent does.

s80
00:08:18.297 --> 00:08:24.897
Like, an agent these days are very persistent and will work around whatever roadblock that they run into, and they won't tell you about it.

s81
00:08:24.897 --> 00:08:27.177
They'll just tell you that it worked on the other side.

s82
00:08:27.177 --> 00:08:34.017
And one common failure mode that we ran into was like, in our test harness, there were a ton of bad instructions that were out of date in the skills,

s83
00:08:34.017 --> 00:08:41.817
but the agent worked around them and never told us because it figured it out just by trying to ping a bunch of port numbers, figure out its environment, and then eventually start doing code.

s84
00:08:41.817 --> 00:08:46.097
So, the most that we knew was like, "Oh, the agent takes like 2 minutes longer than maybe we expect."

s85
00:08:46.097 --> 00:08:48.617
But, under the hood, there's actually a lot to be improved.

s86
00:08:48.617 --> 00:08:57.217
So, that's something you can evaluate as a score to just say, looking through the tool calls, is it efficiently getting to the solution, or is it spinning its wheels a little bit?

s87
00:08:57.217 --> 00:09:00.057
And I can show you what one of these scores looks like.

s88
00:09:00.057 --> 00:09:03.657
Uh it's just markdown, as you would probably expect.

s89
00:09:03.657 --> 00:09:13.257
Uh in our setup here, it'll kind of be like a specialized skill definition, where you can define what thing you're trying to score with an LLM as a judge approach.

s90
00:09:13.257 --> 00:09:22.537
Um so, here we're evaluating efficiency in this score, saying whether the agent worked towards the result or wasted effort on redundant steps and unnecessary back and forth.

s91
00:09:22.537 --> 00:09:27.817
And then we set some labels that define what success and failure look like on that criteria.

s92
00:09:27.817 --> 00:09:36.377
So, if you remember back in high school writing five-paragraph essays, and the teacher gives you a rubric of like, "You wrote a good essay if you have exactly five paragraphs,

s93
00:09:36.377 --> 00:09:39.017
and you only use active voice instead of passive."

s94
00:09:39.017 --> 00:09:40.417
That sort of thing.

s95
00:09:40.417 --> 00:09:48.057
It is still kind of helpful to put an agent through those paces for things that can be pretty clearly defined, efficiency being one of them.

s96
00:09:48.057 --> 00:09:57.057
Um so, we set up one of these rubrics to have that teacher agent go through and decide, do you get good marks on efficiency, or is it kind of below the bar?

s97
00:09:57.057 --> 00:09:59.577
And you can decide what pass and fail look like.

s98
00:09:59.577 --> 00:10:08.457
In our case, maybe the bound is halfway through this list, so if it's a 0.4 or 0.2, that's kind of a failing grade, something went wrong in that conversation,

s99
00:10:08.457 --> 00:10:12.378
and above it, it is a passing grade, and probably doesn't need to be evaluated.

s100
00:10:12.378 --> 00:10:14.978
Things went mostly okay.

s101
00:10:14.978 --> 00:10:20.778
And then down here is the actual definition of what that rubric means and it's going to restate some stuff that you would expect.

s102
00:10:20.778 --> 00:10:25.218
So, cost human, the information gathering phase, routine step overhead.

s103
00:10:25.218 --> 00:10:29.578
And you should keep this as succinct as possible and just build it up as you go.

s104
00:10:29.578 --> 00:10:38.538
Um this is a template that we like to give to people, but you can set it up with literally just like a couple bullet points and just your rubric and test it out yourself.

s105
00:10:38.538 --> 00:10:40.778
It's more of an art than a science.

s106
00:10:40.778 --> 00:10:46.418
Um but once you have all these scores set up, then you can start having agents actually grade the conversations.

s107
00:10:46.418 --> 00:10:56.938
So, over here, let's say we have uh some sample sessions that we want agents to look through uh starting with like an off service that the agent implemented in a full stack context.

s108
00:10:56.938 --> 00:11:08.898
Let's say that we had a full agent conversation about building a full stack implementation of this and the PR was merged and now it's time for an agent to go grade how that conversation went.

s109
00:11:08.898 --> 00:11:10.458
Well, the agent will spin up.

s110
00:11:10.458 --> 00:11:16.298
In our case, we're saying one agent per conversation or one agent run per conversation run.

s111
00:11:16.298 --> 00:11:22.818
You can set it by random sampling for, you know, just the sake of tokens or you can run it across every conversation.

s112
00:11:22.818 --> 00:11:24.058
Really depends.

s113
00:11:24.058 --> 00:11:30.138
Um but let's say in a random sampling, it pulled out this one and then told an agent to grade across each of those criteria.

s114
00:11:30.138 --> 00:11:40.898
So, walking down the line, it might say, "Looking at the test coverage, it wrote a ton of unit test for the off service when our guidelines say that you should mostly prefer end-to-end test."

s115
00:11:40.898 --> 00:11:45.298
And so, maybe we give it a fail because it didn't really follow the instructions that we expected.

s116
00:11:45.298 --> 00:11:48.538
And then maybe for verbosity, it was actually pretty clear.

s117
00:11:48.538 --> 00:11:52.618
It was using a new model selection that doesn't write code comments.

s118
00:11:52.618 --> 00:11:54.818
Uh go GPT, anyone anyone in the room?

s119
00:11:54.818 --> 00:11:55.698
I don't know.

s120
00:11:55.698 --> 00:11:59.258
Uh but let's say that it didn't really write too many comments.

s121
00:11:59.258 --> 00:12:02.098
It was very easy to understand the PR, so it gets a pass.

s122
00:12:02.098 --> 00:12:10.018
And then moving on to efficiency, let's say it was trying to do a lot of exploration in its test environment to figure out how to test this change in the first place

s123
00:12:10.018 --> 00:12:12.979
when really it should have had some upfront instruction on that.

s124
00:12:12.979 --> 00:12:16.619
And that would probably be a failing grade on that score.

s125
00:12:16.619 --> 00:12:21.739
And again, pass and fail is just did it fall below the line on the rubric that we set.

s126
00:12:21.739 --> 00:12:26.659
Uh and then you can just go through each of your agent conversations and continue scoring them.

s127
00:12:26.659 --> 00:12:34.059
So, automatically, usually the trigger being a PR merged, kick in an agent to review that conversation across your grading criteria.

s128
00:12:34.059 --> 00:12:39.739
And then over time, you'll start aggregating a lot of data on how conversations go.

s129
00:12:39.739 --> 00:12:43.339
And each of these will include like the pass and failure rates.

s130
00:12:43.339 --> 00:12:46.219
So, did it pass and fail across each definition?

s131
00:12:46.219 --> 00:12:51.579
And then you also have an agent describe why one of those failures or passes happened.

s132
00:12:51.579 --> 00:12:56.459
And then this becomes context that another agent can use to decide what improvements to make.

s133
00:12:56.459 --> 00:13:02.979
So, the way that we set it up, going back to this wheel, is the idea you grade conversations as soon as they happen.

s134
00:13:02.979 --> 00:13:09.259
And then maybe on a cron job, every 24 hours, go through a sampling of the past 24 hours of conversations.

s135
00:13:09.259 --> 00:13:11.019
Maybe there's been like 200.

s136
00:13:11.019 --> 00:13:12.779
Filter it down to the failures.

s137
00:13:12.779 --> 00:13:19.579
And then pass in all of those failing reports that describe what failed and why to a self-improvement agent.

s138
00:13:19.579 --> 00:13:22.499
And tell it these were the 50 conversations that failed.

s139
00:13:22.499 --> 00:13:24.779
Here are the notes that were left on each conversation.

s140
00:13:24.779 --> 00:13:26.419
Go ahead and find commonalities.

s141
00:13:26.419 --> 00:13:31.899
Here's access to our skills repository and the code base if you want to make architecture changes.

s142
00:13:31.899 --> 00:13:38.659
Um so, in this little snippet I showed, maybe it adds a coverage pyramid because redundant tests are popping up across a lot of PRs.

s143
00:13:38.659 --> 00:13:40.819
And I can show you a couple other examples.

s144
00:13:40.819 --> 00:13:47.339
Um both in a skill that you can take home, and also the automated system that we have inside of Warp Factories.

s145
00:13:47.339 --> 00:13:50.859
So, this is going to be the easiest one that y'all get to try.

s146
00:13:50.859 --> 00:13:52.699
Um so, it's called Skill Doctor.

s147
00:13:52.699 --> 00:13:55.299
It's something you can find at warp.dev/skilldoctor.

s148
00:13:55.299 --> 00:13:56.819
I'll show the link in a second.

s149
00:13:56.819 --> 00:14:07.179
Um but this is a skill that you can run on your single-player setup, and it'll crawl through all of your agent conversations and Claude, Codex Buyer, whatever else, and just do a one-time grading.

s150
00:14:07.179 --> 00:14:14.619
So, if you've been wondering about all the hidden gems in your conversations cuz you haven't done this approach before, you can just run this across your setup and see what happens.

s151
00:14:14.619 --> 00:14:24.421
So, this was a sample I ran across some code sessions building this project here, and we can see on the side that it grades in batches for a little bit of token efficiency,

s152
00:14:24.421 --> 00:14:29.541
and it came up with some score reports across all of the metrics that we care about.

s153
00:14:29.541 --> 00:14:33.661
And this test environment has efficiency and code quality as the ones that it evaluates.

s154
00:14:33.661 --> 00:14:40.101
Uh and then on the other side, the self-improvement agent looks at all those scores and comes back with a report.

s155
00:14:40.101 --> 00:14:42.261
Um so, let's go ahead and take a look at that.

s156
00:14:42.261 --> 00:14:45.061
Uh I like to show the animation.

s157
00:14:45.061 --> 00:14:46.461
We love it.

s158
00:14:46.461 --> 00:14:52.781
Uh but this is like kind of your skill report card that shows how the agent performed across all of those rubrics that we defined.

s159
00:14:52.781 --> 00:14:53.861
Again, this is one shot.

s160
00:14:53.861 --> 00:14:58.341
It's probably something that you want to do continuously on your setup rather than whenever someone remembers.

s161
00:14:58.341 --> 00:15:03.341
But as a first go to just see what you can optimize right now, it's nice to get some low-hanging fruit.

s162
00:15:03.341 --> 00:15:10.301
Um so, in this example, it graded across these metrics and noticed that these were a little bit lower than expected, but verbosity was fine.

s163
00:15:10.301 --> 00:15:17.221
Um so, it went through all of the notes that were given by the scoring agents, and then suggested some changes to the skill makeup.

s164
00:15:17.221 --> 00:15:25.541
And this one is going to mirror what I was mentioning earlier, uh where it noticed that the agent was usually trying to fight for port numbers on the machine that it was testing,

s165
00:15:25.541 --> 00:15:28.021
and it didn't really understand how the environment was set up.

s166
00:15:28.021 --> 00:15:38.821
So, this proposed a change to the end-to-end testing skill to have the agent get better instructions on finding a port number first and then going for it instead of just banging against the wall until it got something running.

s167
00:15:38.821 --> 00:15:42.341
Um so, that can save a little bit of efficiency uh in the tool calls.

s168
00:15:42.341 --> 00:15:51.221
It also noticed that I constantly wanted to write a different changelog report with the language that was specified, and so it rewrote my changelog skill to have a more

s169
00:15:51.221 --> 00:15:53.261
uh prescriptive language on how these things are written.

s170
00:15:53.261 --> 00:15:58.181
So, I don't have to do that manual back and forth every time an agent opens a PR.

s171
00:15:58.181 --> 00:16:08.301
Um and these were things that I could have gone and found myself, but especially this one on port numbers, since I'm not watching the traces super closely, it's usually just behind like a working toggle if you're using

s172
00:16:08.301 --> 00:16:14.461
like an agent harness or it's behind the cloud, uh it's nice to pick up on things like this so you don't miss them.

s173
00:16:14.461 --> 00:16:19.902
Um so, last thing that I'll leave you with is how you can run this continuously over time.

s174
00:16:19.902 --> 00:16:21.702
Uh oh, shoot, it was just there.

s175
00:16:21.702 --> 00:16:25.222
No, I lost access to my staging.

s176
00:16:25.222 --> 00:16:29.302
Uh well, I can at least describe with words what that was.

s177
00:16:29.302 --> 00:16:34.782
Uh which is the set of sort of graphs on the scores that are written.

s178
00:16:34.782 --> 00:16:39.742
Because one thing I was actually explaining this to someone who uses like a memory system in their harness.

s179
00:16:39.742 --> 00:16:43.142
Uh and memories are a really useful way of having agents like introspect.

s180
00:16:43.142 --> 00:16:47.622
Maybe you've heard of like dreaming overnight and thinking about how conversations went.

s181
00:16:47.622 --> 00:16:59.102
It's very similar to what we talked about here, but the one part that's kind of missing from that is it's very invisible what the agent actually decided or what takeaways that it got from all the conversations.

s182
00:16:59.102 --> 00:17:06.862
It's useful when you're using scoring to have an actual dashboard that explains how all of those conversations went.

s183
00:17:06.862 --> 00:17:15.902
So, I'll kind of draw it out here where if you have like the efficiency over time, it's pretty easy to start aggregating data.

s184
00:17:15.902 --> 00:17:16.902
I've not practiced this live.

s185
00:17:16.902 --> 00:17:17.982
Let's try it.

s186
00:17:17.982 --> 00:17:23.862
Uh where maybe you can actually track over time how the efficiency score has been fluctuating.

s187
00:17:23.862 --> 00:17:34.462
So, the agent's constantly scoring setups and it's giving grades of like 50%, 60% on average, and you can attribute changes that you made in your agent setup to inflections in the score.

s188
00:17:34.462 --> 00:17:42.142
So, not only are you having the agent improve themselves, you also have some proof points of like oh, I shipped this change and I got this result out of it.

s189
00:17:42.142 --> 00:17:48.062
So, I actually know that when I merge that skill that it suggested something real happened on the other side.

s190
00:17:48.062 --> 00:17:49.942
So, you could say oh, there was an inflection point here.

s191
00:17:49.942 --> 00:17:54.342
This is when I merged the skill and I can see that efficiency went up by a certain percentage.

s192
00:17:54.342 --> 00:17:59.782
By having scores in the loop with self-improvement, you can get that sort of view into things.

s193
00:17:59.782 --> 00:18:05.022
And this is the dashboard that we have for proposing those self-improvement PRs.

s194
00:18:05.022 --> 00:18:14.182
So, in Warp Factories, if you set up like your software factory with this, instead of running it one time, the agent is continuously scoring conversations all the time.

s195
00:18:14.182 --> 00:18:19.622
And then on a cron, an agent can look through the scores and suggest PRs to improve your setup.

s196
00:18:19.622 --> 00:18:23.783
So, this is the inbox of things that need our team's review right now.

s197
00:18:23.783 --> 00:18:25.503
This is actually our agent setup.

s198
00:18:25.503 --> 00:18:29.143
So, this is across all of the different agents that back our team.

s199
00:18:29.143 --> 00:18:38.463
And we can take a look at one of them that was merged, where it noticed an issue in the uh triage skill, where we were missing some config to let it spawn sub-agents.

s200
00:18:38.463 --> 00:18:41.663
So, it was continuously like banging its head trying to figure out how do I make sub-agents?

s201
00:18:41.663 --> 00:18:43.223
And it just waits a bunch of tool calls.

s202
00:18:43.223 --> 00:18:47.463
This found it and suggested, "You want to patch that part of your factory config next time.

s203
00:18:47.463 --> 00:18:48.983
That'll save you some tokens."

s204
00:18:48.983 --> 00:18:54.823
Um so, this is one of many examples that we've merged in that just make our agents a little bit more efficient and higher quality.

s205
00:18:54.823 --> 00:18:59.823
Um so, with that, uh if you're interested in any of these ideas, you can check out Warp Factories.

s206
00:18:59.823 --> 00:19:01.503
It's an easy Google search.

s207
00:19:01.503 --> 00:19:09.703
Um or you can search for Warp Skill Doctor to try out that skill I mentioned as just a way to try this out on your own setup and see if you get a little bit of benefit out of it.

s208
00:19:09.703 --> 00:19:14.123
With that, uh thank you so much and enjoy the rest of conference.

s209
00:19:14.123 --> 00:19:16.123
[applause]