Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp

AI Engineer · 19 min · 209 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12I'm Ben Holmes and I'm a DevEx engineering lead at Warp and I'm going to be talking about building self-improving agents, agents that look through their past work and figure out how to improve their own setup.
00:24Uh so for a bit of context, I work on Warp Factories, which is open flexible infrastructure to help you all build software factories of your own and crucially ones that improve themselves
00:36as part of the stack.
00:37Uh I'm not going to be talking too much about software factories as a whole today, but I'm going to be talking about one crucial part of it, which is having agents introspect
00:46all of the work that they do inside of one of those systems and suggest improvements so that the agent performs better and better over time.
00:54Uh so I'm going to start by just presenting what you might think of when you hear the word software factory for a bit of definition setting.
01:02It's the idea of automating the software development life cycle.
01:05You may have seen this as an assembly line of different agents doing different things and handing off.
01:10You may have also seen it as a wheel as you see in this case where you start from a task list, a board, Slack messages, wherever you're tracking work
01:19and you have an assembly line of agents go through the process of researching it or triaging, maybe working with a human to write a spec, implementing that change, and bringing computer use into the picture to take screenshots and recordings so that it's actually a good pull request that you get on the other side.
01:35And also maybe doing rounds of adversarial code review in the mix uh to make sure that it's something that is ready for human eyes.
01:43Um and on the other hand you have agents monitor the system and around and around we go.
01:48Uh but the more interesting part I would say is all of the sort of meta-engineering that goes around making that wheel run as efficiently as it can.
01:59Um so a few terms have been maybe floating around right now like meta engineering, gardening, or something that we've been using which is factory engineering.
02:09It's the idea of actually putting in engineering work to improve the system that is driving the agents themselves.
02:16And there are a lot of things that you might want to optimize inside of that system.
02:20It may be the skills that are driving the agents, it may be improving the code architecture and doing static analysis to make sure the agents are working on the right parts of the code base.
02:29I've been hearing a lot about like SonarQube and other sponsors at this event that are doing things in this vein.
02:34The improvement loop that we're going to be talking about is specifically around agent skills and configuration.
02:40And the loop is going to look something like this.
02:42It's the idea that you can have agents review their past conversations.
02:47You can set up some scoring agents to score the quality of how those conversations went.
02:51So, think of this like an LLM as a judge reviewing against rubrics that you define across tricks like efficiency or code quality or test coverage.
03:00And then once you graded a whole lot of conversations across all the work your team is doing, you could find all the failures, figure out reasons that those failures are happening,
03:08and then maybe an agent wakes up on a cron to suggest improvements.
03:12And then you can continue evaluating, seeing how the scores perform, and so on.
03:16So, I'm going to break down that wheel into the individual segments that we can talk about each of them individually.
03:24So, the first part of that loop that I showed you is getting all of your agent conversations in one place so that the agent can actually reason about all the work that your team is doing.
03:35If you're working with you know local agents like Claude code or codex or pie or any of those, those are going to be files on your system.
03:45So, Claude you're going to have a Claude folder world that lives, codex you're going to have a codex folder, and that works well enough if you want to sort of optimize your single player setup.
03:54I've actually talked to teams that set up an S3 bucket and an automation to like take everyone's dot Claude folders and zip them up and put them in a shared S3 instance
04:03just so we have everyone's conversations in one place.
04:06It It's a way of doing things.
04:09Um but there are other ways that you can sort of centralize this information.
04:15Uh if you're using a software factory system, it kind of comes for free.
04:19This is the idea that agents are running inside of maybe a shared computer.
04:23So not on everyone's individual laptops, but like a shared computer that everyone's going to send tasks into.
04:29That might be like a long-running compute instance on Open Claw or it might be like isolated sandboxes that spin up for every run that you do.
04:37Uh if you're using Warp Factory setup, it would be isolated Docker sandboxes for each agent working on tasks.
04:44Using other cloud runners, you'll probably end up in the same sort of system.
04:47But it's all the same idea.
04:49We want to find a way to get everyone's team your whole team's conversations in one place so that they're easy to introspect and figure out how to improve your setup.
04:58And if you're doing it on one cloud runner where all the tasks flow through, you kind of get that for free cuz all the agent traces belong to a single instance.
05:06Um so that's something that's baked in.
05:08And inside of that, you're going to want to make sure you have all the information up front.
05:13So not just the like transcript talking to the agent, what did it send back, what did the human send back.
05:20You also want all of the metadata around it.
05:22So that means the tool calls, the file system calls, the MCPs that it's connecting to, everything that connects the agent to the outside world should also be inside of that agent trace when you start doing this analysis.
05:35And you also want the outputs to be in there.
05:37That can just be the code diffs that live inside of all of your conversations.
05:41It can also be the actual pull request on the other side if you like pin the commit and look at the diff.
05:47In any case, you just want to make sure that you have three things.
05:49You got the transcript of what was said, you've got the tool calls that show what the agent did, and then you have the output which is something that you may want to grade for quality purposes.
06:00Uh and once you have all of that, you can start actually scoring the quality or rather you can have an agent start scoring the quality.
06:07Um so, I like doing a a show of hands for this little part.
06:11Um how many people in here feel like their agents write too many tests?
06:15Too many tests.
06:16Okay, I expected more, but for those at home, it's about like 40% of people, which I think is a pretty big number.
06:24On our team, it was about 100% of hands.
06:26Uh it can depend on your model, and it can depend on your instructions.
06:30But, let's say that your team is noticing your agents are writing a lot of tests.
06:35Maybe extraneous ones, maybe you have your own opinions about do you need end-to-end test coverage versus integration coverage versus unit test coverage.
06:43People are experimenting with all sorts of things there.
06:45So, you might want to set up a scorer for that called redundant test that evaluates how often is the agent writing too many tests across each conversation that it has.
06:55Uh another thing that you may want to score is um the way the agent communicates with you.
07:01So, maybe another show of hands, how many people feel like their agents write too many comments?
07:06Curious about this one.
07:08Yeah, I'm not going to call out the model name, but I I have an idea.
07:12Uh
07:12[laughter]
07:13It it could depend.
07:15Uh but, this is also a common issue with agents, and it's not just for noise or for visuals, because we all know that code review is kind of the biggest bottleneck right now.
07:24It does matter how well the agent communicates changes, both summarizing them, and also in the comments that it leaves within the code itself, because they can be barriers to actually get a code change reviewed quickly.
07:36Um so, you may have another metric like verbosity, which is what we call it, which is is the agent overly verbose when it's talking to you or when it's writing code?
07:44Both of those are examples of kind of the same thing, which is a handle on language.
07:49Um so, you could set up a scorer that sort of defines that.
07:52And a third one I like to bring up is just efficiency of the agent, and this speaks to the need for tool calls and MCP calls in your traces.
08:00Because when you're running things a little bit uh without a human in the loop, so you're sending them off on longer and longer runs, maybe they're running in a cloud sandbox, so you're not looking at the conversation itself.
08:13It becomes easier and easier to miss all of the failed tool calls that an agent does.
08:18Like, an agent these days are very persistent and will work around whatever roadblock that they run into, and they won't tell you about it.
08:24They'll just tell you that it worked on the other side.
08:27And one common failure mode that we ran into was like, in our test harness, there were a ton of bad instructions that were out of date in the skills,
08:34but the agent worked around them and never told us because it figured it out just by trying to ping a bunch of port numbers, figure out its environment, and then eventually start doing code.
08:41So, the most that we knew was like, "Oh, the agent takes like 2 minutes longer than maybe we expect."
08:46But, under the hood, there's actually a lot to be improved.
08:48So, that's something you can evaluate as a score to just say, looking through the tool calls, is it efficiently getting to the solution, or is it spinning its wheels a little bit?
08:57And I can show you what one of these scores looks like.
09:00Uh it's just markdown, as you would probably expect.
09:03Uh in our setup here, it'll kind of be like a specialized skill definition, where you can define what thing you're trying to score with an LLM as a judge approach.
09:13Um so, here we're evaluating efficiency in this score, saying whether the agent worked towards the result or wasted effort on redundant steps and unnecessary back and forth.
09:22And then we set some labels that define what success and failure look like on that criteria.
09:27So, if you remember back in high school writing five-paragraph essays, and the teacher gives you a rubric of like, "You wrote a good essay if you have exactly five paragraphs,
09:36and you only use active voice instead of passive."
09:39That sort of thing.
09:40It is still kind of helpful to put an agent through those paces for things that can be pretty clearly defined, efficiency being one of them.
09:48Um so, we set up one of these rubrics to have that teacher agent go through and decide, do you get good marks on efficiency, or is it kind of below the bar?
09:57And you can decide what pass and fail look like.
09:59In our case, maybe the bound is halfway through this list, so if it's a 0.4 or 0.2, that's kind of a failing grade, something went wrong in that conversation,
10:08and above it, it is a passing grade, and probably doesn't need to be evaluated.
10:12Things went mostly okay.
10:14And then down here is the actual definition of what that rubric means and it's going to restate some stuff that you would expect.
10:20So, cost human, the information gathering phase, routine step overhead.
10:25And you should keep this as succinct as possible and just build it up as you go.
10:29Um this is a template that we like to give to people, but you can set it up with literally just like a couple bullet points and just your rubric and test it out yourself.
10:38It's more of an art than a science.
10:40Um but once you have all these scores set up, then you can start having agents actually grade the conversations.
10:46So, over here, let's say we have uh some sample sessions that we want agents to look through uh starting with like an off service that the agent implemented in a full stack context.
10:56Let's say that we had a full agent conversation about building a full stack implementation of this and the PR was merged and now it's time for an agent to go grade how that conversation went.
11:08Well, the agent will spin up.
11:10In our case, we're saying one agent per conversation or one agent run per conversation run.
11:16You can set it by random sampling for, you know, just the sake of tokens or you can run it across every conversation.
11:22Really depends.
11:24Um but let's say in a random sampling, it pulled out this one and then told an agent to grade across each of those criteria.
11:30So, walking down the line, it might say, "Looking at the test coverage, it wrote a ton of unit test for the off service when our guidelines say that you should mostly prefer end-to-end test."
11:40And so, maybe we give it a fail because it didn't really follow the instructions that we expected.
11:45And then maybe for verbosity, it was actually pretty clear.
11:48It was using a new model selection that doesn't write code comments.
11:52Uh go GPT, anyone anyone in the room?
11:54I don't know.
11:55Uh but let's say that it didn't really write too many comments.
11:59It was very easy to understand the PR, so it gets a pass.
12:02And then moving on to efficiency, let's say it was trying to do a lot of exploration in its test environment to figure out how to test this change in the first place
12:10when really it should have had some upfront instruction on that.
12:12And that would probably be a failing grade on that score.
12:16And again, pass and fail is just did it fall below the line on the rubric that we set.
12:21Uh and then you can just go through each of your agent conversations and continue scoring them.
12:26So, automatically, usually the trigger being a PR merged, kick in an agent to review that conversation across your grading criteria.
12:34And then over time, you'll start aggregating a lot of data on how conversations go.
12:39And each of these will include like the pass and failure rates.
12:43So, did it pass and fail across each definition?
12:46And then you also have an agent describe why one of those failures or passes happened.
12:51And then this becomes context that another agent can use to decide what improvements to make.
12:56So, the way that we set it up, going back to this wheel, is the idea you grade conversations as soon as they happen.
13:02And then maybe on a cron job, every 24 hours, go through a sampling of the past 24 hours of conversations.
13:09Maybe there's been like 200.
13:11Filter it down to the failures.
13:12And then pass in all of those failing reports that describe what failed and why to a self-improvement agent.
13:19And tell it these were the 50 conversations that failed.
13:22Here are the notes that were left on each conversation.
13:24Go ahead and find commonalities.
13:26Here's access to our skills repository and the code base if you want to make architecture changes.
13:31Um so, in this little snippet I showed, maybe it adds a coverage pyramid because redundant tests are popping up across a lot of PRs.
13:38And I can show you a couple other examples.
13:40Um both in a skill that you can take home, and also the automated system that we have inside of Warp Factories.
13:47So, this is going to be the easiest one that y'all get to try.
13:50Um so, it's called Skill Doctor.
13:52It's something you can find at warp.dev/skilldoctor.
13:55I'll show the link in a second.
13:56Um but this is a skill that you can run on your single-player setup, and it'll crawl through all of your agent conversations and Claude, Codex Buyer, whatever else, and just do a one-time grading.
14:07So, if you've been wondering about all the hidden gems in your conversations cuz you haven't done this approach before, you can just run this across your setup and see what happens.
14:14So, this was a sample I ran across some code sessions building this project here, and we can see on the side that it grades in batches for a little bit of token efficiency,
14:24and it came up with some score reports across all of the metrics that we care about.
14:29And this test environment has efficiency and code quality as the ones that it evaluates.
14:33Uh and then on the other side, the self-improvement agent looks at all those scores and comes back with a report.
14:40Um so, let's go ahead and take a look at that.
14:42Uh I like to show the animation.
14:45We love it.
14:46Uh but this is like kind of your skill report card that shows how the agent performed across all of those rubrics that we defined.
14:52Again, this is one shot.
14:53It's probably something that you want to do continuously on your setup rather than whenever someone remembers.
14:58But as a first go to just see what you can optimize right now, it's nice to get some low-hanging fruit.
15:03Um so, in this example, it graded across these metrics and noticed that these were a little bit lower than expected, but verbosity was fine.
15:10Um so, it went through all of the notes that were given by the scoring agents, and then suggested some changes to the skill makeup.
15:17And this one is going to mirror what I was mentioning earlier, uh where it noticed that the agent was usually trying to fight for port numbers on the machine that it was testing,
15:25and it didn't really understand how the environment was set up.
15:28So, this proposed a change to the end-to-end testing skill to have the agent get better instructions on finding a port number first and then going for it instead of just banging against the wall until it got something running.
15:38Um so, that can save a little bit of efficiency uh in the tool calls.
15:42It also noticed that I constantly wanted to write a different changelog report with the language that was specified, and so it rewrote my changelog skill to have a more
15:51uh prescriptive language on how these things are written.
15:53So, I don't have to do that manual back and forth every time an agent opens a PR.
15:58Um and these were things that I could have gone and found myself, but especially this one on port numbers, since I'm not watching the traces super closely, it's usually just behind like a working toggle if you're using
16:08like an agent harness or it's behind the cloud, uh it's nice to pick up on things like this so you don't miss them.
16:14Um so, last thing that I'll leave you with is how you can run this continuously over time.
16:19Uh oh, shoot, it was just there.
16:21No, I lost access to my staging.
16:25Uh well, I can at least describe with words what that was.
16:29Uh which is the set of sort of graphs on the scores that are written.
16:34Because one thing I was actually explaining this to someone who uses like a memory system in their harness.
16:39Uh and memories are a really useful way of having agents like introspect.
16:43Maybe you've heard of like dreaming overnight and thinking about how conversations went.
16:47It's very similar to what we talked about here, but the one part that's kind of missing from that is it's very invisible what the agent actually decided or what takeaways that it got from all the conversations.
16:59It's useful when you're using scoring to have an actual dashboard that explains how all of those conversations went.
17:06So, I'll kind of draw it out here where if you have like the efficiency over time, it's pretty easy to start aggregating data.
17:15I've not practiced this live.
17:16Let's try it.
17:17Uh where maybe you can actually track over time how the efficiency score has been fluctuating.
17:23So, the agent's constantly scoring setups and it's giving grades of like 50%, 60% on average, and you can attribute changes that you made in your agent setup to inflections in the score.
17:34So, not only are you having the agent improve themselves, you also have some proof points of like oh, I shipped this change and I got this result out of it.
17:42So, I actually know that when I merge that skill that it suggested something real happened on the other side.
17:48So, you could say oh, there was an inflection point here.
17:49This is when I merged the skill and I can see that efficiency went up by a certain percentage.
17:54By having scores in the loop with self-improvement, you can get that sort of view into things.
17:59And this is the dashboard that we have for proposing those self-improvement PRs.
18:05So, in Warp Factories, if you set up like your software factory with this, instead of running it one time, the agent is continuously scoring conversations all the time.
18:14And then on a cron, an agent can look through the scores and suggest PRs to improve your setup.
18:19So, this is the inbox of things that need our team's review right now.
18:23This is actually our agent setup.
18:25So, this is across all of the different agents that back our team.
18:29And we can take a look at one of them that was merged, where it noticed an issue in the uh triage skill, where we were missing some config to let it spawn sub-agents.
18:38So, it was continuously like banging its head trying to figure out how do I make sub-agents?
18:41And it just waits a bunch of tool calls.
18:43This found it and suggested, "You want to patch that part of your factory config next time.
18:47That'll save you some tokens."
18:48Um so, this is one of many examples that we've merged in that just make our agents a little bit more efficient and higher quality.
18:54Um so, with that, uh if you're interested in any of these ideas, you can check out Warp Factories.
18:59It's an easy Google search.
19:01Um or you can search for Warp Skill Doctor to try out that skill I mentioned as just a way to try this out on your own setup and see if you get a little bit of benefit out of it.
19:09With that, uh thank you so much and enjoy the rest of conference.
19:14[applause]