Fixing the PR Bottleneck — Matt Pocock, AIHero

AI Engineer · 22 min · 295 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01OK.
00:02I'm here to talk about solving the PR bottleneck, and that's a bit of a big name for trying to fix something that I think most organizations struggle with, and that they've historically struggled with since before AI came along.
00:16We've always had a huge amount of PR that was just sitting around with no one checking it, and now that workload has massively increased due to the new challenges that AI is putting before us , imposing all these strange limitations.
00:28And to do that, I'll use a rubric of my skills that you may have heard of or even used.
00:35I have a couple of new skills I want to announce that I hope will improve your approach to PR, improve the speed at which you review and execute them.
00:46Speed.
00:48Now we are being forced to do more with fewer resources, or essentially, more PR, more work, etc. And that's the main promise of AI: we'll be able to use
00:59these agents to scale and do more work.
01:02This led to the emergence of the “software factory”—perhaps the biggest buzzword today.
01:07Everyone I talk to talks about the software factory.
01:12I think of a software factory as something where instead of a human initiating all of this work, we outsource some of these processes.
01:19Part of the initiation will be performed by agents.
01:23And I think about it this way: Maybe from there you have a classifier like Jeff who says, " Okay, let's turn this into a fix or a replay."
01:56Or maybe I'll notify people right away.
01:58Maybe you have something else.
02:00You may have Planet Scale connected to get reports on slow database queries.
02:05Maybe this triggers another link in your software factory.
02:09All this is not run by people, but by deterministic code, right?
02:13So, they speed up your software factory.
02:16They push more code through it .
02:19But then you need brakes, right?
02:21If you have constant acceleration pushing things through a factory, you get a " garbage cannon," right?
02:27You'll just get a bunch of worthless, terrible PR that you won't even be able to touch, review, or even look at.
02:35So you need brakes.
02:36These are mechanisms that slow down, improve quality, and ensure that your codebase doesn't turn into a nightmare of software entropy, because code is the environment in which your agent runs.
02:48And if you have bad code in your codebase, it will generate even more bad code.
02:54So, in this talk, I'll talk about these three brakes and how we can use them to, paradoxically, move faster.
03:02So , automated checks.
03:03These are deterministic checks in your repository that have been around for thousands of years, well, or since the 50s.
03:11Deterministic checks, where we have linting, tests, type checking, and code quality—all working together and the same way every time.
03:19On top of that, we have an automated review.
03:23So we have agents that look at our code and say, "Okay, this is for things that the tests didn't catch, or this is an analysis of the structure of the codebase in general ."
03:32And then on top of that , the third level, the final level—is human verification.
03:37That is, people who view PR.
03:39And these three levels form a kind of "cake" that we get when we get to human verification.
03:45So higher speed, of course, means more PR.
03:48So the goal here is to make human verification faster by building on these first two steps.
03:54And we are going to stop the negligence.
03:57This is the first principle: if you improve the quality of the code you release, you will spend less time on human review because it is simply a better job, and therefore you will need to do less intervention.
04:09So, automated checks.
04:12Nowadays, checks are cheap.
04:14The cool thing about them is that they don't cost tokens like an automated review, they don't require human effort, they only cost processor time.
04:22For example, you run tests for every code change.
04:25Perhaps these tests require some tokens, because let's say the agent created a bug and the test detected it—then you'll have to spend some tokens to fix it.
04:34But, in my opinion, these are tokens spent quite appropriately.
04:38So, inspections are cheap.
04:39This means you can add a lot of them to your repositories.
04:43And you're probably not using them enough or creatively enough.
04:49But tests can lie.
04:51Does a green CI status mean that the code is ready for merging ?
04:55No, it doesn't mean.
04:56So we always needed an extra layer on top of these checks to find out if there was something catastrophically wrong with the code before we released it .
05:05So all the other stages, human verification and automated verification—they're lie detectors.
05:10They are needed to detect lies in automated checks.
05:13Now I want to show you some examples of such lies, because it helps us , when we think about code and automated testing, to understand how bad it is and what it can lead to.
05:24The first is tautological tests.
05:26A test that simply confirms the implementation.
05:29Opus 5 got hooked on them.
05:31I don't quite understand why.
05:33For example, it would be: the character limit in post X is 280.
05:37Can anyone guess the test written to check for this behavior?
05:40Truth?
05:41You've probably seen this a thousand times.
05:43This is real code from agents or what I found out agents do.
05:47It says: " We expect the character limit for post X to be 280."
05:51So, the implementation looked exactly like this.
05:54And the test essentially just repeated the implementation.
05:56This is the tautological test.
06:00Tautological tests are bad because they are extremely sensitive to structure.
06:05They are very dependent on the internal workings of the system.
06:09This means I can't change the constant without failing the test.
06:14Moreover, I can't rename a constant without failing the test.
06:17This is so closely tied to the structure of my system.
06:21I found another one that I would say is even more blatant.
06:25This is an incredible e-e test.
06:29It essentially checks whether two elements in the interface appear in the correct order.
06:35So, it checks whether the pitch details page , or more specifically, the video section, comes after the content plan.
06:42It doesn't render it to the screen.
06:45It simply reads the file, the module into its own memory, and then finds the right thing.
06:51So, it finds the content plan, finds the video, and then expects the videos to come after it in the source material.
06:58Which is crazy when you think about it, because I can just change the look of the source material and this test will fail.
07:05It is too sensitive to the structure of my codebase.
07:07So this is another way that automated checks can go wrong.
07:11And there's also Oh , sorry.
07:13Automated checks can lie.
07:15There are also tests that literally cannot fail.
07:18This will sound familiar to you if you have abused mochas or used different things before.
07:24For example , here we have the use audio boost function.
07:28And this function use audio boost, its internal logic uses the audio context API in the DOM.
07:34Don't worry if you don't know any of this .
07:36But here we are simply creating " stubs" with dummy methods.
07:40It turns out that the audio context has complex error modes and crashes under strange conditions.
07:47Therefore, such actions mean that our test cannot fail in these modes, and we will encounter strange errors in operation that the tests will not fix.
07:57So the question arises: is it possible to bypass automated checks, even with the best of intentions?
08:03AI doesn't try to write bad tests.
08:07It simply follows our instructions and writes tests that are too tied to the structure, instead of actually running the code.
08:14So how do you make automated checks harder to bypass?
08:18If we do this, we will raise the quality of inspections, and therefore our overall quality standard.
08:24And the first thing I really like here is the code-based design .
08:28You can literally design solutions to avoid these bad checks.
08:33So what does good code-based design look like?
08:36I've talked about this before in previous talks for AI engineers—these are deep modules.
08:40These are modules that hide complex behavior behind simple interfaces.
08:43This is an idea from John Osterhout on software design philosophy .
08:48If we look at these two modules, we see A, where the large implementation is hidden behind a small interface on top.
08:56Of course?
08:57And module B has a large interface, many functions that can be called, but each of them does very little.
09:03Does this make sense?
09:05Yes?
09:06If you have a deep module like A, you will have fewer structure-sensitive tests because you are hiding more of the implementation behind the interface.
09:17If testing only happens through this interface, you will get better tests.
09:21Your task here is to get agents to use this little interface, not to dig into the implementation to test weird details.
09:30I have a skill for this.
09:35You can take the weirdest, worst codebase you've ever seen, apply this skill, and it becomes better.
09:43It essentially gives you opportunities to deepen the modules.
09:47It looks something like this .
09:50Raise your hands if you've used this skill before.
09:52Not sure how many of my people are in this room.
09:54Yes, okay.
09:55This is really really cool.
09:57Essentially, it gives you an HTML document.
09:58I'll step aside so you can see.
10:00With all the different, um, potential possibilities she sees.
10:04So here we see a “before” and “after” state where we reduce duplication and create a beautiful, deep, and tested module.
10:12And then you can safely implement it.
10:15This is also coupled with a language that I developed to describe modules, because if you ever try to learn how to structure a codebase, you'll find 20 different approaches, and they'll all be called
10:28DDD, and you need a consistent language that you can use within a team to discuss these things.
10:33So I have a small set of codebase design skills that define what locality is, what leverage is, and what seam is.
10:41I used joints before it became mainstream.
10:45Locality means how well all the code is placed together.
10:51How can you change one small part in a module without causing a chain reaction?
10:56And leverage is what you get by having a deep module, because the caller of that module gets a lot of benefit from calling a simple function.
11:06Both of these concepts are very useful in codebases.
11:09It turns out they are also useful for agents.
11:12But I'm describing all these high coding standards, but how do we actually get an agent to adhere to them?
11:20How do you make sure that the agent creates deep modules, writes good tests, and doesn't create those worthless tautological or structure-dependent tests?
11:30Well, I think most people do it wrong.
11:34And my first piece of advice: don't implement coding standards directly into implementing agents .
11:42Of course?
11:43Let me explain.
11:45If you imagine that the implementing agent looks something like this, then these are all the tasks that it has to perform within one context window.
11:52He must be able to investigate, that is, look for code that needs to be changed.
11:56Then he has to modify it directly, i.e. update the files marked in green.
12:01And then he needs the time to adjust himself.
12:04So, to run automated checks, to actually check and confirm that everything is working.
12:10As you can see, this is quite a large amount of work.
12:13And if you try to put coding standards enforcement on it , it will perform worse.
12:20So, the implementation is overloaded.
12:22This is the mental model I want you to adopt.
12:25So can we find a way to implement these coding standards in a way that doesn't overload the agent?
12:30Well, this is my attempt to do so.
12:32This is my code review skill.
12:34She gets a diff file.
12:36It reads a file in the repository called " coding standards" which you can write and customize.
12:43Then it checks whether the code meets these standards.
12:47So, if we look at the review agent, he also importantly works as a subagent.
12:52Therefore, it has its own context window for processing tasks.
12:55He has his own budget.
12:57He needs to do some research, right?
13:01Because of course, he gets a diff file, so he knows exactly where the code is, but he should explore the system a little for a broader context to better understand the code.
13:10However, he does not need to deal with implementation.
13:12He doesn't need to do any debugging.
13:15So while the implementation is overloaded, the view is actually underloaded.
13:19So, he doesn't have much work to do.
13:22This means you can add a bunch of coding standards to it , and it will handle it much better than if you tried to do it during implementation.
13:29I think about it this way, and it's a little uncomfortable, right?
13:32Because we all want to get quality code the first time .
13:36But I see it as a two-step process of writing good code: implementation—to make it work, and code review—to make it good quality.
13:45You apply your own coding standards.
13:48And for the “ old school” developers among us, this is essentially a red-green-refactor approach.
13:53We use one context window to make the code work by going through the red-green stage, and then another for refactoring.
14:00At least that's how it works in my head, and it's what has brought me success.
14:03This means that when you have coding standards, you don't add them to the global scope .
14:08You don't put them in agents.md because then they overload your implementation tool.
14:13He may or may not read them .
14:16You place them in coding standards.md, and that way they are only processed by the code review agent .
14:21Now I have another idea: I talked to a lot of people today.
14:26Many say I was talking about review, automated review, and eliminating bottlenecks in the PR process.
14:31So many people say, "Oh, yeah, we just use a third-party service."
14:35We use Cursor Bug Bot.
14:36We use Code Rabbit.
14:38Or something like that.
14:39I think I was really trying to create a universal code review skill that would find all the bugs, perform security audits, etc. before.
14:49It turns out that this is very, very difficult because you either make it too general and get false positives that are irrelevant to your case, or you make it too specific.
14:59You say, "Okay, find everything related to TypeScript."
15:01And then Rust developers can't take advantage of it.
15:04Therefore, I would say: you should not outsource automated testing.
15:07Create your own.
15:08Develop your own coding standards over time.
15:11Share them with your team.
15:12And if you have the ability to implement coding standards, and you have some documents that no one reads, this is exactly the place to put them.
15:21Also , when you have an automated reviewer like this, many people have a natural desire to say, "Oh, yeah, my code review agent reads the code and then leaves comments
15:32on the PR."
15:34So your code review agent in this case is just adding work to the human reviewer.
15:39A human reviewer then has to read all these lengthy comments and decide: do we implement this or do we implement that?
15:46The reviewer must make commits.
15:48He has to make corrections.
15:50That is, he must actually change what he finds.
15:52Because when a person comes, he or she checks a finished, high-quality artifact.
15:57If he finds something that raises questions, he can of course leave a comment, but the default should be a commit.
16:03Stop trying to get the perfect code the first time.
16:06Stop trying to make the executive agent a mere tool and thinking, “Okay, I’ll make it work perfectly.”
16:13It takes some time, you know, to make sense of it, but once you get it, it's just incredible.
16:19So, okay.
16:19With this whole process: we've done our automated checks, now we've done an automated audit to make sure the checks aren't lying.
16:27How do we maximize the quality of PR from a human verification perspective?
16:31How to make it work as best as possible?
16:34We need human-friendly PR.
16:36And this is a new skill that's coming to the repository, it's currently in development, but I'll be releasing it soon—it's the PR skill.
16:45I thought about this for a long, long time.
16:47I still don't fully understand what the modern standard is.
16:50I realized that the best way to master the skill of creating PR is to simply borrow all the best ideas that everyone has.
16:56And that's very good.
16:57So, what does a good PR description look like?
17:01How can you recreate this skill yourself?
17:04Well, the first principle is that some checks are more important than others.
17:09Not every inspection is mandatory, right?
17:12And once you understand that, you realize: okay, this means I can focus my energy on the really important checks.
17:20But how do we classify them?
17:23You need to think about PR using AWS terminology: is it a one-way door or a two-way door?
17:30Most of your PR will be a two-way door.
17:34You can merge a PR and then you can always roll it back.
17:37That's a great advantage of working as a software engineer instead of a civil engineer, right?
17:42Usually when you're a civil engineer, it's always a one-way door, right?
17:45If you make a mistake , the bridge will collapse, but if you have doors on both sides, it means you can easily undo the changes.
17:54This may be a little more complicated than you expect.
17:57It could happen that a very simple change accidentally sends an email to 60,000 people or something like that, in which case it's a one-way door.
18:04You have to check this very, very carefully.
18:06If it involves expensive migrations or data loss, it's a one-way door .
18:10Check such PR as carefully as possible.
18:13But, in addition, we need to understand the radius of this PR.
18:17What could go wrong?
18:18And if things go wrong, how serious is it?
18:22This means that at the end of all my PRs there is a nice short summary—the level of merger risk.
18:28I see that this PR is a two-way door .
18:30Its radius of effect is local, so I see: great, I don't need to pay much attention to it.
18:35I'll just take a look at it.
18:38This is very important.
18:40Next, we need to understand what this PR is actually doing, right?
18:43I've tried many ways to figure this out, and the best I've come up with is using pseudocode.
18:50I want to express my great gratitude for the "show me" skill from the human layer skills repository by Dex Hadley.
18:58This is a phenomenal skill that essentially eliminates most of the text and shows everything through images and diagrams.
19:06This makes it much easier to understand what exactly is changing and why it is happening.
19:13So, you get a standard set of mermaid and UML diagrams to show the sequence of events.
19:20You also get great simple examples like this one.
19:23We can look at this and say, "Okay, we're working in the command line interface.
19:26We see that a new command has been added ."
19:27And we have two new things, these little flags at the top.
19:31Just a brief summary.
19:32This really makes a huge difference.
19:34It is difficult to overestimate its importance.
19:36So, you try to make understanding the reasons as fast as possible.
19:42And I think the third principle here is what you have to think about when you're doing a human verification.
19:49Because we don't do like...Because our processes are so streamlined now, and we all collaborate around these same skill files, these same management files.
20:03We are all building an environment in which our agents will work together.
20:09You should think of the process that creates your code as just as important as the code itself.
20:15In other words, when you do human verification, you are not only checking the code, but also the system that generates it.
20:21And the theory here is that you never want to write the same comment twice.
20:26Is n't that right?
20:27You never want to see an agent doing the same thing in two different merge requests (PRs).
20:31So, what mechanism allows you to make your verification of a person significant?
20:35Well, it's a new skill.
20:37It's called retro.
20:38Retro— from "retrospective."
20:40You are essentially taking the session you had.
20:42It could be either a single agent session, or a PR along with a session, or you could just have her review it: "Okay, look at all the PRs we've done in the last week, all the reviews, put them together."
20:54Let's take a look back at them.
20:56And it will offer automated checks and coding standards to make next time better.
21:03So, it's a cumulative effect, where by doing a human review, you actually increase the quality of the next review and reduce the amount of work for yourself in the future.
21:12And retro is a really smart skill.
21:15She adds a bunch of everything.
21:17Of course, it offers automated checks.
21:19It suggests updating the coding standards.md file.
21:22She does other smart things too.
21:24For example, it analyzes navigation indicators.
21:26How easily did the agent find the information?
21:29Can we provide a hint in agents.md to help him next time?
21:33It analyzes the effectiveness of using tools.
21:35Are there other tools we use in the session that could be more effective with tokens?
21:41It's amazing how much this reveals, because such things are often very difficult to debug from the outside.
21:45This also checks the system for redundancy.
21:47So, are there overloaded control files?
21:49Are there excessive skills that lead to such poor results?
21:53Can we make them more orderly?
21:56So, that's the goal.
21:57This is to make human verification faster.
22:00We achieve this by overlaying automated verification to make human review as painless, simple, and optional as possible.
22:10You really don't need to test every decision that can be changed.
22:13But every irreversible decision is mandatory.
22:16So, these are my skills.
22:18I'm here at hero.dev/skills, and this week we're releasing version 1.3. It was great spending time with you.
22:25It was a really enjoyable conference.
22:26I'll be in the lobby if anyone wants to chat.
22:30Thank you very much for the invitation.
22:32Thank you, Paris.