How to Build an AI Code Review Agent - Daksh Gupta, Greptile

AI Engineer · 18 min · 229 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Um all right, my name is Dax.
00:14I am one of the co-founders of a company called Griptile and we build agents that review and test pull requests with the full context of the code base.
00:21And by reviewing and testing these pull requests, we reveal bugs, logical errors, code quality concerns, primarily from the AI-generated code.
00:32Now, to give you a sense of scale, many thousands of companies use Griptile every day.
00:37About 1% of all pull requests on GitHub go through us and a variety of many different types of code bases, whether it's the CUDA code base at Nvidia or the wallet code base at Coinbase and many, many others.
00:49And every now and then, we meet a company where we're actually not the right code review or code testing product for them.
00:56Their needs are so specialized and so esoteric that it's actually beneficial for them to build their own.
01:02And when that happens, we tend to help them for free.
01:04There's two reasons for this.
01:05One, we have a lot of specialized knowledge about how to do this really, really well.
01:09And second, we get to learn a lot about how we can serve customers like those in the future.
01:14And in this presentation, I want to tell you what we tell them, which is how to build a really great AI code review agent yourself.
01:22Now, there's kind of three things that people like about Griptile and I'm going to talk through how to implement those yourself.
01:28The first one is code base context.
01:30When customers use Griptile on their pull requests, they find that it really understands their code base.
01:35As a matter of fact, before we were a code validation company, we were a code base context company.
01:40We were teaching language models from 2023 how to understand the very large code bases and as a result of that, we got very good at indexing code.
01:47Uh the second thing that people like about Griptile is that it runs your code in the sense that spins up your stack in a fully loaded dependency-rich sandbox and it clicks around and tests things using computer user
01:57through pressure testing API endpoints and so on.
02:00And the third thing is we're actually very efficient allocators of inference.
02:03We pick the right model, the right quantity of inference budget for each subtask in the review and validation process.
02:09As a result of that, we're able to get a lot more performance per dollar than an internal tool could or any competitor could.
02:16So, there's kind of two reasons that you would want this presentation.
02:20Uh the first one is you want to build your own review agent.
02:24And the second reason, which is a bit more subtle, is if you're trying to pick a review or testing agent for your company and you want to understand what makes a good one and what makes a great one.
02:34All right.
02:35So, let's start right in.
02:36Now, blank slate.
02:38You are part of the platform team at your company or you're just the uh local uh AI psychotic member of your team and you want to build the review validator
02:46of your company.
02:48Um Companies are writing lots of code.
02:50People are vibe coding.
02:52Uh you have, of course, engineers vibe coding.
02:54Some of them are a little too close to the sun.
02:56Uh you have marketing people writing code.
02:58And uh the new problem is the review bottleneck, the validation bottleneck.
03:02And so, that's what you want to address.
03:04The first thing that you should do is you should get yourself a sandbox.
03:07There's many, many companies here that will sell you a sandbox.
03:09Uh you can also just use firecracker or something like that.
03:12And you'll want to put an agent in that sandbox.
03:14Uh the agent can be Code X or Cloud Code.
03:17Personally, I recommend going model independent for reasons that I will get into, so you can use something like Open Code or Pi as your harness.
03:23Those are both open source.
03:24And uh you will need to then connect your GitHub or your GitLab.
03:28Have a PR webhook send it the contents of a pull request to any pull requests that are opened.
03:32The agent can have some type of prompt that just says, "Hey, review this and send back some feedback."
03:37And then you can wire that back using GitHub's review APIs.
03:39Very simple.
03:41In fact, many people think that's actually all it takes to build a review agent.
03:44There's a few things that are going to go wrong when you start using this.
03:46But what will happen is you will, in fact, have a pull request go live.
03:51The contents will go to an agent and about 5 minutes later you will get some feedback on your pull request.
03:55But here's what's going to go wrong in the time that it takes you to build this.
04:00The first one is you're going to miss a lot of bugs.
04:01As it turns out that actually bugs are interactively large.
04:03They're In fact, I saw a a tweet not very long ago that said that bugs are fractally infinite.
04:10So you will never run out of a of bugs in a particular piece of code.
04:13That's the first thing that's going to go wrong is you're going to miss bugs.
04:15Of course, that's the one thing that your system needs to do is to prevent at least very catastrophic outages and incidents.
04:21Second thing that's going to happen is it's going to miss context and it's it's going to misunderstand parts of how your code base works.
04:27It maybe won't understand where you're going with your code base.
04:29It won't understand how a specific part of it works.
04:31It maybe will not understand how one of your libraries works.
04:34It's possible you're one of those companies that never recovered from the microservices apocalypse of the 2010s and so you just have thousands and thousands of repos and you have to have context of all of them to be able to operate.
04:43So that's the second thing that's going to go wrong.
04:45The third thing is it'll be very expensive.
04:47I think people are very surprised by this.
04:49So generally from people who have seen try to do this, they'll find that they're spending between $3 and up to $25 per pull request on reviewing it using coding agents.
04:58These tokens can add up really really fast.
05:00Very quietly they got very expensive.
05:04The fourth problem they'll find is nitpicky comments.
05:06Now this matters more than you might think.
05:08Nitpicky comments are comments that are technically correct but overly pedantic.
05:11There's two things that go wrong when you have nitpicky comments.
05:13If you are a as a human reading those comments, they're very annoying.
05:16They're like a boy that cries wolf.
05:17You'll just see lots and lots of comments and it'll be kind of annoying for you to just read through them and figure out which ones are just minor stylistic complaints and which ones are actually catastrophic problems.
05:26The second thing that will go wrong is they will wreck your code base because typically the way the review agents work is they'll leave comments on GitHub and your coding agent picks them up and resolves them.
05:35And coding agents are not discerning.
05:36They're going to look at review comments and just address all of them and you'll end up with extremely painfully defensive code as we found the coding agents are particularly
05:44astute at writing.
05:47The next thing that go wrong is that every time you run a review agent on the same pull request, you will somehow find new bugs in the sense that it will keep catching new things
05:55in the pull request very annoyingly, and you will And like I said, bugs will appear to be fractally infinite.
06:02So, the first thing that you want to do, the lowest hanging fruit, what is you want to optimize the prompt.
06:07You started with a very simple prompt, something like review this PR and then output the comments in GitHub's API schema format.
06:15Generally, there's two themes within the bugs.
06:17Look for specific bug types, and second, skip the nitpicks.
06:20And that's what you want to orient your prompt around.
06:22You'll want to ask for specific bug types, and you can kind of do some internal data science on existing outages and incidents that have happened in your code base and figure out what types of bugs you have the highest propensity to produce.
06:33Create a catalog of them, and then have your prompt look for those specific bug types.
06:37The second is you'll need to expressly tell it what a nitpick is and not to look for them.
06:41And nitpicks can be anything from asking why there isn't three layers of defense on this particular thing, or why error handling is done in this slightly shoddy way.
06:51Now, the thing that you'll run into when you change these prompts is when you change a small thing in the prompt, it'll have a weirdly big impact on the number of comments that your agent is writing on your pull request.
07:01This is quite scary because it means you don't really know if it's missing a bug or not.
07:04You don't know the counterfactual.
07:06The validation agents, or really any reviewer or tester, can can tell you about the presence of bugs, but it fortunately can't tell you about the absence of bugs.
07:13That's that's sort of a a limitation of the format.
07:16And what makes this particularly difficult are bugs that are extraordinarily rare and getting rarer by the day.
07:21In fact, according to our data science, only about one out of every thousand pull requests has a generally catastrophic bug in it.
07:27Or maybe one out of every thousand lines of code has a catastrophic bug in it.
07:30And so, because these bugs are very rare, you might release a version of your agent that you think is very good, but it secretly misses a catastrophic bug in a pull request, and you wouldn't know for a long time because you don't produce those bugs very often.
07:44And so that brings us to the second improvement.
07:46You will want a data set of historical PRs.
07:49This is relatively easy if you're one of those companies that very nicely catalogs every time something goes wrong in your code base.
07:54If you have that, then you'll have a list of pull requests that have caused outages and incidents in the past.
07:58You'll want to catalog them and then essentially have any version of your review agent run through those and see with what frequency it can catch those bugs.
08:07Now, this will allow you to tune the prompt a little bit more confidently.
08:10But at the same time, the interesting thing about bugs is in a changing world, the topology and ontology of bugs changes over time.
08:17And so you'll need to have a real-time updating data set.
08:21And this is tricky because unless you have millions and millions of pull requests, it's actually hard to have reasonably high coverage of bug types.
08:27It's true that about 80 or 90% of your bugs will be just repeats of prior mistakes, but increasingly as code agents get good, they're going to start making novel mistakes.
08:34They'll fail in new ways that we've not anticipated before.
08:37And so this is where you will start to hit limitations.
08:39Now, quick side note, this is kind of what we do at Reptile, except we actually have a lot of code from a lot of different companies and we can test across a very, very large corpus of pull requests and therefore improve our agent very confidently.
08:50It's one of the reasons why we're able to eke out little bits of performance on the margins in ways that internal review agents can't.
08:56But you can get decently far, especially for your particular company, to do this with your internal data.
09:02The next thing you want to do is model inversion.
09:04So, for in case anyone was at my talk earlier this year in San Francisco at AI Engineer, I talked a lot about the ways in which coding agents fail.
09:13And I discovered an interesting pattern through that research, which is that the types of bugs produced by Claude models or Codex models are actually quite different from each other.
09:20And they're different in a number of ways.
09:23As an example, Claude models are something like three times more likely to produce one-off errors than GPT models.
09:29GPT models, on the other hand, are significantly more likely to produce auth bypasses than Claude models.
09:33And these are all just artifacts of the post-training, as far as I understand.
09:37Now, to prevent this, you'll find that if you're using the same model to review the code written by the same model, um you'll actually have relatively fewer bugs caught because they fail in the same ways during review that they do during code generation.
09:50And so you'll want to invert the model.
09:51If you've used Claude code to write the code, you'll want to use a GPT model to review it and so on.
09:56Now, doing this is somewhat straightforward for the 80th percentile and quite tricky for the last 20.
10:04So, if you're using coding agents, you'll notice they leave some type of fingerprint to tell you what coding agent was used for a particular pull request.
10:09For instance, Codex will name branches in a codex/branch name format.
10:14Or Claude is going to write a co-authored by Claude in the PR description and commit message.
10:18And about 80% of the time, this will be true.
10:20In the final 20, when you happen to manually create a branch name or there was some other reason that a different coding agent was perhaps used, it's significantly harder to figure out which model was used during generation.
10:28And you have to develop some type of heuristics to figure out what are the signatures of models being used.
10:32Often you might have an internal catalog where people report what coding agents are using these days or have some type of telemetry that's coming out of your dev machines and and your engineers' computers.
10:43We found that there's a 10% higher catch rate if you flip models.
10:46And this might sound very small, but you have to know that 10% is in terms of incidents and outages is actually quite a lot.
10:52It means that there's a 10% higher chance that something will go wrong if you're not using model inversion, which can be quite catastrophic.
10:58And so we highly recommend that you use some type of model inversion in your process.
11:04The third improvement is code base context.
11:06Now, this is something that I care a lot about.
11:08Before we were doing code validation, we were doing code base context.
11:12Uh we were trying to figure out better ways to do rag over very large code bases.
11:15And even today, people consider grep to effectively solved all of code base context.
11:20I actually think that's not true for a number of reasons.
11:21One, grep is extremely imprecise.
11:24And second, it's very expensive.
11:26To give you some sense of scale, single-digit percentage points of GDP growth last year were inference.
11:30A very large percentage of inference was coding agents.
11:33And if you actually look at a coding agent trace, the vast majority of tokens used by coding agents are all the same.
11:39grep, file read, cache write for the file contents.
11:42That means a non-trivial percentage of GDP growth is grep, file read, cache write.
11:47That's very bad.
11:47And this is because grep is very imprecise.
11:49It's reading files in very large quantities, and I don't know, there's perhaps some adverse incentives for companies paid by the token to figure out ways to burn more tokens.
11:56But even so, you'll find that you'll need to use slightly more surgical methods of figuring out where you should be reading.
12:02And the way we do this is we produce an internal knowledge base where we essentially check across the entire code base all the different parts, catalog them, and then index saying, "Okay, well, this is authentication.
12:12Here's how it works.
12:12Here are the flows.
12:13Here's a diagram of the flows.
12:14Here's every file that's involved."
12:16One example of where code base context matters is, for instance, when you're trying to catch an off bypass.
12:20It's actually very difficult to figure out if an off bypass has occurred because you need to have a pretty concrete understanding of whether you're in an authenticated environment when you're in that part of the code base or not.
12:28And that's very tricky.
12:29And so, you'll find that you will need some type of uh of sort of code base context to be able to do this.
12:36Now, I've been very centered around the concept of bugs, but code can be very, very bad even if it's not buggy.
12:43For one, it can be terrible quality, but second, it can just not fulfill the expressed intent of the author.
12:48Recently, I saw a post that said that people were increasingly using Slack as their IDE.
12:53Now, I think that's a little funny, but at the same time, there's not that much ex- expressed intent in a Slack message.
12:58You're only saying maybe a sentence or two, if even, asking an agent to go and fix something, which means the there's a lot of ambiguity in your instruction.
13:06As a result of this, you might find that the coding agent has either misinterpreted your instructions or just it was too ambiguous and it didn't know exactly what to do.
13:13And so, your correct code can actually still be quite wrong.
13:16Now, one of the ways in which we fight this is by giving agents increasingly more context.
13:20And we find that the two types of context that matter most are docs and tickets, and then of course, the code base from your code base indexer or knowledge base.
13:29You can get remarkably far if you give it access to your Jira and Linear, and then your Confluence or Notion or whatever it is that you store your docs.
13:35You'll find that there's a lot of tribal knowledge that is subconsciously helping you produce better code.
13:40And when you have agents do the same thing, it feels like they just don't know what's happening.
13:43And because you you underestimate how much depth of knowledge you yourself have.
13:51Now, the way that Reptile approaches this problem is we actually disintermediate all of these context sources with a living wiki.
13:58This wiki is essentially a sort of world model of how your company works.
14:02It's everything that it knows about your code base, everything it knows about your application through production telemetry, through logs, everything it knows about your road map from your documentation
14:11and your linear tickets and so on and so forth.
14:14And it uses this wiki to supply context both making the agent more context aware and making its code searches significantly more precise because it will never read a file that it didn't need to read.
14:26The last thing that you should do, or this might be four out of five.
14:29The next thing that you should do is autonomous testing.
14:31[clears throat]
14:31One of the things that we think about a lot is that for instance, for UI changes, you might be able to read through a bunch of changed files for React code and conclude that there's nothing wrong with the code.
14:43But in fact, when you render it, it might be, for instance, hidden on an iPhone.
14:47Now, this is one of the more technically difficult problems to solve.
14:51The way that we approach this at Reptile is we essentially build these dependency-rich sandboxes, which are completely configurable virtual machines, and we have customers, we work with them to essentially spin up their stack in these sandboxes.
15:05And then Reptile's agent uses computer vision to test them end to end.
15:09If you haven't been paying attention, computer vision has gotten very good, and it's like far better in these new generation of GPT models and Opus models than it was in the previous.
15:16For the first time, I think end to end QA is nearly solved.
15:19The thing that you need to do though is you need to document all the flows through your application.
15:22And second, you need to be able to have a sufficiently simple build script so that the agent can spin up your stack in your environments.
15:30We have a team of forward deployed engineers that help companies do this.
15:33We found that it's one of the trickier parts.
15:34Unless of course, you have very good dev boxes already, in which case congratulations on working at Stripe.
15:43So, oh, I This is the diagram I'm going to show you.
15:45But yes, you will need a dev box, you will need PR branch dependencies where you're running the app, and you'll need some type of flow map that says, "This is the auth code and then this is the sign-up flow."
15:53and so on.
15:54And using this, your agent can now test your code end-to-end and use vision models to say, "Oh, like this button technically appears, but it actually is hidden on iPhones, or it's not in the first fold on
16:03the iPhone duo, etc." Now, if you do all of this, you'll find that the costs are still quite high.
16:08It can cost you anywhere from three bucks to 15 to 20 to do a single pull request review.
16:14And the thing you'll want to solve for now is to use cheaper models for some things.
16:18Now, to me, frontier models are still generally speaking more capable than open source models.
16:23This has been true for some time, even though the rubber band is tightening.
16:26So, our share today is about 20% frontier models for the most critical tasks.
16:31And then the long tail of tokens, which is exploratory, file reads, and so on, we use a variety of smaller models.
16:37And so, if you do this, you can actually reduce the cost of running your review agent quite drastically.
16:43So, this is sort of a illustrative example of what we do, which is we'll use a small model for our finding context, we'll use a very lightweight computer use model for testing user flows,
16:53and we use a frontier model for things like analyzing bugs, applying the review process, and first submission.
16:57In practice, we have actually 20 different subtasks, and each of them has a very specific set of models that we use for each of them.
17:03And we have a small research team that is constantly iterating exactly what models we use for what task.
17:07You can probably get 70 to 80% of the way there using something of this kind where you just swap out very simple tasks with very simple models.
17:15Now, this is another example of where it really helps to have a very large data set.
17:18If you don't have a good data set, you might be able to have trigger happily replace a lot of different sub agents with smaller, faster models, and you won't know that it's this agent is actually now quite a bit worse at catching bugs, and you would only find out when it misses something that then causes an outage
17:31or incident.
17:34And you will also find that you can kind of combat this problem of overactive file reads using a cheap model that essentially looks through all of your context pieces and figures out which ones actually should be included.
17:45In theory, you should be able to use a decision model for this, something like Java, or the Decisions API, to do effective re-ranking on all of the different things, and deciding what should be and shouldn't be loaded into Frontier Model Cache, which is in some ways your most expensive and most scarce resource.
17:58If you do all of these things, you might be able to actually get quite far in building your own review agent.
18:03And if you do, please let me know.
18:06My email is daksh@prod.com.
18:08I'd love to help you.
18:09And uh if you choose not to build it, then you can also just use our product, um which is free in comparison.
18:16But, uh thank you so
18:17[snorts]
18:17much.