WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=4UtrRVtUN_g

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/4UtrRVtUN_g.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.600 --> 00:00:14.000
Um all right, my name is Dax.

s3
00:00:14.000 --> 00:00:21.840
I am one of the co-founders of a company called Griptile and we build agents that review and test pull requests with the full context of the code base.

s4
00:00:21.840 --> 00:00:32.520
And by reviewing and testing these pull requests, we reveal bugs, logical errors, code quality concerns, primarily from the AI-generated code.

s5
00:00:32.520 --> 00:00:37.640
Now, to give you a sense of scale, many thousands of companies use Griptile every day.

s6
00:00:37.640 --> 00:00:49.960
About 1% of all pull requests on GitHub go through us and a variety of many different types of code bases, whether it's the CUDA code base at Nvidia or the wallet code base at Coinbase and many, many others.

s7
00:00:49.960 --> 00:00:56.920
And every now and then, we meet a company where we're actually not the right code review or code testing product for them.

s8
00:00:56.920 --> 00:01:02.240
Their needs are so specialized and so esoteric that it's actually beneficial for them to build their own.

s9
00:01:02.240 --> 00:01:04.800
And when that happens, we tend to help them for free.

s10
00:01:04.800 --> 00:01:05.720
There's two reasons for this.

s11
00:01:05.720 --> 00:01:09.680
One, we have a lot of specialized knowledge about how to do this really, really well.

s12
00:01:09.680 --> 00:01:14.840
And second, we get to learn a lot about how we can serve customers like those in the future.

s13
00:01:14.840 --> 00:01:22.360
And in this presentation, I want to tell you what we tell them, which is how to build a really great AI code review agent yourself.

s14
00:01:22.360 --> 00:01:28.640
Now, there's kind of three things that people like about Griptile and I'm going to talk through how to implement those yourself.

s15
00:01:28.640 --> 00:01:30.040
The first one is code base context.

s16
00:01:30.040 --> 00:01:35.040
When customers use Griptile on their pull requests, they find that it really understands their code base.

s17
00:01:35.040 --> 00:01:40.000
As a matter of fact, before we were a code validation company, we were a code base context company.

s18
00:01:40.000 --> 00:01:47.600
We were teaching language models from 2023 how to understand the very large code bases and as a result of that, we got very good at indexing code.

s19
00:01:47.600 --> 00:01:57.400
Uh the second thing that people like about Griptile is that it runs your code in the sense that spins up your stack in a fully loaded dependency-rich sandbox and it clicks around and tests things using computer user

s20
00:01:57.400 --> 00:02:00.080
through pressure testing API endpoints and so on.

s21
00:02:00.080 --> 00:02:03.240
And the third thing is we're actually very efficient allocators of inference.

s22
00:02:03.240 --> 00:02:09.320
We pick the right model, the right quantity of inference budget for each subtask in the review and validation process.

s23
00:02:09.320 --> 00:02:16.080
As a result of that, we're able to get a lot more performance per dollar than an internal tool could or any competitor could.

s24
00:02:16.080 --> 00:02:20.880
So, there's kind of two reasons that you would want this presentation.

s25
00:02:20.880 --> 00:02:24.160
Uh the first one is you want to build your own review agent.

s26
00:02:24.160 --> 00:02:34.480
And the second reason, which is a bit more subtle, is if you're trying to pick a review or testing agent for your company and you want to understand what makes a good one and what makes a great one.

s27
00:02:34.520 --> 00:02:35.000
All right.

s28
00:02:35.000 --> 00:02:36.160
So, let's start right in.

s29
00:02:36.160 --> 00:02:38.280
Now, blank slate.

s30
00:02:38.280 --> 00:02:46.959
You are part of the platform team at your company or you're just the uh local uh AI psychotic member of your team and you want to build the review validator

s31
00:02:46.959 --> 00:02:48.520
of your company.

s32
00:02:48.520 --> 00:02:50.280
Um Companies are writing lots of code.

s33
00:02:50.280 --> 00:02:52.120
People are vibe coding.

s34
00:02:52.120 --> 00:02:54.720
Uh you have, of course, engineers vibe coding.

s35
00:02:54.720 --> 00:02:56.840
Some of them are a little too close to the sun.

s36
00:02:56.840 --> 00:02:58.680
Uh you have marketing people writing code.

s37
00:02:58.680 --> 00:03:02.280
And uh the new problem is the review bottleneck, the validation bottleneck.

s38
00:03:02.280 --> 00:03:04.239
And so, that's what you want to address.

s39
00:03:04.239 --> 00:03:07.320
The first thing that you should do is you should get yourself a sandbox.

s40
00:03:07.320 --> 00:03:09.840
There's many, many companies here that will sell you a sandbox.

s41
00:03:09.840 --> 00:03:12.280
Uh you can also just use firecracker or something like that.

s42
00:03:12.280 --> 00:03:14.959
And you'll want to put an agent in that sandbox.

s43
00:03:14.959 --> 00:03:17.959
Uh the agent can be Code X or Cloud Code.

s44
00:03:17.959 --> 00:03:23.440
Personally, I recommend going model independent for reasons that I will get into, so you can use something like Open Code or Pi as your harness.

s45
00:03:23.440 --> 00:03:24.880
Those are both open source.

s46
00:03:24.880 --> 00:03:28.760
And uh you will need to then connect your GitHub or your GitLab.

s47
00:03:28.760 --> 00:03:32.840
Have a PR webhook send it the contents of a pull request to any pull requests that are opened.

s48
00:03:32.840 --> 00:03:37.080
The agent can have some type of prompt that just says, "Hey, review this and send back some feedback."

s49
00:03:37.080 --> 00:03:39.880
And then you can wire that back using GitHub's review APIs.

s50
00:03:39.880 --> 00:03:41.080
Very simple.

s51
00:03:41.080 --> 00:03:44.160
In fact, many people think that's actually all it takes to build a review agent.

s52
00:03:44.160 --> 00:03:46.480
There's a few things that are going to go wrong when you start using this.

s53
00:03:46.480 --> 00:03:51.080
But what will happen is you will, in fact, have a pull request go live.

s54
00:03:51.080 --> 00:03:55.520
The contents will go to an agent and about 5 minutes later you will get some feedback on your pull request.

s55
00:03:55.520 --> 00:04:00.240
But here's what's going to go wrong in the time that it takes you to build this.

s56
00:04:00.240 --> 00:04:01.760
The first one is you're going to miss a lot of bugs.

s57
00:04:01.760 --> 00:04:03.880
As it turns out that actually bugs are interactively large.

s58
00:04:03.880 --> 00:04:10.040
They're In fact, I saw a a tweet not very long ago that said that bugs are fractally infinite.

s59
00:04:10.040 --> 00:04:13.040
So you will never run out of a of bugs in a particular piece of code.

s60
00:04:13.040 --> 00:04:15.160
That's the first thing that's going to go wrong is you're going to miss bugs.

s61
00:04:15.160 --> 00:04:21.680
Of course, that's the one thing that your system needs to do is to prevent at least very catastrophic outages and incidents.

s62
00:04:21.680 --> 00:04:27.280
Second thing that's going to happen is it's going to miss context and it's it's going to misunderstand parts of how your code base works.

s63
00:04:27.280 --> 00:04:29.440
It maybe won't understand where you're going with your code base.

s64
00:04:29.440 --> 00:04:31.160
It won't understand how a specific part of it works.

s65
00:04:31.160 --> 00:04:34.280
It maybe will not understand how one of your libraries works.

s66
00:04:34.280 --> 00:04:43.400
It's possible you're one of those companies that never recovered from the microservices apocalypse of the 2010s and so you just have thousands and thousands of repos and you have to have context of all of them to be able to operate.

s67
00:04:43.400 --> 00:04:45.960
So that's the second thing that's going to go wrong.

s68
00:04:45.960 --> 00:04:47.840
The third thing is it'll be very expensive.

s69
00:04:47.840 --> 00:04:49.560
I think people are very surprised by this.

s70
00:04:49.560 --> 00:04:58.560
So generally from people who have seen try to do this, they'll find that they're spending between $3 and up to $25 per pull request on reviewing it using coding agents.

s71
00:04:58.560 --> 00:05:00.320
These tokens can add up really really fast.

s72
00:05:00.320 --> 00:05:03.400
Very quietly they got very expensive.

s73
00:05:04.200 --> 00:05:06.720
The fourth problem they'll find is nitpicky comments.

s74
00:05:06.720 --> 00:05:08.000
Now this matters more than you might think.

s75
00:05:08.000 --> 00:05:11.520
Nitpicky comments are comments that are technically correct but overly pedantic.

s76
00:05:11.520 --> 00:05:13.960
There's two things that go wrong when you have nitpicky comments.

s77
00:05:13.960 --> 00:05:16.440
If you are a as a human reading those comments, they're very annoying.

s78
00:05:16.440 --> 00:05:17.600
They're like a boy that cries wolf.

s79
00:05:17.600 --> 00:05:26.520
You'll just see lots and lots of comments and it'll be kind of annoying for you to just read through them and figure out which ones are just minor stylistic complaints and which ones are actually catastrophic problems.

s80
00:05:26.520 --> 00:05:35.320
The second thing that will go wrong is they will wreck your code base because typically the way the review agents work is they'll leave comments on GitHub and your coding agent picks them up and resolves them.

s81
00:05:35.320 --> 00:05:36.400
And coding agents are not discerning.

s82
00:05:36.400 --> 00:05:44.840
They're going to look at review comments and just address all of them and you'll end up with extremely painfully defensive code as we found the coding agents are particularly

s83
00:05:44.840 --> 00:05:47.280
astute at writing.

s84
00:05:47.280 --> 00:05:55.880
The next thing that go wrong is that every time you run a review agent on the same pull request, you will somehow find new bugs in the sense that it will keep catching new things

s85
00:05:55.880 --> 00:06:02.360
in the pull request very annoyingly, and you will And like I said, bugs will appear to be fractally infinite.

s86
00:06:02.360 --> 00:06:07.480
So, the first thing that you want to do, the lowest hanging fruit, what is you want to optimize the prompt.

s87
00:06:07.480 --> 00:06:15.440
You started with a very simple prompt, something like review this PR and then output the comments in GitHub's API schema format.

s88
00:06:15.440 --> 00:06:17.480
Generally, there's two themes within the bugs.

s89
00:06:17.480 --> 00:06:20.240
Look for specific bug types, and second, skip the nitpicks.

s90
00:06:20.240 --> 00:06:22.919
And that's what you want to orient your prompt around.

s91
00:06:22.919 --> 00:06:33.080
You'll want to ask for specific bug types, and you can kind of do some internal data science on existing outages and incidents that have happened in your code base and figure out what types of bugs you have the highest propensity to produce.

s92
00:06:33.080 --> 00:06:37.720
Create a catalog of them, and then have your prompt look for those specific bug types.

s93
00:06:37.720 --> 00:06:41.680
The second is you'll need to expressly tell it what a nitpick is and not to look for them.

s94
00:06:41.680 --> 00:06:51.400
And nitpicks can be anything from asking why there isn't three layers of defense on this particular thing, or why error handling is done in this slightly shoddy way.

s95
00:06:51.400 --> 00:07:01.400
Now, the thing that you'll run into when you change these prompts is when you change a small thing in the prompt, it'll have a weirdly big impact on the number of comments that your agent is writing on your pull request.

s96
00:07:01.400 --> 00:07:04.600
This is quite scary because it means you don't really know if it's missing a bug or not.

s97
00:07:04.600 --> 00:07:06.680
You don't know the counterfactual.

s98
00:07:06.680 --> 00:07:13.640
The validation agents, or really any reviewer or tester, can can tell you about the presence of bugs, but it fortunately can't tell you about the absence of bugs.

s99
00:07:13.640 --> 00:07:16.480
That's that's sort of a a limitation of the format.

s100
00:07:16.480 --> 00:07:21.120
And what makes this particularly difficult are bugs that are extraordinarily rare and getting rarer by the day.

s101
00:07:21.120 --> 00:07:27.520
In fact, according to our data science, only about one out of every thousand pull requests has a generally catastrophic bug in it.

s102
00:07:27.520 --> 00:07:30.600
Or maybe one out of every thousand lines of code has a catastrophic bug in it.

s103
00:07:30.600 --> 00:07:44.280
And so, because these bugs are very rare, you might release a version of your agent that you think is very good, but it secretly misses a catastrophic bug in a pull request, and you wouldn't know for a long time because you don't produce those bugs very often.

s104
00:07:44.600 --> 00:07:46.440
And so that brings us to the second improvement.

s105
00:07:46.440 --> 00:07:49.160
You will want a data set of historical PRs.

s106
00:07:49.160 --> 00:07:54.360
This is relatively easy if you're one of those companies that very nicely catalogs every time something goes wrong in your code base.

s107
00:07:54.360 --> 00:07:58.400
If you have that, then you'll have a list of pull requests that have caused outages and incidents in the past.

s108
00:07:58.400 --> 00:08:07.120
You'll want to catalog them and then essentially have any version of your review agent run through those and see with what frequency it can catch those bugs.

s109
00:08:07.240 --> 00:08:10.760
Now, this will allow you to tune the prompt a little bit more confidently.

s110
00:08:10.760 --> 00:08:17.600
But at the same time, the interesting thing about bugs is in a changing world, the topology and ontology of bugs changes over time.

s111
00:08:17.600 --> 00:08:21.000
And so you'll need to have a real-time updating data set.

s112
00:08:21.000 --> 00:08:27.480
And this is tricky because unless you have millions and millions of pull requests, it's actually hard to have reasonably high coverage of bug types.

s113
00:08:27.480 --> 00:08:34.200
It's true that about 80 or 90% of your bugs will be just repeats of prior mistakes, but increasingly as code agents get good, they're going to start making novel mistakes.

s114
00:08:34.200 --> 00:08:37.080
They'll fail in new ways that we've not anticipated before.

s115
00:08:37.080 --> 00:08:39.360
And so this is where you will start to hit limitations.

s116
00:08:39.360 --> 00:08:50.920
Now, quick side note, this is kind of what we do at Reptile, except we actually have a lot of code from a lot of different companies and we can test across a very, very large corpus of pull requests and therefore improve our agent very confidently.

s117
00:08:50.920 --> 00:08:56.040
It's one of the reasons why we're able to eke out little bits of performance on the margins in ways that internal review agents can't.

s118
00:08:56.040 --> 00:09:02.680
But you can get decently far, especially for your particular company, to do this with your internal data.

s119
00:09:02.720 --> 00:09:04.360
The next thing you want to do is model inversion.

s120
00:09:04.360 --> 00:09:13.839
So, for in case anyone was at my talk earlier this year in San Francisco at AI Engineer, I talked a lot about the ways in which coding agents fail.

s121
00:09:13.839 --> 00:09:20.400
And I discovered an interesting pattern through that research, which is that the types of bugs produced by Claude models or Codex models are actually quite different from each other.

s122
00:09:20.400 --> 00:09:23.320
And they're different in a number of ways.

s123
00:09:23.320 --> 00:09:29.120
As an example, Claude models are something like three times more likely to produce one-off errors than GPT models.

s124
00:09:29.120 --> 00:09:33.800
GPT models, on the other hand, are significantly more likely to produce auth bypasses than Claude models.

s125
00:09:33.800 --> 00:09:37.840
And these are all just artifacts of the post-training, as far as I understand.

s126
00:09:37.840 --> 00:09:50.720
Now, to prevent this, you'll find that if you're using the same model to review the code written by the same model, um you'll actually have relatively fewer bugs caught because they fail in the same ways during review that they do during code generation.

s127
00:09:50.720 --> 00:09:51.880
And so you'll want to invert the model.

s128
00:09:51.880 --> 00:09:56.920
If you've used Claude code to write the code, you'll want to use a GPT model to review it and so on.

s129
00:09:56.920 --> 00:10:04.040
Now, doing this is somewhat straightforward for the 80th percentile and quite tricky for the last 20.

s130
00:10:04.040 --> 00:10:09.360
So, if you're using coding agents, you'll notice they leave some type of fingerprint to tell you what coding agent was used for a particular pull request.

s131
00:10:09.360 --> 00:10:14.120
For instance, Codex will name branches in a codex/branch name format.

s132
00:10:14.120 --> 00:10:18.440
Or Claude is going to write a co-authored by Claude in the PR description and commit message.

s133
00:10:18.440 --> 00:10:20.280
And about 80% of the time, this will be true.

s134
00:10:20.280 --> 00:10:28.680
In the final 20, when you happen to manually create a branch name or there was some other reason that a different coding agent was perhaps used, it's significantly harder to figure out which model was used during generation.

s135
00:10:28.680 --> 00:10:32.560
And you have to develop some type of heuristics to figure out what are the signatures of models being used.

s136
00:10:32.560 --> 00:10:43.360
Often you might have an internal catalog where people report what coding agents are using these days or have some type of telemetry that's coming out of your dev machines and and your engineers' computers.

s137
00:10:43.680 --> 00:10:46.560
We found that there's a 10% higher catch rate if you flip models.

s138
00:10:46.560 --> 00:10:52.200
And this might sound very small, but you have to know that 10% is in terms of incidents and outages is actually quite a lot.

s139
00:10:52.200 --> 00:10:58.160
It means that there's a 10% higher chance that something will go wrong if you're not using model inversion, which can be quite catastrophic.

s140
00:10:58.160 --> 00:11:04.200
And so we highly recommend that you use some type of model inversion in your process.

s141
00:11:04.560 --> 00:11:06.920
The third improvement is code base context.

s142
00:11:06.920 --> 00:11:08.600
Now, this is something that I care a lot about.

s143
00:11:08.600 --> 00:11:12.000
Before we were doing code validation, we were doing code base context.

s144
00:11:12.000 --> 00:11:15.680
Uh we were trying to figure out better ways to do rag over very large code bases.

s145
00:11:15.680 --> 00:11:20.360
And even today, people consider grep to effectively solved all of code base context.

s146
00:11:20.360 --> 00:11:21.760
I actually think that's not true for a number of reasons.

s147
00:11:21.760 --> 00:11:24.040
One, grep is extremely imprecise.

s148
00:11:24.040 --> 00:11:26.240
And second, it's very expensive.

s149
00:11:26.240 --> 00:11:30.920
To give you some sense of scale, single-digit percentage points of GDP growth last year were inference.

s150
00:11:30.920 --> 00:11:33.480
A very large percentage of inference was coding agents.

s151
00:11:33.480 --> 00:11:39.240
And if you actually look at a coding agent trace, the vast majority of tokens used by coding agents are all the same.

s152
00:11:39.240 --> 00:11:42.160
grep, file read, cache write for the file contents.

s153
00:11:42.160 --> 00:11:47.160
That means a non-trivial percentage of GDP growth is grep, file read, cache write.

s154
00:11:47.160 --> 00:11:47.920
That's very bad.

s155
00:11:47.920 --> 00:11:49.160
And this is because grep is very imprecise.

s156
00:11:49.160 --> 00:11:56.240
It's reading files in very large quantities, and I don't know, there's perhaps some adverse incentives for companies paid by the token to figure out ways to burn more tokens.

s157
00:11:56.240 --> 00:12:02.240
But even so, you'll find that you'll need to use slightly more surgical methods of figuring out where you should be reading.

s158
00:12:02.240 --> 00:12:12.160
And the way we do this is we produce an internal knowledge base where we essentially check across the entire code base all the different parts, catalog them, and then index saying, "Okay, well, this is authentication.

s159
00:12:12.160 --> 00:12:12.840
Here's how it works.

s160
00:12:12.840 --> 00:12:13.520
Here are the flows.

s161
00:12:13.520 --> 00:12:14.520
Here's a diagram of the flows.

s162
00:12:14.520 --> 00:12:16.440
Here's every file that's involved."

s163
00:12:16.440 --> 00:12:20.640
One example of where code base context matters is, for instance, when you're trying to catch an off bypass.

s164
00:12:20.640 --> 00:12:28.360
It's actually very difficult to figure out if an off bypass has occurred because you need to have a pretty concrete understanding of whether you're in an authenticated environment when you're in that part of the code base or not.

s165
00:12:28.360 --> 00:12:29.800
And that's very tricky.

s166
00:12:29.800 --> 00:12:36.360
And so, you'll find that you will need some type of uh of sort of code base context to be able to do this.

s167
00:12:36.400 --> 00:12:43.120
Now, I've been very centered around the concept of bugs, but code can be very, very bad even if it's not buggy.

s168
00:12:43.120 --> 00:12:48.600
For one, it can be terrible quality, but second, it can just not fulfill the expressed intent of the author.

s169
00:12:48.600 --> 00:12:53.360
Recently, I saw a post that said that people were increasingly using Slack as their IDE.

s170
00:12:53.360 --> 00:12:58.839
Now, I think that's a little funny, but at the same time, there's not that much ex- expressed intent in a Slack message.

s171
00:12:58.839 --> 00:13:06.400
You're only saying maybe a sentence or two, if even, asking an agent to go and fix something, which means the there's a lot of ambiguity in your instruction.

s172
00:13:06.400 --> 00:13:13.400
As a result of this, you might find that the coding agent has either misinterpreted your instructions or just it was too ambiguous and it didn't know exactly what to do.

s173
00:13:13.400 --> 00:13:16.880
And so, your correct code can actually still be quite wrong.

s174
00:13:16.880 --> 00:13:20.720
Now, one of the ways in which we fight this is by giving agents increasingly more context.

s175
00:13:20.720 --> 00:13:29.720
And we find that the two types of context that matter most are docs and tickets, and then of course, the code base from your code base indexer or knowledge base.

s176
00:13:29.720 --> 00:13:35.800
You can get remarkably far if you give it access to your Jira and Linear, and then your Confluence or Notion or whatever it is that you store your docs.

s177
00:13:35.800 --> 00:13:40.200
You'll find that there's a lot of tribal knowledge that is subconsciously helping you produce better code.

s178
00:13:40.200 --> 00:13:43.640
And when you have agents do the same thing, it feels like they just don't know what's happening.

s179
00:13:43.640 --> 00:13:49.440
And because you you underestimate how much depth of knowledge you yourself have.

s180
00:13:51.120 --> 00:13:58.320
Now, the way that Reptile approaches this problem is we actually disintermediate all of these context sources with a living wiki.

s181
00:13:58.320 --> 00:14:02.560
This wiki is essentially a sort of world model of how your company works.

s182
00:14:02.560 --> 00:14:11.440
It's everything that it knows about your code base, everything it knows about your application through production telemetry, through logs, everything it knows about your road map from your documentation

s183
00:14:11.440 --> 00:14:14.040
and your linear tickets and so on and so forth.

s184
00:14:14.040 --> 00:14:25.400
And it uses this wiki to supply context both making the agent more context aware and making its code searches significantly more precise because it will never read a file that it didn't need to read.

s185
00:14:26.360 --> 00:14:29.040
The last thing that you should do, or this might be four out of five.

s186
00:14:29.040 --> 00:14:31.814
The next thing that you should do is autonomous testing.

s187
00:14:31.814 --> 00:14:31.839
[clears throat]

s188
00:14:31.839 --> 00:14:43.560
One of the things that we think about a lot is that for instance, for UI changes, you might be able to read through a bunch of changed files for React code and conclude that there's nothing wrong with the code.

s189
00:14:43.560 --> 00:14:47.600
But in fact, when you render it, it might be, for instance, hidden on an iPhone.

s190
00:14:47.600 --> 00:14:51.200
Now, this is one of the more technically difficult problems to solve.

s191
00:14:51.200 --> 00:15:05.640
The way that we approach this at Reptile is we essentially build these dependency-rich sandboxes, which are completely configurable virtual machines, and we have customers, we work with them to essentially spin up their stack in these sandboxes.

s192
00:15:05.640 --> 00:15:09.480
And then Reptile's agent uses computer vision to test them end to end.

s193
00:15:09.480 --> 00:15:16.520
If you haven't been paying attention, computer vision has gotten very good, and it's like far better in these new generation of GPT models and Opus models than it was in the previous.

s194
00:15:16.520 --> 00:15:19.800
For the first time, I think end to end QA is nearly solved.

s195
00:15:19.800 --> 00:15:22.960
The thing that you need to do though is you need to document all the flows through your application.

s196
00:15:22.960 --> 00:15:30.480
And second, you need to be able to have a sufficiently simple build script so that the agent can spin up your stack in your environments.

s197
00:15:30.480 --> 00:15:33.160
We have a team of forward deployed engineers that help companies do this.

s198
00:15:33.160 --> 00:15:34.920
We found that it's one of the trickier parts.

s199
00:15:34.920 --> 00:15:40.800
Unless of course, you have very good dev boxes already, in which case congratulations on working at Stripe.

s200
00:15:43.720 --> 00:15:45.760
So, oh, I This is the diagram I'm going to show you.

s201
00:15:45.760 --> 00:15:53.000
But yes, you will need a dev box, you will need PR branch dependencies where you're running the app, and you'll need some type of flow map that says, "This is the auth code and then this is the sign-up flow."

s202
00:15:53.000 --> 00:15:54.240
and so on.

s203
00:15:54.240 --> 00:16:03.160
And using this, your agent can now test your code end-to-end and use vision models to say, "Oh, like this button technically appears, but it actually is hidden on iPhones, or it's not in the first fold on

s204
00:16:03.160 --> 00:16:08.600
the iPhone duo, etc." Now, if you do all of this, you'll find that the costs are still quite high.

s205
00:16:08.600 --> 00:16:14.400
It can cost you anywhere from three bucks to 15 to 20 to do a single pull request review.

s206
00:16:14.400 --> 00:16:18.200
And the thing you'll want to solve for now is to use cheaper models for some things.

s207
00:16:18.200 --> 00:16:23.560
Now, to me, frontier models are still generally speaking more capable than open source models.

s208
00:16:23.560 --> 00:16:26.800
This has been true for some time, even though the rubber band is tightening.

s209
00:16:26.800 --> 00:16:31.360
So, our share today is about 20% frontier models for the most critical tasks.

s210
00:16:31.360 --> 00:16:37.440
And then the long tail of tokens, which is exploratory, file reads, and so on, we use a variety of smaller models.

s211
00:16:37.440 --> 00:16:43.760
And so, if you do this, you can actually reduce the cost of running your review agent quite drastically.

s212
00:16:43.760 --> 00:16:53.040
So, this is sort of a illustrative example of what we do, which is we'll use a small model for our finding context, we'll use a very lightweight computer use model for testing user flows,

s213
00:16:53.040 --> 00:16:57.600
and we use a frontier model for things like analyzing bugs, applying the review process, and first submission.

s214
00:16:57.600 --> 00:17:03.240
In practice, we have actually 20 different subtasks, and each of them has a very specific set of models that we use for each of them.

s215
00:17:03.240 --> 00:17:07.880
And we have a small research team that is constantly iterating exactly what models we use for what task.

s216
00:17:07.880 --> 00:17:15.360
You can probably get 70 to 80% of the way there using something of this kind where you just swap out very simple tasks with very simple models.

s217
00:17:15.360 --> 00:17:18.439
Now, this is another example of where it really helps to have a very large data set.

s218
00:17:18.439 --> 00:17:31.560
If you don't have a good data set, you might be able to have trigger happily replace a lot of different sub agents with smaller, faster models, and you won't know that it's this agent is actually now quite a bit worse at catching bugs, and you would only find out when it misses something that then causes an outage

s219
00:17:31.560 --> 00:17:33.680
or incident.

s220
00:17:34.720 --> 00:17:45.680
And you will also find that you can kind of combat this problem of overactive file reads using a cheap model that essentially looks through all of your context pieces and figures out which ones actually should be included.

s221
00:17:45.680 --> 00:17:58.920
In theory, you should be able to use a decision model for this, something like Java, or the Decisions API, to do effective re-ranking on all of the different things, and deciding what should be and shouldn't be loaded into Frontier Model Cache, which is in some ways your most expensive and most scarce resource.

s222
00:17:58.920 --> 00:18:03.600
If you do all of these things, you might be able to actually get quite far in building your own review agent.

s223
00:18:03.600 --> 00:18:06.480
And if you do, please let me know.

s224
00:18:06.480 --> 00:18:08.440
My email is daksh@prod.com.

s225
00:18:08.440 --> 00:18:09.560
I'd love to help you.

s226
00:18:09.560 --> 00:18:16.040
And uh if you choose not to build it, then you can also just use our product, um which is free in comparison.

s227
00:18:16.040 --> 00:18:17.792
But, uh thank you so

s228
00:18:17.792 --> 00:18:17.800
[snorts]

s229
00:18:17.800 --> 00:18:19.800
much.