How to Kill the Code Review — Ankit Jain, Aviator

AI Engineer · 16 min · 185 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Okay.
00:14Ooh, hello.
00:15Hey everyone.
00:16Thanks for joining in.
00:17Today, uh we will be talking about how to kill the code reviews.
00:22Everyone's favorite topic.
00:23I'm Ankit, co-founder of Aviator.
00:26At Aviator, we are building AI code verification platform.
00:29So, we'll bring in some of the ideas and concepts that we talked about in the uh that we build in our product.
00:36Uh but first, let's dive into a little bit.
00:40So, a few months ago, I wrote a post on LinkedIn space about uh how to kill code review, creating a framework, a five-layer trust model.
00:49So, this model was focused around how do we actually layer by layer build trust into the code that can then be merged without needing line-by-line review.
01:03And I got some things right, and I got some things wrong.
01:06So, this talk will be about just really diving a bit more more into it.
01:09I'm not going to talk about specific layers, but we will talk about some of the concepts that emerged from this session.
01:17So, let's just talk about the problem.
01:19We are looking today at the volume of code is increasing every day, and we are struggling to keep up.
01:28So, when we think about like how long will it take us to actually stop reading code line by line?
01:35And the reality is we've already stopped reviewing it.
01:39There is 861% code churn.
01:41That means we are like producing more code.
01:44The incidents to PR ratio is increasing.
01:46That means even if you're doing reviews today, they're not effective.
01:51So, the medium time of review is increasing.
01:53We have just increased the bottleneck to from the coding now kind of like coding is solved to now reviewing, where everything just gets stuck there.
02:03You're spending four x the time that you were spending before just waiting for the reviews.
02:08And today, over 30% of changes are actually getting merged without a review at all.
02:14So, let's just think about it a little bit.
02:19Think about AI reviews.
02:20So, everyone is probably using some form of AI reviews today.
02:23When AI write the code and AI review the code, why are we doing it in a UI?
02:28Right?
02:28Like we open a GitHub, there's kind of like, you know, maybe two or three AI coding agents who are like doing the reviews.
02:34You know, you the review goes back passed to the user and or the agent and it gets resolved and they kind of like you're doing back and forth with the agent.
02:42Where is human here in the loop?
02:44Like you're eventually just looking at, okay, you know, if AI has reviewed it, most of the things probably found, let's just kind of like, you know, do a skimming of it and merge it.
02:53So, when AI reviews and nobody reads, we have configured the wrong thing.
03:00So, let's just take a step back.
03:02Code reviews are not very old.
03:04They are maybe 15 to 20 years ago.
03:06In 2006 was when Google launched Mondrian internally and they made formal code review as a thing.
03:13If you think about it, Windows back in the days, the first versions were actually built without reviews.
03:19But if we look carefully, code review is not just about code reviews.
03:24You know, obviously we are looking at it like catching bugs, understanding kind of conventions, identifying security issues.
03:31But, code review is also about alignment.
03:34And this was one piece which was missing from my five layers model that I talked about a few months ago.
03:40So, a big part of code reviews is knowledge sharing, mentorship, architectural feedback, onboarding, being able to collaborate.
03:49This is Again, if you're doing uh white coding, you're working as a solo project, this is not a talk for you.
03:56If you are working in teams, which I believe most of you folks are, collaborating in teams, you're not likely using completely dark factories, orchestrators, where nobody looks at the code.
04:05You're actually collaborating in teams, you need to do knowledge sharing.
04:08Which is the alignment part.
04:10And that is the part which is the most important aspect of release.
04:14So, for semantic accuracy, we can build better tooling, but alignment must survive.
04:21So, let's just kind of like dive into alignment.
04:23What does it mean?
04:24Like, in today's world, can we actually think of better model than aligning just based on reading line by line code?
04:34So, most folks have probably heard about spectrum development by now.
04:38So, spectrum development is like, okay, we write a spec, you know, it covers all the details, we pass it to an agent, it generates a code, and then we verify.
04:48So, what's wrong here?
04:49If you look back in 1970, this is what waterfall model was.
04:53You know, you have requirements, you have specification, you implement, and then you verify.
04:57But, there's no feedback loop.
05:00You know, the spec is written before we identified everything else.
05:04Right?
05:04That's why today everyone still wants to use your coding sessions, whether it's cloud code, codex, cursor, whatever you're using, you want to interact with the agents.
05:13And the reason you're interacting with agents is because there were certain things which were not clear in the spec, and we still need to capture that.
05:20And second is, as you implement, you identify more issues, and you never go back and update the spec, because like, if you're doing the spectrum development, it's already done.
05:29Once the spec is done, you expect, like, you know, the code will come deterministically.
05:33But guess what?
05:34LLM is not deterministic.
05:35It's going to make decisions itself.
05:37So, that's why spectrum development is a great uh methodology, but it falls short in day-to-day software development.
05:45But there are some interesting aspects of this, which we should carry forward.
05:49The most important part is the intent.
05:53An intent doesn't only live in the spec.
05:56Intent live in your Jira ticket.
05:58That's the goal, right?
05:59Like it's kind of like where you express what we want to do.
06:03It lives in your PRDs.
06:04It's kind of like details, like it's a plan.
06:07But most importantly, it lives in your prompts today.
06:11This is where the real decisions are being made.
06:13You start with like, okay, this is a Jira ticket I'm going to look at, but you're going back and forth with the agent, and this is where all the user decisions are being made.
06:21But what we do today is we create a change, we create a pull request, and then we throw away the prompts.
06:28And this is one of the things that we need to change.
06:32Let us first talk also about semantic accuracy because like, you know, you're saying, "Hey, Ankit, okay, I understand the alignment part, but like there are still bugs in the code.
06:39Who's going to look at that?"
06:40LLMs are also not great at this, and we already talked about how AI agents may reviewers may not always be perfect.
06:49So, this is where I introduce you to the concept of AI slop registry.
06:53So, think about this.
06:55We are reviewing If you're reviewing code today manually, and I expect everyone should be doing some degree of this.
07:02We are essentially possibly identifying the same issues over and over again.
07:07Can we actually capture these concepts and codify them so that we don't have to always create those review feedback one by one.
07:18You actually also have all of those things automatically identified.
07:22The beauty of this is if you do it a few times, you now build sort of like a system which actually learns over time.
07:29So, think of this as sort of like you know, you know, you're doing more training on top of the standard LLM that you have actually extracted like built on top of.
07:38So, AI slop registers now can create better results because it's trained it's learning from the review experience that you as humans are providing.
07:52Every recurring comment is now a guardrail that you don't have to review again.
07:58Okay, so let's kind of like try to put both of these alignment and the semantic accuracy together.
08:05It is two halves of the same problem.
08:08We are trying to understand what are the core mechanics of review.
08:12How do we actually break it down into two parts which is the alignment and the semantic accuracy and bring them together into a single loop.
08:19So, first thing is you could take your session and you capture the user responses and that essentially forms your acceptance criteria.
08:30The acceptance criteria then tied with your AI slot register that you are now constantly maintaining finally creates a test plan.
08:38And this is the test plan which then gets verified.
08:42This is part of the system that we are building is the verification system where it spins up a preview, takes your test plan and make sure it actually works end-to-end.
08:51Even if the code looks right, does it actually work?
08:55So, this verification part has become interesting.
08:57And now the kicker is this is now your review surface.
09:02You're not reviewing code line by line, but rather you're looking at the evidence of what was the intent, did the user actually implement the capability that was defined in the intent,
09:14and did the behavior actually meet the requirements that we had in the as acceptance criteria.
09:21So, you're still having the architectural decisions, you're still having these arguments, but the review surface changes.
09:30So, just giving a walk through of like how we have built our system.
09:35The session becomes a criteria, so all these decisions that you are doing here with the agent, you're asking uh you know, you're providing this feedback.
09:43Like even if there is a simple task, many times you're going back and forth.
09:47Agent will stop to ask questions.
09:49These are the decisions that we need to capture.
09:52This is the intent.
09:53This is what makes your review, makes your collaboration more valuable.
10:00This is how you teach your junior engineers on how to improve over time.
10:05These are the decisions which make a software engineer valuable today.
10:10We convert those into an acceptance criteria.
10:13This is where you can also leverage LLM to do so.
10:15So, you capture these user uh decisions and make sure you can actually create a test plan based on this.
10:22I know test plan creation is always painful, which is where I would always recommend people to use LLM for this purpose.
10:30And finally, the criteria plus invariant is what makes the test plan.
10:35Uh and we build the verification systems to actually capture the test plan, run your previews, and be able to test based on this test plan.
10:44So, even imagine if you're building a new feature, you don't have to maintain tests at all.
10:49This is creating tests in real time, and this is where you can leverage the power of LLM because the test plan maintenance and creation can be really painful, but
11:01the value of human in the loop here is the governance and the review part, and the part where you're reviewing the test plan and not the code.
11:13Right?
11:14I know it's been like over 20 years we came up with test-driven development.
11:17This in some ways is closer to behavior-driven development, where the test plan is now something which even you can share with your product managers, your designers.
11:28Everyone can participate because these things are now in English.
11:31At the same time, we have now deterministic verification, which actually verifies whether the particular test criteria has been met or not.
11:44Let's move on.
11:47So, this is where I would say the system is not supposed to be perfect.
11:53It's deterministic where it can be, but LLM where you must.
11:58Not everything can be deterministic.
12:03Not every system can be built in a way which is 100% built on deterministic systems.
12:11This is where you use LLM as a fallback.
12:16Let me give an example.
12:17Like if you're making a change in your web application, what you can do is it creates a test plan of what the behavior changes.
12:25Let's say you introduce a new payment form.
12:28So, it creates a new payment form.
12:30The verification here is does the payment system changes?
12:34Uh an AI agent can go and browse through your application to fill out a form and capture screenshots as evidence.
12:45And then take those screenshots as well as your database snapshots to identify whether the criteria was met.
12:52So, the screenshot testing or like sand flying can then still be done by agents.
12:57But at the same time, you're creating more solid evidence, which now a reviewer can look at and build more confidence that this actually works.
13:08So, now reviewers are reviewing the intent, not the diff.
13:13You're reviewing the intent decisions.
13:15Uh what we said to build out.
13:18What we tried and rejected, and capture all of these things from the sessions.
13:22Remember, capturing it from the sessions is is key.
13:25If we try to build it from the code, you'll end up in the same situation that we were talking about I think Dex was talking about yesterday, which is
13:32if your code is built by the same agent which is actually building a test plan, it's not going to build a test plan which will actually catch issues.
13:42So, that's why it's important to actually use the session information to build out a test plan.
13:48You can discuss architectural decisions.
13:50So, how you're creating the data models, how these services interact with each other.
13:53So, you've moved one level above.
13:56So, you're essentially instead of like reviewing line by line, you're actually having discussions on architecture, which are very critical for any kind of collaboration.
14:05And then you look at the evidence, everything that was collected from the verification.
14:12So, here's a homework for everyone.
14:16Go home and mine your last 1,000 review comments and build out a AI slot register for the things which are repeatable.
14:25So, a vast majority of the comments that you're providing in your code review are something that we repeat over and over again.
14:33This compounds with every merge PR.
14:37Every time you capture something as a register, you don't have to capture that comment again.
14:43And this is where you can actually codify some of the best practices of doing code, maintain semantic accuracy, at the same time, do not lose the collaboration part of the review.
14:58It does follow a J curve.
15:00So, pain is real.
15:01Uh you will have to spend some time to actually make it pay off because initially creating a registry can take some time.
15:08And this is where I would recommend you folks can come and try out our product.
15:13So, code review is not just about code review.
15:16It is about uh really getting the alignment.
15:20And where we can build better tools is creating the semantic accuracy and defining your AI slot register.
15:29So, if you remember one thing from today, remember code review is not just about code review, it is about getting the alignment.
15:38And yes, we are piloting our new product called Verify.
15:43Please join and be our early design partners.
15:46We are working with a few companies to uh pilot out a new verification system.
15:52This combines both the alignment side of things as well as building tools and capabilities for detecting semantic accuracy using the AI slot register.
16:04Thank you, everyone.
16:04Thanks for joining.
16:06[applause]