WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=YgEv7IQzGdM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/YgEv7IQzGdM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.080 --> 00:00:14.040
Okay.

s3
00:00:14.040 --> 00:00:15.000
Ooh, hello.

s4
00:00:15.000 --> 00:00:16.360
Hey everyone.

s5
00:00:16.360 --> 00:00:17.760
Thanks for joining in.

s6
00:00:17.760 --> 00:00:22.120
Today, uh we will be talking about how to kill the code reviews.

s7
00:00:22.120 --> 00:00:23.640
Everyone's favorite topic.

s8
00:00:23.640 --> 00:00:26.000
I'm Ankit, co-founder of Aviator.

s9
00:00:26.000 --> 00:00:29.440
At Aviator, we are building AI code verification platform.

s10
00:00:29.440 --> 00:00:36.640
So, we'll bring in some of the ideas and concepts that we talked about in the uh that we build in our product.

s11
00:00:36.640 --> 00:00:40.400
Uh but first, let's dive into a little bit.

s12
00:00:40.400 --> 00:00:49.680
So, a few months ago, I wrote a post on LinkedIn space about uh how to kill code review, creating a framework, a five-layer trust model.

s13
00:00:49.680 --> 00:01:03.320
So, this model was focused around how do we actually layer by layer build trust into the code that can then be merged without needing line-by-line review.

s14
00:01:03.320 --> 00:01:06.000
And I got some things right, and I got some things wrong.

s15
00:01:06.000 --> 00:01:09.640
So, this talk will be about just really diving a bit more more into it.

s16
00:01:09.640 --> 00:01:16.800
I'm not going to talk about specific layers, but we will talk about some of the concepts that emerged from this session.

s17
00:01:17.120 --> 00:01:19.280
So, let's just talk about the problem.

s18
00:01:19.280 --> 00:01:28.280
We are looking today at the volume of code is increasing every day, and we are struggling to keep up.

s19
00:01:28.280 --> 00:01:35.560
So, when we think about like how long will it take us to actually stop reading code line by line?

s20
00:01:35.560 --> 00:01:39.160
And the reality is we've already stopped reviewing it.

s21
00:01:39.160 --> 00:01:41.800
There is 861% code churn.

s22
00:01:41.800 --> 00:01:44.200
That means we are like producing more code.

s23
00:01:44.200 --> 00:01:46.760
The incidents to PR ratio is increasing.

s24
00:01:46.760 --> 00:01:51.000
That means even if you're doing reviews today, they're not effective.

s25
00:01:51.000 --> 00:01:53.880
So, the medium time of review is increasing.

s26
00:01:53.880 --> 00:02:03.080
We have just increased the bottleneck to from the coding now kind of like coding is solved to now reviewing, where everything just gets stuck there.

s27
00:02:03.080 --> 00:02:08.880
You're spending four x the time that you were spending before just waiting for the reviews.

s28
00:02:08.880 --> 00:02:14.560
And today, over 30% of changes are actually getting merged without a review at all.

s29
00:02:14.560 --> 00:02:19.120
So, let's just think about it a little bit.

s30
00:02:19.120 --> 00:02:20.440
Think about AI reviews.

s31
00:02:20.440 --> 00:02:23.720
So, everyone is probably using some form of AI reviews today.

s32
00:02:23.720 --> 00:02:28.080
When AI write the code and AI review the code, why are we doing it in a UI?

s33
00:02:28.080 --> 00:02:28.240
Right?

s34
00:02:28.240 --> 00:02:34.560
Like we open a GitHub, there's kind of like, you know, maybe two or three AI coding agents who are like doing the reviews.

s35
00:02:34.560 --> 00:02:42.680
You know, you the review goes back passed to the user and or the agent and it gets resolved and they kind of like you're doing back and forth with the agent.

s36
00:02:42.680 --> 00:02:44.480
Where is human here in the loop?

s37
00:02:44.480 --> 00:02:53.480
Like you're eventually just looking at, okay, you know, if AI has reviewed it, most of the things probably found, let's just kind of like, you know, do a skimming of it and merge it.

s38
00:02:53.480 --> 00:03:00.080
So, when AI reviews and nobody reads, we have configured the wrong thing.

s39
00:03:00.160 --> 00:03:02.360
So, let's just take a step back.

s40
00:03:02.360 --> 00:03:04.640
Code reviews are not very old.

s41
00:03:04.640 --> 00:03:06.560
They are maybe 15 to 20 years ago.

s42
00:03:06.560 --> 00:03:13.640
In 2006 was when Google launched Mondrian internally and they made formal code review as a thing.

s43
00:03:13.640 --> 00:03:19.720
If you think about it, Windows back in the days, the first versions were actually built without reviews.

s44
00:03:19.720 --> 00:03:24.680
But if we look carefully, code review is not just about code reviews.

s45
00:03:24.680 --> 00:03:31.440
You know, obviously we are looking at it like catching bugs, understanding kind of conventions, identifying security issues.

s46
00:03:31.440 --> 00:03:34.640
But, code review is also about alignment.

s47
00:03:34.640 --> 00:03:40.800
And this was one piece which was missing from my five layers model that I talked about a few months ago.

s48
00:03:40.800 --> 00:03:49.360
So, a big part of code reviews is knowledge sharing, mentorship, architectural feedback, onboarding, being able to collaborate.

s49
00:03:49.360 --> 00:03:56.040
This is Again, if you're doing uh white coding, you're working as a solo project, this is not a talk for you.

s50
00:03:56.040 --> 00:04:05.720
If you are working in teams, which I believe most of you folks are, collaborating in teams, you're not likely using completely dark factories, orchestrators, where nobody looks at the code.

s51
00:04:05.720 --> 00:04:08.800
You're actually collaborating in teams, you need to do knowledge sharing.

s52
00:04:08.800 --> 00:04:10.680
Which is the alignment part.

s53
00:04:10.680 --> 00:04:14.160
And that is the part which is the most important aspect of release.

s54
00:04:14.160 --> 00:04:21.480
So, for semantic accuracy, we can build better tooling, but alignment must survive.

s55
00:04:21.680 --> 00:04:23.880
So, let's just kind of like dive into alignment.

s56
00:04:23.880 --> 00:04:24.720
What does it mean?

s57
00:04:24.720 --> 00:04:34.360
Like, in today's world, can we actually think of better model than aligning just based on reading line by line code?

s58
00:04:34.360 --> 00:04:38.280
So, most folks have probably heard about spectrum development by now.

s59
00:04:38.280 --> 00:04:48.080
So, spectrum development is like, okay, we write a spec, you know, it covers all the details, we pass it to an agent, it generates a code, and then we verify.

s60
00:04:48.080 --> 00:04:49.760
So, what's wrong here?

s61
00:04:49.760 --> 00:04:53.160
If you look back in 1970, this is what waterfall model was.

s62
00:04:53.160 --> 00:04:57.080
You know, you have requirements, you have specification, you implement, and then you verify.

s63
00:04:57.080 --> 00:05:00.040
But, there's no feedback loop.

s64
00:05:00.040 --> 00:05:04.080
You know, the spec is written before we identified everything else.

s65
00:05:04.080 --> 00:05:04.320
Right?

s66
00:05:04.320 --> 00:05:13.240
That's why today everyone still wants to use your coding sessions, whether it's cloud code, codex, cursor, whatever you're using, you want to interact with the agents.

s67
00:05:13.240 --> 00:05:20.840
And the reason you're interacting with agents is because there were certain things which were not clear in the spec, and we still need to capture that.

s68
00:05:20.840 --> 00:05:29.640
And second is, as you implement, you identify more issues, and you never go back and update the spec, because like, if you're doing the spectrum development, it's already done.

s69
00:05:29.640 --> 00:05:33.160
Once the spec is done, you expect, like, you know, the code will come deterministically.

s70
00:05:33.160 --> 00:05:34.240
But guess what?

s71
00:05:34.240 --> 00:05:35.640
LLM is not deterministic.

s72
00:05:35.640 --> 00:05:37.440
It's going to make decisions itself.

s73
00:05:37.440 --> 00:05:45.480
So, that's why spectrum development is a great uh methodology, but it falls short in day-to-day software development.

s74
00:05:45.480 --> 00:05:49.520
But there are some interesting aspects of this, which we should carry forward.

s75
00:05:49.520 --> 00:05:53.160
The most important part is the intent.

s76
00:05:53.160 --> 00:05:56.600
An intent doesn't only live in the spec.

s77
00:05:56.600 --> 00:05:58.240
Intent live in your Jira ticket.

s78
00:05:58.240 --> 00:05:59.120
That's the goal, right?

s79
00:05:59.120 --> 00:06:03.560
Like it's kind of like where you express what we want to do.

s80
00:06:03.560 --> 00:06:04.720
It lives in your PRDs.

s81
00:06:04.720 --> 00:06:07.720
It's kind of like details, like it's a plan.

s82
00:06:07.720 --> 00:06:11.000
But most importantly, it lives in your prompts today.

s83
00:06:11.000 --> 00:06:13.280
This is where the real decisions are being made.

s84
00:06:13.280 --> 00:06:21.680
You start with like, okay, this is a Jira ticket I'm going to look at, but you're going back and forth with the agent, and this is where all the user decisions are being made.

s85
00:06:21.680 --> 00:06:28.120
But what we do today is we create a change, we create a pull request, and then we throw away the prompts.

s86
00:06:28.120 --> 00:06:32.560
And this is one of the things that we need to change.

s87
00:06:32.760 --> 00:06:39.480
Let us first talk also about semantic accuracy because like, you know, you're saying, "Hey, Ankit, okay, I understand the alignment part, but like there are still bugs in the code.

s88
00:06:39.480 --> 00:06:40.720
Who's going to look at that?"

s89
00:06:40.720 --> 00:06:49.400
LLMs are also not great at this, and we already talked about how AI agents may reviewers may not always be perfect.

s90
00:06:49.400 --> 00:06:53.919
So, this is where I introduce you to the concept of AI slop registry.

s91
00:06:53.919 --> 00:06:55.360
So, think about this.

s92
00:06:55.360 --> 00:07:02.160
We are reviewing If you're reviewing code today manually, and I expect everyone should be doing some degree of this.

s93
00:07:02.160 --> 00:07:07.960
We are essentially possibly identifying the same issues over and over again.

s94
00:07:07.960 --> 00:07:18.600
Can we actually capture these concepts and codify them so that we don't have to always create those review feedback one by one.

s95
00:07:18.600 --> 00:07:22.480
You actually also have all of those things automatically identified.

s96
00:07:22.480 --> 00:07:29.040
The beauty of this is if you do it a few times, you now build sort of like a system which actually learns over time.

s97
00:07:29.040 --> 00:07:38.040
So, think of this as sort of like you know, you know, you're doing more training on top of the standard LLM that you have actually extracted like built on top of.

s98
00:07:38.040 --> 00:07:52.000
So, AI slop registers now can create better results because it's trained it's learning from the review experience that you as humans are providing.

s99
00:07:52.280 --> 00:07:57.640
Every recurring comment is now a guardrail that you don't have to review again.

s100
00:07:58.000 --> 00:08:05.760
Okay, so let's kind of like try to put both of these alignment and the semantic accuracy together.

s101
00:08:05.880 --> 00:08:08.160
It is two halves of the same problem.

s102
00:08:08.160 --> 00:08:12.360
We are trying to understand what are the core mechanics of review.

s103
00:08:12.360 --> 00:08:19.280
How do we actually break it down into two parts which is the alignment and the semantic accuracy and bring them together into a single loop.

s104
00:08:19.280 --> 00:08:30.520
So, first thing is you could take your session and you capture the user responses and that essentially forms your acceptance criteria.

s105
00:08:30.520 --> 00:08:38.680
The acceptance criteria then tied with your AI slot register that you are now constantly maintaining finally creates a test plan.

s106
00:08:38.680 --> 00:08:42.000
And this is the test plan which then gets verified.

s107
00:08:42.000 --> 00:08:51.520
This is part of the system that we are building is the verification system where it spins up a preview, takes your test plan and make sure it actually works end-to-end.

s108
00:08:51.520 --> 00:08:55.160
Even if the code looks right, does it actually work?

s109
00:08:55.160 --> 00:08:57.880
So, this verification part has become interesting.

s110
00:08:57.880 --> 00:09:02.480
And now the kicker is this is now your review surface.

s111
00:09:02.480 --> 00:09:14.640
You're not reviewing code line by line, but rather you're looking at the evidence of what was the intent, did the user actually implement the capability that was defined in the intent,

s112
00:09:14.640 --> 00:09:21.600
and did the behavior actually meet the requirements that we had in the as acceptance criteria.

s113
00:09:21.600 --> 00:09:29.040
So, you're still having the architectural decisions, you're still having these arguments, but the review surface changes.

s114
00:09:30.040 --> 00:09:35.000
So, just giving a walk through of like how we have built our system.

s115
00:09:35.000 --> 00:09:43.440
The session becomes a criteria, so all these decisions that you are doing here with the agent, you're asking uh you know, you're providing this feedback.

s116
00:09:43.440 --> 00:09:47.080
Like even if there is a simple task, many times you're going back and forth.

s117
00:09:47.080 --> 00:09:49.800
Agent will stop to ask questions.

s118
00:09:49.800 --> 00:09:52.520
These are the decisions that we need to capture.

s119
00:09:52.520 --> 00:09:53.960
This is the intent.

s120
00:09:53.960 --> 00:10:00.880
This is what makes your review, makes your collaboration more valuable.

s121
00:10:00.880 --> 00:10:05.680
This is how you teach your junior engineers on how to improve over time.

s122
00:10:05.680 --> 00:10:10.880
These are the decisions which make a software engineer valuable today.

s123
00:10:10.880 --> 00:10:13.240
We convert those into an acceptance criteria.

s124
00:10:13.240 --> 00:10:15.880
This is where you can also leverage LLM to do so.

s125
00:10:15.880 --> 00:10:22.240
So, you capture these user uh decisions and make sure you can actually create a test plan based on this.

s126
00:10:22.240 --> 00:10:29.960
I know test plan creation is always painful, which is where I would always recommend people to use LLM for this purpose.

s127
00:10:30.040 --> 00:10:35.200
And finally, the criteria plus invariant is what makes the test plan.

s128
00:10:35.200 --> 00:10:44.000
Uh and we build the verification systems to actually capture the test plan, run your previews, and be able to test based on this test plan.

s129
00:10:44.000 --> 00:10:49.120
So, even imagine if you're building a new feature, you don't have to maintain tests at all.

s130
00:10:49.120 --> 00:11:01.040
This is creating tests in real time, and this is where you can leverage the power of LLM because the test plan maintenance and creation can be really painful, but

s131
00:11:01.040 --> 00:11:13.160
the value of human in the loop here is the governance and the review part, and the part where you're reviewing the test plan and not the code.

s132
00:11:13.160 --> 00:11:14.000
Right?

s133
00:11:14.000 --> 00:11:17.640
I know it's been like over 20 years we came up with test-driven development.

s134
00:11:17.640 --> 00:11:28.040
This in some ways is closer to behavior-driven development, where the test plan is now something which even you can share with your product managers, your designers.

s135
00:11:28.040 --> 00:11:31.680
Everyone can participate because these things are now in English.

s136
00:11:31.680 --> 00:11:43.120
At the same time, we have now deterministic verification, which actually verifies whether the particular test criteria has been met or not.

s137
00:11:44.000 --> 00:11:46.800
Let's move on.

s138
00:11:47.200 --> 00:11:53.720
So, this is where I would say the system is not supposed to be perfect.

s139
00:11:53.720 --> 00:11:58.760
It's deterministic where it can be, but LLM where you must.

s140
00:11:58.760 --> 00:12:03.920
Not everything can be deterministic.

s141
00:12:03.920 --> 00:12:11.960
Not every system can be built in a way which is 100% built on deterministic systems.

s142
00:12:11.960 --> 00:12:16.200
This is where you use LLM as a fallback.

s143
00:12:16.200 --> 00:12:17.400
Let me give an example.

s144
00:12:17.400 --> 00:12:25.280
Like if you're making a change in your web application, what you can do is it creates a test plan of what the behavior changes.

s145
00:12:25.280 --> 00:12:28.200
Let's say you introduce a new payment form.

s146
00:12:28.200 --> 00:12:30.040
So, it creates a new payment form.

s147
00:12:30.040 --> 00:12:34.839
The verification here is does the payment system changes?

s148
00:12:34.839 --> 00:12:45.760
Uh an AI agent can go and browse through your application to fill out a form and capture screenshots as evidence.

s149
00:12:45.760 --> 00:12:52.040
And then take those screenshots as well as your database snapshots to identify whether the criteria was met.

s150
00:12:52.040 --> 00:12:57.480
So, the screenshot testing or like sand flying can then still be done by agents.

s151
00:12:57.480 --> 00:13:06.480
But at the same time, you're creating more solid evidence, which now a reviewer can look at and build more confidence that this actually works.

s152
00:13:08.400 --> 00:13:13.080
So, now reviewers are reviewing the intent, not the diff.

s153
00:13:13.080 --> 00:13:15.720
You're reviewing the intent decisions.

s154
00:13:15.720 --> 00:13:18.400
Uh what we said to build out.

s155
00:13:18.400 --> 00:13:22.520
What we tried and rejected, and capture all of these things from the sessions.

s156
00:13:22.520 --> 00:13:25.880
Remember, capturing it from the sessions is is key.

s157
00:13:25.880 --> 00:13:32.920
If we try to build it from the code, you'll end up in the same situation that we were talking about I think Dex was talking about yesterday, which is

s158
00:13:32.920 --> 00:13:42.400
if your code is built by the same agent which is actually building a test plan, it's not going to build a test plan which will actually catch issues.

s159
00:13:42.400 --> 00:13:48.480
So, that's why it's important to actually use the session information to build out a test plan.

s160
00:13:48.480 --> 00:13:50.600
You can discuss architectural decisions.

s161
00:13:50.600 --> 00:13:53.960
So, how you're creating the data models, how these services interact with each other.

s162
00:13:53.960 --> 00:13:56.520
So, you've moved one level above.

s163
00:13:56.520 --> 00:14:05.000
So, you're essentially instead of like reviewing line by line, you're actually having discussions on architecture, which are very critical for any kind of collaboration.

s164
00:14:05.000 --> 00:14:10.880
And then you look at the evidence, everything that was collected from the verification.

s165
00:14:12.880 --> 00:14:16.040
So, here's a homework for everyone.

s166
00:14:16.040 --> 00:14:25.280
Go home and mine your last 1,000 review comments and build out a AI slot register for the things which are repeatable.

s167
00:14:25.280 --> 00:14:33.920
So, a vast majority of the comments that you're providing in your code review are something that we repeat over and over again.

s168
00:14:33.920 --> 00:14:37.320
This compounds with every merge PR.

s169
00:14:37.320 --> 00:14:43.040
Every time you capture something as a register, you don't have to capture that comment again.

s170
00:14:43.040 --> 00:14:57.280
And this is where you can actually codify some of the best practices of doing code, maintain semantic accuracy, at the same time, do not lose the collaboration part of the review.

s171
00:14:58.400 --> 00:15:00.200
It does follow a J curve.

s172
00:15:00.200 --> 00:15:01.720
So, pain is real.

s173
00:15:01.720 --> 00:15:08.600
Uh you will have to spend some time to actually make it pay off because initially creating a registry can take some time.

s174
00:15:08.600 --> 00:15:13.560
And this is where I would recommend you folks can come and try out our product.

s175
00:15:13.560 --> 00:15:16.760
So, code review is not just about code review.

s176
00:15:16.760 --> 00:15:20.600
It is about uh really getting the alignment.

s177
00:15:20.600 --> 00:15:29.280
And where we can build better tools is creating the semantic accuracy and defining your AI slot register.

s178
00:15:29.280 --> 00:15:38.280
So, if you remember one thing from today, remember code review is not just about code review, it is about getting the alignment.

s179
00:15:38.640 --> 00:15:43.320
And yes, we are piloting our new product called Verify.

s180
00:15:43.320 --> 00:15:46.640
Please join and be our early design partners.

s181
00:15:46.640 --> 00:15:52.120
We are working with a few companies to uh pilot out a new verification system.

s182
00:15:52.120 --> 00:16:04.200
This combines both the alignment side of things as well as building tools and capabilities for detecting semantic accuracy using the AI slot register.

s183
00:16:04.200 --> 00:16:04.800
Thank you, everyone.

s184
00:16:04.800 --> 00:16:06.136
Thanks for joining.

s185
00:16:06.136 --> 00:16:08.136
[applause]