ReviewDebt: a practical framework for scoring every pull request — Sachin Gupta, Ebay

AI Engineer · 25 min · 370 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Hi everyone.
00:01I'm Sachin Gupta and I'm a software engineer.
00:03The title is exactly what it sounds like.
00:06Your coding agent is creating review debt.
00:09Before I start, sit with the title for a second.
00:12Notice what it is not saying.
00:14I'm not saying that coding agents are bad.
00:17I'm not saying they don't make us faster.
00:20I'm saying they they're creating a kind of debt that nobody is measuring.
00:25And that debt is going to come to you.
00:28Over the next few minutes, here's what I'm going to do.
00:31I will define what review debt is.
00:33I'll walk five signal families that are going to compose it.
00:37I'll score three real pull requests side by side and I will show you a cross repo scan of over 500 PRs, which is nothing but three public code bases.
00:45So let's go.
00:49Look here.
00:50Here's the gap nobody's measuring.
00:52GitHub's 2025 October's report that covers almost every public pull request on the planet shows that the commits climbed 25% year over year.
01:03And now over the same year, comments on the comments dropped 27%.
01:07Now, these comments are nothing but the proxy for review activity.
01:11Code production volume actually went up, but the review attention went down.
01:16They moved in opposite directions that to in the same year.
01:19Now, look at the teams for this along the AI adoption curve.
01:23Faros AI tracks this cohort in their 2026 benchmark.
01:28Median PR review time is up by 441.5% and if you see like if you calculate, you'll figure out that the reviewed PRs take 5.4 times longer than what they used to.
01:41And then 31% more PR are now merged with no review at all.
01:46So AI produce AI is producing the code very fast.
01:49AI is producing the pull request very fast, but humans cannot responsibly review them at that pace.
01:56This gap is called as review debt.
01:59It actually accrues quite a bit.
02:01It compounds and right now nobody has a number for it.
02:05But by the end of this talk, I will make sure that you will have a certain number.
02:12Now, if you look at this particular slide, here is the story every team is telling right now.
02:17PR per developer are up 16%.
02:20That's the Faros AI acceleration Whiplash benchmark and that from April 2026.
02:27There were like 22,000 developers, 4,000 teams.
02:30And the median PR size is up 63%.
02:3444 lines to 72 lines per pull request.
02:38That is basically 16 months of long-term data from DX 2026 study, which includes 400 organizations.
02:46Now, if you see the cycle time, open to merge, that is modestly down and it is coming from the same DX study.
02:54The framing is really generous.
02:55PR throughput grew actually about 8% and the AI usage rose about 65%.
03:02The gain that we see today is real, but it is actually smaller than the hype that that is present.
03:08Every one of these numbers are real.
03:10None of them is a lie, but everyone is a vanity metric.
03:14PR count goes up when one PR splits into seven PRs.
03:18Median PR size going up is not a benefit, it's actually a bloating.
03:22Cycle time going down when reviewers stop pushing back.
03:25These things tell you the speed of production.
03:28They do not tell you the speed of trust.
03:32Now, we will go into the next slide to see more.
03:42Now, if you go looking for the second story, here is what you're going to find.
03:47Reviewer fatigue.
03:48Engineers carry far more review load than a year ago.
03:52They're not happier about it.
03:54Late-night merges.
03:55PRs are sitting unreviewed for 3 days, 4 days, and then they suddenly get a thumbs up at 11:00 p.m. before a Friday deadline.
04:03Test theater.
04:05Tests are getting added, but they're actually asserting what the code did, not what the code should do.
04:11They lack in behavior, including bugs.
04:15Architectural drift.
04:16Again, the same problem that is solved in three different ways in three different files.
04:21Nobody is actually holding up the architectural trade.
04:25Finally, incident lag.
04:27When something breaks, the bug lags week or months after month uh after merge.
04:33Nobody connects the dot back to the AI authored change.
04:37These costs don't show up in any dashboard until now.
04:42We'll look into the next slide to know more about it.
04:46So now, let me give you the definition.
04:48Review debt.
04:50It's the accumulating gap between code your agent has produced and the code humans have actually reviewed, trusted, and understood.
04:59It rhymes with technical debt, but it is more like a financial debt because it compounds.
05:05It actually accrues interest, but this interest is not money.
05:08This interest is paid in human attention.
05:10It compounds because of the three feedback loop.
05:13First, the agent learns from your code base, fine-tuning, rag grounding, in-context suggestions.
05:19Code that was not deeply reviewed yesterday grounds tomorrow's PR.
05:24This debt becomes generative in nature.
05:27Second one, reviewers seed the architectural call.
05:31When most of a PR was generated, attention contracts to syntax and obvious bugs.
05:36Big picture decision moved from review time to never.
05:40Third one, velocity expectations reset.
05:43Once leadership sees the new throughput, you don't get to hire reviewers in proportion.
05:48There is no slack left to pay the debt back.
05:51Each loop on its own is survivable.
05:54Together, you will have a runway, but the next question is, how are you going to measure it?
06:00So, how do we measure review debt?
06:03There are five signal families that has like 10 deterministic checks.
06:07Remember, the keyword here is deterministic.
06:09Every check is computable from a pull request and its repository.
06:14No, we don't need any language model.
06:17Why?
06:18Because LLM, when it acts as a judge, it possibly can break two things.
06:23Number one, the score becomes a moving target.
06:26The same PR, basically the score that you got for the same PR, will score differently when your model will change.
06:34Two, the score stop being defensible in an engineering review.
06:38You cannot put it on a slide.
06:40You cannot put it in front of a leadership.
06:43So, what do you want?
06:44You want a number that is traceable to a deterministic computation, different size and coupling.
06:51Second, test evidence gap.
06:53Third, directory and ownership spread.
06:56Fourth, AI authorship indicators.
06:59Fifth, evidence and rational gaps.
07:02These are the five families that will have 10 checks under them.
07:08Let's walk each of them.
07:11The first one, the first signal that we have is dev size and coupling.
07:16This is the simplest one to calculate, basically.
07:18It's also the most often misread.
07:21What it measures is the net lines changed, the files touched, whether changes cluster in one module, or is it sprawling across many.
07:29Why the agent is struggling here?
07:31Agents are biased toward fix at the call site.
07:34A human engineer routes a fix to the root cause.
07:37That keeps the difference very small.
07:39But an agent will reach into many files looking for the same symptom.
07:43The review cost of a sprawling difference is not proportional to the size.
07:48It is actually much steeper.
07:50So, the cross-file coupling explodes the mental model and reviewer has to hold.
07:56We'll look into the second signal, which is test evidence gap.
08:03So, what is test evidence gap?
08:05Test evidence gap is nothing but test lines added divided by the production lines added per pull request.
08:11So, AI authored PRs, they ship with a far lower test to code ratio.
08:16Sometimes there is no test file in place.
08:19And if there is any, it's minimal.
08:22So, this gap is actually a consistent gap.
08:24And the problem with this particular thing is the number The reason this number is basically brutal is the agent generate test.
08:31They generate a lot of test.
08:33But these test basically assert what the code is doing.
08:36It base It does not generate the test cases what the code should actually do.
08:42They lock in the behavior including the bugs.
08:44The ratio does not capture the quality gap.
08:47It just measures whether the test showed up at all.
08:51So, the deeper signal is the next step that is a layer down.
08:56We'll see the next one, which is directory and ownership spread.
09:03So, what is directory and ownership spread?
09:06Count the distinct code owner teams whose file appear in the difference.
09:10Now, what does it mean?
09:12Basically, a well-shaped PR concentrates in one team directory.
09:16But a sprawling one reaches across many team file.
09:19And the review cost is enormous in this scenario.
09:22No single human hold the whole mental order model.
09:26You need multiple approvals from multiple engineers in multiple different context.
09:31The coordination overhead easily exceeds the time the agent save producing the code.
09:37This is where you start to feel the economics of review debt.
09:40The agent gave you three hours of typing.
09:43You spend those three hours by multi-party reviewer attention.
09:48And the next one we are going to see is AI authorship indicators.
09:53So, what is AI authorship indicator?
09:56Before everyone gets defensive, this is not for the blame.
09:58We are not flagging that engineers use coding agents.
10:02We wait the scores for a PR shape like agent assisted authorship get extra reviewer attention.
10:07So, that way you know, right like, okay, this is the PR that is directly coming from the agent.
10:11And there are like the very common three detection mode.
10:14I must I I think everyone must have seen it by now.
10:18One is the co-authored footer, which is basically one of your strongest signal.
10:21You see that co-authored by Copilot.
10:24Second one is the branch name pattern.
10:26Generally, you must have seen it says codex, Copilot, cursor prefixes.
10:30Third one is basic generated by assisted by, which is nothing but your PR body.
10:35your commit message phrases.
10:36We did actually a real data check on three public repos, which comprised of 524 PRs.
10:42I'm not going to name the company, but for the sake uh we are just calling it A, B, and C. Now, if you see the highest signal was the co-authored footer.
10:52And the second one uh the lowest signal was 0% on the repo C. Even though it was done by coding agent, but they might have blocked it uh saying any co-authored
11:01or uh like generated by etc. etc. The next thing that we are going to see is evidence and rational gaps.
11:12So, evidence and rational gaps.
11:14I think this is one of the most deterministic that I personally feel.
11:19It's the one that destroys review ability the fastest.
11:23What it measures basically?
11:24Does the PR explain the why or just the what?
11:28On the left, if you see, we have an high gap.
11:30On the right, we have a low gap.
11:32The title says "Fix leaky test" and all the data that you're seeing here is actually from some public repos.
11:40Now, the PR body length is 18 characters, the commit message says updates.
11:44Obviously, I will not be if you give it to me, I won't be able to review this.
11:48Okay, I cannot accept this.
11:50But then on the right, if you see when there is actually a low gap, the title actually is telling you what the change is all about.
11:56The body has a symptom, the diagnosis, the change, and the link to the benchmark.
12:01Now, a reviewer can do the job.
12:04So, in our regression fixtures and hands code teams, this signal destroys reviewability fastest.
12:11In the real open source data, PR bodies follow conventional commit format.
12:16So, it fires most rarely there.
12:18But if you see the regression fixtures are just the demos.
12:21Okay, let's see how those five signals combine.
12:25You will only have one number that is going from zero to 100.
12:28And the exact weights are basically your defaults.
12:31Now, if you're planning to adopt this, I would like you to do this first.
12:35Run it backwards over the last 200 PRs that have been merged in your company.
12:39Calibrate the weights against your team's actual reviewer experience.
12:43The score has to feel right against your actual guts.
12:46We you you are going to divide it into four bands.
12:49If you are zero to 24, which means you have very low review burden.
12:54If you have 25 to 49, you are normal, proceed with the standard care.
12:5952 74, you need evidence from the author before senior review.
13:03And if you're 75 and above, definitely it's high split or request more context.
13:08Same shape as technical debt categories, but the unit is different.
13:13Now, we are going to see what the scanner actually did.
13:23Now, if you see this is the clean PR.
13:27Unfortunately, I won't be able to give you a live demo today, but I will walk you through a CLI.
13:32I wanted you to show three scored pull requests side by side, the level where the framework become useful as a repeatable review conversation.
13:41This is the first one.
13:43If you see, this is what the scanner says when a PR is all changed.
13:46Score out zero out of 100.
13:48The burden is basically nil.
13:51It's zero.
13:51We have a low review burden.
13:53The estimated minutes is six.
13:55No checks were fired.
13:57And if you look the structure here, that is why the list is empty.
14:02Reviewer focus is empty.
14:03Author next action is empty.
14:05The framework only generate advice when it has something specific to say.
14:10A healthy PR produces a tiny, almost ceremonial report.
14:14That's exactly what you want.
14:16And the takeaway is most healthy PRs produce zero noise.
14:20They get the work done without making any noise.
14:23This is not a tool that complains by default.
14:27Running it on every PR cost you one comment that says look good.
14:31That's it.
14:33Let's see the high debt PR now.
14:37Now, you see what you see actually on the screen is from a public repo.
14:43This is a real scenario from the scanner regression suite.
14:48If you see, the score is here 60 out of 100.
14:51It requires evidence.
14:54It's on the audit and it says needs evidence.
14:5686 estimated minutes of review effort.
15:00Look at the structured output.
15:01It's actually not a score.
15:03A wire list explaining what fired.
15:05A reviewer focus list telling the reviewer what to do next.
15:09An author next action list telling the author how to bring the score down.
15:14This is the part most PR quality tool miss.
15:17The score alone is useful.
15:19The structured advice is what actually moves the team behavior.
15:23Now, if you read the bullet point, it says PR has soft indicators of AI assisted authorship.
15:29This is information only, not a definitive claim, and not a penalty on its own.
15:37This particular step sentence is in the scanner output.
15:42The AI indicator check contributes five of the 60 points, which is nearly about 8%.
15:49The other 55 came from a different size claim mismatch and missing test.
15:54Those would be high burden signals on any PR, agent authored or not authored.
16:00This is not an anti-AI scorecard.
16:02Basically, this is more of a review burden scorecard.
16:05The agent did not cause the score, but the shape of the PR that is created by the agent did this.
16:12Let's see the next one.
16:20This is a PR, basically, which is AI authored.
16:24It's very well shaped, and the score is seven, seven, which is low review burden.
16:29Now, if you see the same agent, it has done the same kind of work, but the test were added.
16:34So, the CI is green.
16:36The risky path is called out in the PR description.
16:39That's why we got the score as seven out of 100.
16:43It's very low burden, took only 14 minutes.
16:46The AI indicator check still fires, but it contributes 2.1. One, low severity.
16:52Second, the framework explicitly says, quoting the report itself, "Information only, not a definitive claim, and not a penalty on its own."
17:02So, when your team is going to ask, "Will this tool penalize us for using coding agents?"
17:08No, the answer is on your screen, low review burden.
17:12It's a very well shaped AI PR.
17:15We are going to see the next one, which is basically our three public repos over 524 PRs.
17:22I wanted to show you a 90-day AI search slope across three public repos and 524 pull request.
17:29The review burden climbs even when the AI authorship doesn't.
17:34If you see this particular scan, we observe three things.
17:37One of them is the volume is the actual variable.
17:40AI authorship was flat and it was 5 to 20% steady state, but the review burden was not.
17:46If you see one repo accumulated 186 senior reviewer hours in 27 days and another one took 43.
17:54The window length is same, but the volume was not the same.
17:57The burden was not the same.
17:59That is what PR volume is.
18:01Second is amplifier only holds up under real data, which means AI indicators fired on 5 to 20% of PRs every week across all three repos.
18:11None of those AI authored PRs disproportionately landed in the high burden band.
18:16The positioning contract holds against real code bases.
18:19Third one is complexity drives burden, not authorship, which means across 524 PRs four landed in needs evidence or high bands, which means all four were structural changes, which were large migration, SDK rewrites, multi-team refactors.
18:35The framework score complexity fairly.
18:37AI driven volume creates the condition under which these accumulate.
18:42So now, what 524 PR look like under the lens?
18:46Where does the review debt actually show up?
18:49In absolute terms, the scanner saw 524 real pull request in three public repos, 228 senior reviewer hours cumulative across three public repos that were scanned over 27 to 90-day windows.
19:04Second one is nine PR per day sustained merge rate the high velocity repo.
19:10So, if you see, it's all about the volume that is basically driving the burden.
19:16Third one is 5,036.
19:18These are the minutes that are basically spent on a single PR, which is nothing but 84 hours of review effort estimated for one single pull request.
19:31The score that we received there was 73, which said it requires an evidence band.
19:38Now, 5 to 25% of these PR every week across every week fired at AI indicator signal.
19:45Again, remember this is volume, which is causing the review debt, which is changing the game altogether.
19:55We'll move to the next slide.
19:57Measure the problem.
19:58We have shown the cost.
20:00Now, what do we do about it?
20:02So, first thing is you should have one logical change per PR.
20:06I'm not saying like not small PR in the abstract.
20:09One logical change, that is enough.
20:11Second, test ship with the change.
20:14Even if the agent wrote the code, even if the agent wrote the test, the human author confirms the test assert what the code should do, not what the code is doing, what the code is supposed to do.
20:26Okay?
20:27Third, stay in one owner territory.
20:29Cross-cutting work splitting into per per team PR.
20:34Remain in your own territory.
20:35One approval, one context, one mental model.
20:38Author writes the why.
20:40The agent should not write the PR body.
20:42That's the moment the human author commits to understanding what they are actually shipping.
20:47Same review standard for the AI PRs as human PRs.
20:51The AI indicator amplifier only fires when other signals are weak.
20:57Keep the other four strong.
20:58The amplifier is invisible.
21:00So, make sure there is no exception for AI.
21:04None of these is actually requiring you to have a new tool.
21:08These are the moves that you already are aware of.
21:10You just need to implement it.
21:15Now, how are you planning to adopt this?
21:20If you see, we again have five steps.
21:22Backfill, threshold, surface, aggregate, and talk about it.
21:27Backfill.
21:28So, you run the scorer over your last 200 merge PR.
21:32Look at the one basically that has the highest score.
21:36Second, threshold.
21:37Set a basically justify line.
21:39Default, let's say you want to give it 50.
21:42So, any PR scored at 50 or above requires a comment from the author.
21:45As simple as that.
21:46Surface it.
21:48Post the score as a PR comment on every PR.
21:50Don't block it.
21:52Just for the visibility purpose, surface it so that everybody should know what's happening over there.
21:57Aggregate weekly per team.
21:59Which means the slope of each team deadline is the leading indicator.
22:03That's what your engineering manager should watch.
22:06Talk about it.
22:07Bring the number to a retrospective, every roadmap view.
22:12In every meeting, you should discuss about that number.
22:15Like, what was your score when you did this particular PR.
22:19That's it.
22:26Why this number matter?
22:27Because of this conversation?
22:29Absolutely not.
22:31Without a number, what you are just conversing is why.
22:34But if you have a number, it's actually structured.
22:37You can tell like, okay, with AI coding agents, we have a rollout which has added X percentage of the throughput.
22:44But you need to make sure you are also adding the Y points of the review debt that has been occurred over Z weeks.
22:51Now, at your current slope, that's roughly N senior N senior engineer hours that you need to talk about.
22:58That moves the discussion from feeling to measurement.
23:01So, you should always have a number.
23:04If you see 2026, we are adopting things.
23:07But, 2027 is the year when this conversation will shift to the governance model.
23:12And, it's actually starting right now.
23:15Can you trust the code that you're shipping?
23:17If yes, we are good.
23:19If no, then who is accountable when an AI authored change causes an incident?
23:24Where is the audit trail?
23:25So, reviewed it is basically the bridge between these two columns.
23:29It's the first number that lets you have the right-hand side conversation without abandoning the left-hand side gains.
23:38We'll look into the final slide now.
23:44Now, three things I would like you to take away.
23:48First one is, measure the gap.
23:50Don't build the tooling yet.
23:52Take 20 PRs from the last week, score them using the five signals that we discussed.
23:58You will basically get a number so that you will understand, okay, whether the framework is working right or not.
24:03Try to make the slope visible.
24:05The slope of a reviewed it over time matters most than the level.
24:09So, plot it weekly, and then discuss it with your team maybe in a meeting or in a sprint planning, whatever the way you feel like.
24:16Have this conversation.
24:17Bring the number to your next review.
24:20Like, say that, okay, we do have extra report, wide it, and this is the slope.
24:25And, then you are basically moving away from the wide culture to the measurement culture.
24:30One quick callout on anti-patterns to avoid.
24:33First thing is, approve with comment merges.
24:36The AI did the boring part.
24:38No. We'll catch it in QA.
24:40No. PRs smaller now.
24:41No. Stop writing LGTM on the PRs.
24:46Make sure whatever we have discussed, the habits that we have discussed, we are going to follow them.
24:53This is the entire point of the talk.
24:55Thank you very much for listening.