{
  "video": {
    "id": "_mi3alkqy4s",
    "title": "The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI",
    "duration": 1481,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "_mi3alkqy4s",
    "title": "The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI",
    "one_liner": "AI agents have made writing code nearly free, but trusting it hasn't got any cheaper. Voss argues that humans should stop reading PRs line by line and put their judgment into building review harnesses, rubrics and evals, because skipping review entirely has repeatedly gone wrong.",
    "summary": "Voss (head of developer relations at Arize AI, co-founder of npm, Inc.) argues that code review is now the bottleneck. One study found developers using autonomous agents wrote 741% more code but shipped only 30% more software. Humans can't simply review harder, and taking humans out entirely has failed in practice. Passing tests is a poor stand-in for mergeability: models score far lower when graded on whether a maintainer would actually merge the change. Production automated reviewers like Copilot, Cursor and CodeRabbit are tuned to win human acceptance, and they can be fooled by confidently framed bad code. His conclusion is that code review isn't dead but is being rebuilt as an engineered system, with humans moving from reading code line by line to piloting that system, and production monitoring as the last reviewer standing.",
    "key_points": [
      "Three economists tracked more than 100,000 GitHub developers against AI-usage telemetry. Those who turned on autonomous agents wrote 741% more code but shipped only 30% more software, and the authors name review as the bottleneck.",
      "A Cisco study of 2,500 reviews and 3.2 million lines over 10 months found reviewers stop finding defects effectively beyond about 400 lines per sitting, and their effectiveness falls off a cliff above 450 lines an hour. At that pace a 10,000-line agent PR needs 3-4 working days of real human review.",
      "In a METR study, maintainers of the projects SWE-bench tests against judged PRs that had already passed SWE-bench. Only about half were good enough to merge. The failures were about code quality and changes that quietly broke code outside the test suite, not correctness.",
      "Cognition's FrontierCode benchmark asks 'would you merge this?' More than 20 maintainers built 150 tasks, each over 40 hours of expert work, graded on behavioral correctness, regression safety, scope discipline, test quality and maintainability. Fable 5 scores 88% on SWE-bench Pro but only 29% on FrontierCode's hardest slice, and GPT 5.5 scores under 6%.",
      "Citing Sarah Guo: compilers and test suites are free verifiers, and anything cheap to verify gets trained against until it's beaten. A mergeability benchmark would therefore become a training signal immediately. OpenAI's 2024 CriticGPT is the precedent: a model trained to catch bugs in model-written code, where humans working with it beat both the model alone and humans alone.",
      "Automated review is already mainstream. Copilot's reviewer has done 60 million reviews and handles more than one in five reviews on GitHub, and CodeRabbit has reviewed over 13 million PRs. Cursor's first reviewer ran 8 passes with the reviewer order shuffled to filter out false positives; a Peking University team found multi-pass agreement raised review quality by up to 44%. Cursor had to tell its rebuilt agentic reviewer to be suspicious by default, it now spawns fix agents from its findings, and its resolution rate rose from 52% to over 70%. All of these vendors optimize for human acceptance.",
      "Removing humans has failed or hidden costs. Dexter Horthy retracted his 'don't read the code' advice: 'We tried not reading the code for like six months. It did not end well.' Bun's agent-written million-line Zig-to-Rust port passed 99.8% of tests but has 13,044 unsafe blocks, versus about 74 in a comparable human-written codebase. Nicholas Carlini's 16-agent C compiler had no human in the loop, but had a human on the loop who wrote the harness. OpenAI's no-manually-written-code project (about 1M lines, about 1,500 PRs, 3 engineers) still needed humans cleaning AI slop by hand every Friday until they trained agents to do it.",
      "Automated reviewers are easy to fool. Anthropic's security review action warns it is not hardened against prompt injection and should only review trusted PRs. A March study found vulnerable code with an innocent commit message fooled an autonomous review agent 88% of the time, against 35% for human reviewers."
    ],
    "takeaways": [
      "Stop reviewing PRs line by line; Voss calls that the wrong level of abstraction for 2026. Spend your time building a reliable review harness instead.",
      "Write down what 'mergeable' means for your codebase, including your definitions of good, company context and domain knowledge, so agents can check against it.",
      "Use multiple review passes and keep only the findings they agree on, which cuts false positives. Tell automated reviewers to be suspicious of code by default.",
      "Keep humans at the checkpoints where correctness isn't cheaply checkable, where the blast radius is large, where security matters, or where someone has to put their name on the result. Only let automated reviewers review trusted PRs.",
      "Treat production as the last reviewer. Monitor what shipped code actually does step by step against the real world, not just whether its tests passed."
    ],
    "topics": [
      "code review",
      "coding agents",
      "benchmarks",
      "evals",
      "review automation",
      "prompt injection",
      "observability",
      "developer productivity"
    ],
    "tools": [
      "Arize AI",
      "npm",
      "Anthropic",
      "Fable 5",
      "Stripe",
      "Bun",
      "Zig",
      "Rust",
      "Ruby",
      "OpenClaw",
      "OpenAI",
      "Codex",
      "CriticGPT",
      "GPT 5.5",
      "METR",
      "SWE-bench",
      "SWE-bench Pro",
      "Cognition",
      "Devin",
      "FrontierCode",
      "GitHub Copilot",
      "Cursor",
      "CodeRabbit",
      "Greptile",
      "Graphite",
      "Cisco",
      "Linux kernel"
    ],
    "quotes": [
      {
        "text": "producing code is suddenly a whole lot cheaper. Knowing whether to trust it is still very expensive",
        "at": "02:19",
        "url": "https://www.youtube.com/watch?v=_mi3alkqy4s&t=139s"
      },
      {
        "text": "whoever writes today's review standard is writing next year's default model behavior",
        "at": "10:17",
        "url": "https://www.youtube.com/watch?v=_mi3alkqy4s&t=617s"
      },
      {
        "text": "Please please read the code. We tried not reading the code for like six months. It did not end well.",
        "at": "19:02",
        "url": "https://www.youtube.com/watch?v=_mi3alkqy4s&t=1142s"
      },
      {
        "text": "stop reviewing PRs. Um it is the wrong level of abstraction for 2026.",
        "at": "23:32",
        "url": "https://www.youtube.com/watch?v=_mi3alkqy4s&t=1412s"
      }
    ],
    "words": 5044
  },
  "summary_url": "/#_mi3alkqy4s",
  "transcript": {
    "html": "/transcripts/_mi3alkqy4s.html",
    "txt": "/transcripts/_mi3alkqy4s.txt",
    "vtt": "/transcripts/_mi3alkqy4s.vtt"
  }
}