{
  "video": {
    "id": "4UtrRVtUN_g",
    "title": "How to Build an AI Code Review Agent - Daksh Gupta, Greptile",
    "duration": 1107,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "4UtrRVtUN_g",
    "title": "How to Build an AI Code Review Agent - Daksh Gupta, Greptile",
    "one_liner": "A naive review agent (a PR webhook feeding a coding agent in a sandbox) misses bugs, lacks context, costs $3–25 per PR and posts nitpicks. Greptile's fixes are a prompt targeting your own historical bug types, an eval set of past incident PRs, model inversion (review Claude-written code with GPT and vice versa, which catches 10% more), a codebase knowledge base, autonomous sandboxed testing, and cheap models for the long tail of tokens.",
    "summary": "Daksh Gupta, co-founder of Greptile (about 1% of all GitHub PRs go through it), walks through what Greptile tells companies that build their own review agent. The baseline is simple: a sandbox, a model-independent harness, a PR webhook and GitHub's review APIs. It then fails in predictable ways: missed bugs, missing context, high cost, nitpicky comments, and new bugs found on every re-run. He goes through the fixes one at a time: prompt tuning, a historical PR dataset, model inversion, codebase context through a living wiki, end-to-end testing with vision models in dependency-rich sandboxes, and routing cheap models to minor subtasks.",
    "key_points": [
      "Baseline setup: a sandbox (bought, or Firecracker), an agent harness (Codex or Claude Code, but he recommends model-independent open-source harnesses like OpenCode or Pi), a GitHub/GitLab PR webhook, and replies posted through GitHub's review APIs. Feedback arrives about 5 minutes later.",
      "Naive review agents cost between $3 and $25 per pull request; 'these tokens can add up really really fast.'",
      "Nitpicks do double damage: they are 'like a boy that cries wolf' for humans, and coding agents fix every comment without discretion, leaving 'extremely painfully defensive code.'",
      "By Greptile's data, only about 1 in 1,000 PRs has a catastrophic bug. Because they are so rare, you can't tell from day-to-day use whether a new agent version silently misses them. You need an eval set of historical incident-causing PRs, kept updated in real time because the kinds of bugs change over time. 80–90% of bugs repeat earlier mistakes, but coding agents increasingly make new ones.",
      "Model inversion: Claude models are about 3x more likely to produce off-by-one errors than GPT models, and GPT models are much more likely to produce auth bypasses. A model reviewing its own output fails the same way, and flipping models gives a 10% higher catch rate. About 80% of PRs carry a fingerprint of the agent that wrote them (Codex's codex/ branch names, Claude's 'co-authored by Claude'); the rest need heuristics, an internal catalog or dev-machine telemetry.",
      "Grep isn't a complete answer for codebase context: it's imprecise and expensive. Most coding-agent tokens go to grep, file read and cache write. Greptile builds an indexed knowledge base of each subsystem (e.g. 'this is authentication, here are the flows, a diagram, every file involved'), which matters for catching auth bypasses.",
      "Correct code can still miss the author's intent, especially with short, ambiguous prompts (e.g. 'Slack as their IDE'). The most useful context is docs and tickets (Jira, Linear, Confluence, Notion) plus the codebase index. Greptile combines all of these into a 'living wiki', a world model of the company that also draws on production telemetry and logs.",
      "Autonomous testing: Greptile spins up the customer's stack in configurable VMs and tests it end to end with computer vision, e.g. catching a button that is hidden on iPhones. It needs documented app flows and a simple build script. He says that 'end to end QA is nearly solved.' On cost, Greptile uses frontier models for about 20% of work (bug analysis, the review itself), across ~20 subtasks each with its own models. Small models handle context-finding and exploratory file reads, and a lightweight computer-use model handles UI flows."
    ],
    "takeaways": [
      "Mine your past outages and incidents to build a catalog of the bug types your team produces most. Prompt for those types specifically, and spell out what counts as a nitpick so the agent skips them.",
      "Build an eval set of historical PRs that caused incidents and run every version of the review agent against it before shipping prompt or model changes. Small prompt edits swing comment volume a lot, and catastrophic bugs are too rare to notice missing in normal use.",
      "Detect which model wrote a PR (branch names, co-author tags, telemetry) and review it with a different model family.",
      "Give the reviewer a precomputed codebase knowledge base plus access to tickets and docs, instead of letting it grep and read files freely. Use a cheap model to decide which context gets loaded into the expensive frontier-model cache.",
      "Swap simple subtasks to small models (he says this gets you 70–80% of the way), but only with a good eval set. Otherwise you won't know reviews got worse until an outage."
    ],
    "topics": [
      "code review",
      "agents",
      "evals",
      "codebase context",
      "model selection",
      "inference cost",
      "sandboxes",
      "e2e testing"
    ],
    "tools": [
      "Greptile",
      "GitHub",
      "GitLab",
      "Firecracker",
      "Codex",
      "Claude Code",
      "OpenCode",
      "Pi",
      "Claude",
      "GPT",
      "Opus",
      "Jira",
      "Linear",
      "Confluence",
      "Notion",
      "Slack",
      "Nvidia",
      "Coinbase",
      "Stripe"
    ],
    "quotes": [
      {
        "text": "coding agents are not discerning. They're going to look at review comments and just address all of them",
        "at": "05:34",
        "url": "https://www.youtube.com/watch?v=4UtrRVtUN_g&t=334s"
      },
      {
        "text": "That means a non-trivial percentage of GDP growth is grep, file read, cache write.",
        "at": "11:41",
        "url": "https://www.youtube.com/watch?v=4UtrRVtUN_g&t=701s"
      },
      {
        "text": "you'll find that if you're using the same model to review the code written by the same model, um you'll actually have relatively fewer bugs caught",
        "at": "09:38",
        "url": "https://www.youtube.com/watch?v=4UtrRVtUN_g&t=578s"
      },
      {
        "text": "For the first time, I think end to end QA is nearly solved.",
        "at": "15:16",
        "url": "https://www.youtube.com/watch?v=4UtrRVtUN_g&t=916s"
      }
    ],
    "words": 4752
  },
  "summary_url": "/#4UtrRVtUN_g",
  "transcript": {
    "html": "/transcripts/4UtrRVtUN_g.html",
    "txt": "/transcripts/4UtrRVtUN_g.txt",
    "vtt": "/transcripts/4UtrRVtUN_g.vtt"
  }
}