{
  "video": {
    "id": "4VmugsE-iP0",
    "title": "How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs",
    "duration": 803,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "4VmugsE-iP0",
    "title": "How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs",
    "one_liner": "Brainbase took a Domino's competitor-pricing agent from 41% to 97% accurate over 11 versions, while cutting cost from $2 to $1 per source and run time from 129 to 71 minutes, by changing only instructions, skills and playbooks rather than the model.",
    "summary": "Milind Sunil Shah walks through how Brainbase hill-climbed a back-office agent for Domino's CMO office that fills a spreadsheet with the lowest competitor prices across websites and delivery apps. He presents a repeatable six-step framework: kickoff, ontology, benchmarks and evals, first agent, iterating, and handing off. Most of the gains came from defining terms, turning real failures into rules and benchmark cases, starting with a very narrow scope, and scoring outputs with code, an LLM judge and an agent judge. He argues you should hand off the entire improvement loop to the customer, not just the agent.",
    "key_points": [
      "Results from V1 to V11: accuracy rose from 41% to 97%, cost fell from $2 to $1 per source (one item, per competitor, per website), and run time fell from 129 to 71 minutes, all through changes to instructions, skills and playbooks.",
      "The manual baseline was an analyst spending 2 hours daily actively building the spreadsheet; each step looks easy, but one mistake produces a cell that looks finished and is just wrong.",
      "The kickoff brief agrees what a win looks like: the job today, what the human signs off on, time, cost, required accuracy, who judges the output, and what the agent must never do alone.",
      "The ontology gives the agent a Postgres-like memory with a targets table, a price observations table recording the seen price, full price and source, and a spreadsheet of placeholder item IDs to replace, so a wrong cell traces back to the exact page and rule.",
      "Rules came from real failures and live traces: record the price paid at checkout (19, not the struck-out 24), define what counts as a large pizza, treat delivery and takeaway as different, do promotion math before entry, and log anything unverifiable as a gap rather than guessing.",
      "Evals use three tiers: code for deterministic checks (every ID replaced, discount math, flagging of prices 20% above or below the previous run), an LLM judge reading the trace for correct item and promo, and an agent judge that takes multiple turns and tool calls to confirm whether each flagged price was valid.",
      "To build an offline benchmark they tried archive.org snapshots of competitor sites, which failed for delivery apps, so they stored browser screenshots of everything the agent did from version one onward.",
      "A promo page with prices inside 26 images sent the agent into a zoom loop (4x, 5x, 6x) that found the right card at minute 13 and was manually stopped at minute 15; a rule to check the HTML DOM, image titles and alt text first and stop past a certain zoom level gave 1/3 the image reads, over 50% lower cost, and a finished report."
    ],
    "takeaways": [
      "Write a definition for every term the agent relies on, along with what to collect, where it comes from, where it goes and what to do when stuck, because anything left unwritten gets guessed.",
      "Ship a narrow first agent you can score from day one (Domino's began with one app, one item, one city) and widen scope only after you have a baseline.",
      "Iterate on the most expensive and most frequent failures, change one thing at a time, fix instructions before changing the model, and turn each failure into a benchmark case or eval criterion.",
      "Score as much as you can with code, an LLM judge and an agent judge, but still have engineers read traces and browser tool calls, because some failures only show up there.",
      "Hand off the whole loop by writing down who runs the agent, who reviews outputs, who converts corrections into rules and benchmark cases, and who decides a new version ships, and let users make small changes themselves."
    ],
    "topics": [
      "agents",
      "evals",
      "benchmarks",
      "hill climbing",
      "browser agents",
      "ontology",
      "back-office automation",
      "agent handoff"
    ],
    "tools": [
      "Brainbase",
      "Domino's",
      "Postgres",
      "archive.org"
    ],
    "quotes": [
      {
        "text": "If you don't write something down, the agent's going to guess and probably get something wrong.",
        "at": "04:02",
        "url": "https://www.youtube.com/watch?v=4VmugsE-iP0&t=242s"
      },
      {
        "text": "Version one's job is to be measured, it's not to be good.",
        "at": "08:05",
        "url": "https://www.youtube.com/watch?v=4VmugsE-iP0&t=485s"
      },
      {
        "text": "you're trying to hand off an entire loop and not just the agent.",
        "at": "12:16",
        "url": "https://www.youtube.com/watch?v=4VmugsE-iP0&t=736s"
      }
    ],
    "words": 2959
  },
  "summary_url": "/#4VmugsE-iP0",
  "transcript": {
    "html": "/transcripts/4VmugsE-iP0.html",
    "txt": "/transcripts/4VmugsE-iP0.txt",
    "vtt": "/transcripts/4VmugsE-iP0.vtt"
  }
}