{
  "video": {
    "id": "nxokqOq1imY",
    "title": "Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust",
    "duration": 1453,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "nxokqOq1imY",
    "title": "Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust",
    "one_liner": "Braintrust's field CTO argues that each model-capability step function forces you to re-architect your AI system — and that your evals must be re-architected with it, moving from final-answer scoring to node-level checks to pass@k / pass^k distribution analysis.",
    "summary": "Ameya Bhatawdekar traces six generations of AI application architecture — single prompt, RAG chain, early ReAct loop, workflow graph/state machine, the revived ReAct loop, and today's 'product system' with memory, sandboxes and MCP/skills — and shows what each generation broke about the previous generation's evals. His core claim is that 'architecture follows model updates and your evals have to follow your architecture,' with evals as the durable asset that encodes how your system is supposed to work across replatformings. He closes on the production-data flywheel: most teams accept it conceptually but let evals go stagnant, so you need something that surfaces not just known failure modes but novel ones — which is what Braintrust's Topics cluster analysis is pitched as doing.",
    "key_points": [
      "Model releases are step-function changes, not incremental upgrades — reliable tool use, long context, safely executable generated code, practical memory systems — so teams are moving from iterating on their apps to replatforming them.",
      "You can't just drop in a new model: the old system encoded workarounds for the old model's limitations (e.g. logic compensating for weak tool calling), so you can't tap the new capability without rearchitecting.",
      "Everything is grounded in an SRE agent example that has both read tools and write tools — it can roll back a deployment, escalate to a human, or page someone.",
      "Generation by generation: single prompt/single call → eval final answer accuracy, factuality, hallucination against a golden dataset; RAG chain → new failure points in the parser, the retrieval, and context stuffing the model can't reason over.",
      "The early ReAct loop (popular mid/late 2023–early 2024) fell short because models of that era got tool arguments wrong, called the wrong tools, and hit context collapse; teams responded by taking control back into workflow graphs and state machines, letting models operate only at the node level.",
      "Graphs bought reliability but broke on out-of-distribution intents, and the special-case branches added to fix that multiplied failure surfaces — branch consistency, node-to-node contracts, classifier node mistakes, retry loops — forcing node-level evals on top of orchestration evals.",
      "Anthropic and OpenAI's mid/late-2025 releases (reliable tool calling, better orchestration control, planning, long-horizon tasks, introspection and course correction) left graph-based systems unable to exploit the new state of the art, so the ReAct loop came back — but with high trajectory variance: the same input yields dramatically different trajectories that still reach the right answer.",
      "That variance changes the unit of evaluation from one run to a distribution: run the eval k times and use pass@k (does it succeed at least once — a measure of capability) versus pass^k / 'pass wedge k' (how many of the k runs succeed — a measure of reliability).",
      "Today's systems are product systems, not just a model in a loop: memory storage/retrieval within and across sessions so runs learn from previous runs, robust code execution sandboxes, MCP and skill directories, and skills repositories that extend models via symbolic instructions — all of which prior-generation evals only partially cover.",
      "Braintrust's pitch: evals plus observability plus 'Topics', which runs cluster analysis over production data to surface new categories of failure that no eval or guardrail was ever written for."
    ],
    "takeaways": [
      "Treat an eval suite as versioned alongside architecture: when you rearchitect for a new model capability, rewrite the evals for the new failure surface rather than carrying the old ones forward for partial coverage.",
      "For loop-based agents with variable trajectories, stop scoring single runs — run each case k times and report pass@k for capability and pass^k for reliability, so you know whether a capable system is also a dependable one.",
      "Match eval granularity to the architecture: final-answer scoring for single calls, retrieval and parsing checks for chains, node-level plus orchestration, branching and retry-behaviour evals for graphs.",
      "Actually run the production-to-eval flywheel instead of just agreeing with it — static evals go stagnant even when your architecture doesn't change.",
      "Build a mechanism (e.g. clustering over production traces) that surfaces unanticipated failure categories, not only more examples of the failure modes you already defined."
    ],
    "topics": [
      "evals",
      "agents",
      "ai-architecture",
      "observability",
      "rag",
      "react-loop",
      "reliability",
      "production-feedback-loops"
    ],
    "tools": [
      "Braintrust",
      "Braintrust Topics",
      "Anthropic",
      "OpenAI",
      "ReAct (paper)",
      "MCP",
      "skills / skill directories",
      "code execution sandboxes"
    ],
    "quotes": [
      {
        "text": "architecture follows model updates and your evals have to follow your architecture.",
        "at": "04:42",
        "url": "https://www.youtube.com/watch?v=nxokqOq1imY&t=282s"
      },
      {
        "text": "what do you do when your model can't be controlled, right? You take the control and you bake that control into the system that you're building around the model.",
        "at": "09:49",
        "url": "https://www.youtube.com/watch?v=nxokqOq1imY&t=589s"
      },
      {
        "text": "every trajectory for the same input if you ran it a couple of times you would see dramatically different trajectories while yielding the right answer.",
        "at": "14:37",
        "url": "https://www.youtube.com/watch?v=nxokqOq1imY&t=877s"
      },
      {
        "text": "ultimately it's the eval that are sort of your durable asset that describe how your system is supposed to work.",
        "at": "18:16",
        "url": "https://www.youtube.com/watch?v=nxokqOq1imY&t=1096s"
      }
    ],
    "words": 4204
  },
  "summary_url": "/#nxokqOq1imY",
  "transcript": {
    "html": "/transcripts/nxokqOq1imY.html",
    "txt": "/transcripts/nxokqOq1imY.txt",
    "vtt": "/transcripts/nxokqOq1imY.vtt"
  }
}