{
  "video": {
    "id": "bZISsg7H7DA",
    "title": "Your Agents Need a Save Button - Hamza Tahir, ZenML",
    "duration": 1027,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "bZISsg7H7DA",
    "title": "Your Agents Need a Save Button - Hamza Tahir, ZenML",
    "one_liner": "Agents have traces but no save button — Hamza Tahir argues for a durable runtime that checkpoints agent state so you can replay production runs with a swapped model or mocked tool, diff the forks, and decide from a cohort rather than an anecdote.",
    "summary": "Tahir (ZenML co-founder) argues that a trace is a read-only, after-the-fact artifact sitting in a tool far from the code, disconnected from the runtime where variables, filesystem, decisions and the code itself lived — so agents can't answer 'what if it had done something different?'. The fix is a durable runtime below the harness that checkpoints state alongside the OTel spans, giving a methodology of checkpoint → replay → diff → decide, which turns production runs into evals. He demos Kitaru on a customer-support/chargeback agent: replaying from a mid-run checkpoint with GPT-5 Nano, mocking the lookup-policy tool, diffing the original against both forks, then running the same change across a cohort and having an agent analyze the JSON report via Kitaru's MCP server — which returns the verdict 'don't ship', despite the single replay looking cheap and equivalent.",
    "key_points": [
      "Traces are the closest thing agents have to a save button, but they only capture emitted telemetry of tool-call inputs and outputs; the in-flight state, filesystem, in-code decisions and the code itself are lost, leaving a read-only stamp in another tool far from the code. The missing piece is a connection between OTel spans and the actual execution.",
      "Saving enables replay and the 'what if' question: swap in a cheaper open-source model, mock a tool and override what it returns, or intentionally degrade it to see what breaks. An emerging stack category sits on top of the harness/agent frameworks and puts a durable runtime below them.",
      "Production already holds what you need — traces plus runtime state checkpoints — so evaluation becomes replaying real production checkpoints, not synthetic tests. The methodology: checkpoint, replay, diff, decide.",
      "DoorDash (blog post, 1st of June) built a simulated environment replaying customer bots for what-if scenarios: work that took hours dropped to 5 minutes, with hundreds of simulations, 90% fewer hallucinations, and results within two points of what they saw in production — because the simulations are grounded in what already happened.",
      "Kitaru demo (new tool from ZenML, open source and free): each checkpoint shows configuration, where it ran, the code it took, and the artifacts in and out, with the Docker image or sandbox environment snapshotted between checkpoints.",
      "Replaying execution 71 with the model changed to GPT-5 Nano after a specific tool call skips the first three checkpoints — the runtime already has that state — and executes only from the fork point. A second replay instead mocks the lookup-policy tool with another function from the codebase, holding the model constant.",
      "A `diff` command against the original execution ID produces a URL with a side-by-side view of the baseline and both forks: all three restricted the chargeback, but the first two ended at 'needs review' while the policy-mocked third ended at 'safe to answer' — and the two forks were cheaper in input and output tokens.",
      "Naive model swaps are a false economy: a BrainTrust study shows you can look faster and cheaper on paper while the support bot stops resolving requests. On tau-bench, a model that passes 60% of the time is only self-consistent about a quarter of the time — so one replay is just an anecdote.",
      "Cohort replay (the `replay many` command) applies one change — same tool calls, or a cheaper model — across a set of runs sorted by cost, emitting a JSON report; Tahir feeds that report to an agent through the Kitaru MCP server to analyze thousands of replays, and the cohort verdict comes back 'don't ship' even though the single replay looked fine."
    ],
    "takeaways": [
      "Stop treating the trace as the record. Run the agent on a runtime that checkpoints state — code, artifacts, environment — so a run can be forked and re-executed from any point instead of only inspected read-only.",
      "Build your eval set from real production runs and checkpoints, not synthetic cases, and build cohorts that matter: the expensive ones, the long ones, the risky ones.",
      "Never ship a change off one or two replays — tau-bench self-consistency means a single replay is an anecdote. Run the change across a cohort and look at the distribution.",
      "Don't judge a model swap on cost alone; measure the value created (did the support request actually get resolved?) alongside tokens and latency, or you buy a false economy.",
      "At thousands of replays the comparison stops fitting in a UI — emit the results as JSON and use skills and an MCP server over a queryable runtime to have an agent do the cohort analysis, with a human in the loop on the final ship/route/hold decision."
    ],
    "topics": [
      "agents",
      "observability",
      "replay",
      "evals",
      "checkpointing",
      "tracing",
      "cost-optimization",
      "mcp"
    ],
    "tools": [
      "Kitaru",
      "ZenML",
      "GPT-5 Nano",
      "OpenTelemetry (OTel)",
      "MCP server",
      "Docker",
      "DoorDash",
      "BrainTrust",
      "tau-bench"
    ],
    "quotes": [
      {
        "text": "all of that is lost and it is only stamped as a read-only trace by the end, which is sitting in another tool far away from where the actual code is.",
        "at": "01:05",
        "url": "https://www.youtube.com/watch?v=bZISsg7H7DA&t=65s"
      },
      {
        "text": "So, checkpoint replay diff decide. And this is really the methodology that that I've seen and I've seen others do, uh which has really scaled.",
        "at": "04:08",
        "url": "https://www.youtube.com/watch?v=bZISsg7H7DA&t=248s"
      },
      {
        "text": "a model that passes 60% of the time is only self-consistent about a quarter of the time. So which basically means that one replay is just an anecdote",
        "at": "14:31",
        "url": "https://www.youtube.com/watch?v=bZISsg7H7DA&t=871s"
      },
      {
        "text": "Okay, so we're done. So, the verdict is don't ship. So, even though it looked like from a single replay that it was cheaper to do and we reached the same result, across a bunch of those support cases, you actually saw that our agent concludes that you shouldn't be using a cheaper model in this particular case for your data.",
        "at": "15:55",
        "url": "https://www.youtube.com/watch?v=bZISsg7H7DA&t=955s"
      }
    ],
    "words": 3397
  },
  "summary_url": "/#bZISsg7H7DA",
  "transcript": {
    "html": "/transcripts/bZISsg7H7DA.html",
    "txt": "/transcripts/bZISsg7H7DA.txt",
    "vtt": "/transcripts/bZISsg7H7DA.vtt"
  }
}