{
  "video": {
    "id": "Lc8zRh9muoY",
    "title": "Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft",
    "duration": 850,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "Lc8zRh9muoY",
    "title": "Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft",
    "one_liner": "Stop trying to make your LLM bitwise-deterministic — record every agent node's inputs and outputs at the boundary so a production failure can be replayed, debugged, and turned into a free, model-free test case.",
    "summary": "Tisha and her co-presenter argue that the thing you lose the moment an agent misbehaves in production is reproducibility, and that the standard reflex — pull the prompt from telemetry, set temperature to zero, rerun locally — is a dead end because temperature zero doesn't fix a broken reasoning path and isn't even bitwise deterministic on real hardware (floating-point non-associativity, batch invariance, MoE expert capacity/rerouting). They reframe the goal from bitwise determinism (controllability, which you can't get from a hosted API and don't want) to replayability (observability): record at each node boundary what goes in and what comes out, plus the envelope — model version, build ID, sampling settings. They demo Chronicle, a proof-of-concept whose `boundary` annotation wraps any node (tool, LLM, RAG) to freeze a run as a trace, then replays that trace with selected nodes stubbed so you can run a fixed tool live and assert on it. The payoff: a recorded production failure becomes a deterministic, rerunnable, zero-model-call regression test.",
    "key_points": [
      "Motivating failure: an agent hooked to a broker API is told to 'sell $1,000 of stock', skips the math and dumps the raw 1000 into the quantity field — selling 1,000 shares at ~$190 each, a $190,000 disaster, while the API returned a clean 200 OK in 30ms with zero exceptions, zero alerts and perfectly green dashboards.",
      "Temperature zero is a misconception twice over: it doesn't fix a broken reasoning path (the model just makes the same logical error the same way), and per engineering threads on Reddit and Hacker News, running the same prompt a thousand times at temp 0 can still return dozens of different responses.",
      "Four first principles for non-determinism: (1) sampling determinism isn't system determinism — argmax doesn't guarantee identical underlying scores; (2) floating-point math isn't associative, so a tiny shift in matrix operations alters the logits and flips the winning token; (3) it isn't concurrency — the same matmul alone on a GPU gives the same bits a thousand times — the culprit is batch invariance, your request grouped with whatever else hits the server that millisecond; (4) MoE routing has the same bottleneck: experts have strict capacity limits, and an overflowing batch reroutes tokens depending on the traffic you were batched with.",
      "Bitwise determinism (same input, same output) is controllability you can't get from a hosted API and shouldn't want — the randomness is what makes the model good; replayability is observability: re-validate a run that already happened well enough to debug it.",
      "Don't record at the network layer — half the agent never touches the network (local retrieval, in-process tools, memory) and network capture shreds under streaming and async. Record at the boundary: what enters and leaves each node, the meaning of each step, not the packets.",
      "Chronicle (their PoC) centres on a `boundary` annotation — a bounding box around any node, whether a tool call, an LLM call or a RAG retrieval — that records every input/output pair plus parameters like model version and code version, freezing the whole run state as a trace with hyper-detailed per-node JSON.",
      "The demo traces a three-node stock agent (planning agent → place_order tool → finalize agent) and shows the exact node where the LLM emitted place_order(symbol=ACME, quantity=1000); because you can't control the LLM, the fix is guardrails on the tool.",
      "Replay mode turns the trace into a test: load the recorded trace, stub the LLM node with its recorded output, run the fixed tool live, and assert on the tool's captured output — the demo shows agent one stubbed, the order blocked, and the assertion passing.",
      "Two distinct kinds of agent testing: deterministic testing of deterministic nodes (guardrails, tool calls) where stubbing 'kicks the probability out of the window', making the run rerunnable and free because it never calls the model; and behavioral testing of tone and trajectory, which is subjective and better served by LLM-as-a-judge."
    ],
    "takeaways": [
      "Stop chasing bitwise determinism through the API — the fundamental principles the APIs are built on today make it impossible; ask instead how to debug and retest a run you can't reproduce.",
      "Identify and log every variable of a session — LLM version, build ID, RAG chunks — and capture the full envelope, not just the prompt.",
      "Instrument at node boundaries (tools, LLM calls, retrievals) rather than at the network layer, so in-process tools, local retrieval and memory are covered and streaming/async don't shred the record.",
      "Promote each recorded failure trace into a regression test: stub the non-deterministic nodes from the trace, run the node you fixed live, and assert on its output — deterministic CI with zero model calls.",
      "Keep generation-time variation alive — don't pin temperature to zero, because that variation is what brings the agency into your agent — and put guardrails on the tools instead, since that's the layer you actually control."
    ],
    "topics": [
      "agents",
      "observability",
      "determinism",
      "replay",
      "testing",
      "tracing",
      "production-debugging",
      "guardrails"
    ],
    "tools": [
      "Chronicle",
      "Microsoft",
      "Reddit",
      "Hacker News",
      "LLM-as-a-judge",
      "mixture-of-experts (MoE)"
    ],
    "quotes": [
      {
        "text": "You can't reproduce it. And if you can't reproduce it, you can't debug it. And if you can't debug it, you can't promise it won't happen to your next customer or user, right?",
        "at": "00:47",
        "url": "https://www.youtube.com/watch?v=Lc8zRh9muoY&t=47s"
      },
      {
        "text": "Setting the temperature to zero doesn't fix a broken reasoning path. It just means the model is going to make the exact same logical error, the exact same way, at the exact same time",
        "at": "02:49",
        "url": "https://www.youtube.com/watch?v=Lc8zRh9muoY&t=169s"
      },
      {
        "text": "because determinism was never the North Star. Debugging was.",
        "at": "05:30",
        "url": "https://www.youtube.com/watch?v=Lc8zRh9muoY&t=330s"
      },
      {
        "text": "You don't need the model deterministic. You need the run recorded, and you don't freeze the model. You capture what it did.",
        "at": "06:05",
        "url": "https://www.youtube.com/watch?v=Lc8zRh9muoY&t=365s"
      }
    ],
    "words": 2765
  },
  "summary_url": "/#Lc8zRh9muoY",
  "transcript": {
    "html": "/transcripts/Lc8zRh9muoY.html",
    "txt": "/transcripts/Lc8zRh9muoY.txt",
    "vtt": "/transcripts/Lc8zRh9muoY.vtt"
  }
}