{
  "video": {
    "id": "wEc9aG7cRQc",
    "title": "Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog",
    "duration": 1538,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "wEc9aG7cRQc",
    "title": "Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog",
    "one_liner": "Agent flip-flopping isn't LLM randomness — the cases that flip cluster on the decision boundary where even human experts disagree, so surface them via run-to-run disagreement and fix them with semantic and episodic memory rather than fine-tuning.",
    "summary": "Diane Lin (Datadog, formerly co-founder of a security-alert auto-triage startup acquired by Datadog) argues that when the same model on the same input returns semantically different verdicts, the fault is usually ambiguous labels and missing information, not the model. The data points that flip concentrate in the 'gray zone' near the decision boundary, where the right answer depends on a particular company's policy or preference and human experts disagree too. She adapts classic active learning to LLM agents: use disagreement across runs/models (query-by-committee) rather than LLM confidence scores to select cases for human attention, then close the loop with semantic memory (distilled policy rules) and episodic memory (similar past cases and their verdicts) instead of expensive fine-tuning. On 93 real security alerts run three times, 25% flip-flopped; episodic memory made about 15% of them consistent, leaving 10% for human review.",
    "key_points": [
      "The problem isn't different wording — it's semantically different verdicts from the same model on the same input, which means a single evaluation run doesn't tell you the whole story and you must average over repeated runs.",
      "Inconsistency is a trust and sales problem, not a technical inconvenience: in a POC bake-off, the vendor with consistent verdicts beats the one that flip-flops, and customers ask 'which one should I trust?'",
      "Flipping points concentrate around the decision boundary — the 'gray zone' — where traditional ML classifiers and human experts also struggle; the agent is simply surfacing ambiguity that already exists.",
      "In both examples the 'correct' label is a policy choice: whether a hotel wants complaints outside its control marked negative, and whether an enterprise wants alerts about an attacker who is still locked outside (failed logins / password spray) versus one who got in past the password and MFA.",
      "Active learning already solves 'find the gray zone', but the LLM version needs two refinements: the selection strategy and the retraining step.",
      "LLM uncertainty scores are unreliable as a selection signal — 'the LLM doesn't know what it doesn't know', so high confidence doesn't mean correct; disagreement across multiple runs or multiple models is the more reliable signal.",
      "Replace retraining with memory: semantic memory holds distilled factual/policy knowledge that sharpens the decision boundary (and helps human labelers be consistent too); episodic memory references similar past cases and their verdicts, needing no new human intervention because the labeling already happened.",
      "The two memories are complementary, not competing: episodic memory auto-resolves recurring cases (especially recurring false-positive noise in security alerts — the low-hanging fruit), and only what it can't resolve goes to human review, whose reasoning is distilled into semantic memory for next time.",
      "Experiment: 93 cybersecurity alerts run three times. Without the approach, a quarter flip-flopped. With episodic memory, about 15% became consistent; 10% still flipped — either no similar past case existed, or the agent disagreed even with a reference available."
    ],
    "takeaways": [
      "Stop blaming the model for inconsistency — spend your energy on label quality and on the information the agent is missing to disambiguate boundary cases.",
      "Select cases for human review by disagreement across repeated runs or different models, not by the model's own confidence/uncertainty score.",
      "Treat every disagreement as a learning opportunity: it points exactly at the small, high-return subset worth labeling, so you get efficient quality control without checking every agent output.",
      "Before reaching for fine-tuning, augment the agent with semantic memory (explicit policy rules like 'password spray without successful login is benign; with successful login it's malicious') and episodic memory (nearest past cases and their verdicts).",
      "Run episodic memory first to auto-close recurring cases, route the remainder to humans, and feed the humans' distilled reasoning back into semantic memory — which also adapts the product to each customer's environment."
    ],
    "topics": [
      "agents",
      "evals",
      "consistency",
      "active-learning",
      "agent-memory",
      "labeling",
      "security-operations",
      "human-in-the-loop"
    ],
    "tools": [
      "Datadog",
      "Zscaler",
      "Vicarious",
      "Google DeepMind",
      "Amazon Alexa",
      "Gmail",
      "Imperial College London",
      "MIT"
    ],
    "quotes": [
      {
        "text": "So in other words it's not your AI agent's fault. Your AI agent simply point out the ambiguity that already exists.",
        "at": "10:27",
        "url": "https://www.youtube.com/watch?v=wEc9aG7cRQc&t=627s"
      },
      {
        "text": "It's kind of a LM model doesn't know what it doesn't know. So when it's very confident it doesn't mean that it would is correct.",
        "at": "14:54",
        "url": "https://www.youtube.com/watch?v=wEc9aG7cRQc&t=894s"
      },
      {
        "text": "the model disagreement isn't a bug, but feature. Treat each disagreement as an opportunity for your model to learn.",
        "at": "24:16",
        "url": "https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1456s"
      },
      {
        "text": "Yes, one option is retrain the model, but fine-tuning model is expensive. Here, we are proposing a more lighter weight solution, easier to iterate.",
        "at": "15:43",
        "url": "https://www.youtube.com/watch?v=wEc9aG7cRQc&t=943s"
      }
    ],
    "words": 3459
  },
  "summary_url": "/#wEc9aG7cRQc",
  "transcript": {
    "html": "/transcripts/wEc9aG7cRQc.html",
    "txt": "/transcripts/wEc9aG7cRQc.txt",
    "vtt": "/transcripts/wEc9aG7cRQc.vtt"
  }
}