{
  "video": {
    "id": "PwCSYy-oGsM",
    "title": "Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI",
    "duration": 1320,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "PwCSYy-oGsM",
    "title": "Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI",
    "one_liner": "Containment rate is a vanity metric: an agent that escalates more can be the better agent when the escalations are deliberate handoffs. Druid AI scores the whole agent against 26 weighted components, with governance weighted highest, and re-runs the score every week against a human baseline.",
    "summary": "Druid AI's Michael and Dan use the analogy of two summer interns at a bank to argue that containment is misleading. Intern/Agent A contains 80% and escalates 20%. Agent B contains 96%, but A is the right hire because many of its escalations are designed handoffs, such as routing a customer who asks about early mortgage payoff to a senior banker as a refinance opportunity. They present a reference model that groups pillars into 'configured' (what exists before the agent runs) and 'runtime' (what it does), with governance spanning both. It has 26 components, each scored 0/1/2 and weighted toward business outcomes. Under this model A scores 83% and beats B. They back it with their 2026 AI adoption benchmark, built from 15 months of production agent data, which shows the 'right' containment rate differs by industry.",
    "key_points": [
      "Containment (the percentage of conversations handled without a human) is easy to measure and appears on every dashboard, but it can't tell a failure escalation from a designed handoff. In the example, the bank's rule is to answer the customer and then route the context to a senior banker when a question signals refinance revenue.",
      "The framework has 26 components grouped into pillars. Dan says 'six pillars' in two groups, configured and runtime, with governance across them; Michael later calls it 'seven pillars'. Each component scores 0 if missing, 1 if it can be observed and measured (even by hand), and 2 if it is 'managed', meaning tracked and analysed over time.",
      "The weighting favours business effectiveness over technical aspects: governance 39%, mandate 26%, platform 12%, outputs 9%, and so on.",
      "Governance covers error tolerance, risk, compliance (e.g. SOC 2 and bank policies), observability (logging input, reasoning, decisions and integration calls), drift against the go-live baseline, accuracy/hallucination (3% in the example, part automated and part manual annotation) and tone/sentiment. Error tolerance should be stricter for autonomous agents ('should be perfect or cancel the job'), while around 60% can be acceptable in human-AI collaboration where a human makes the final decision.",
      "The agent mandate is more than the system prompt. It covers: scope definition, business impact/ROI (e.g. a target of 70% contract renewal), policy boundaries and isolated context (which documents and systems it reasons over), security (which endpoints it may use and who may engage with it), and budget (avoiding a surprise like '20K' spent in a week).",
      "Platform capabilities check that agents share memory, orchestration and skills rather than reinventing them, and that handoff goes through a central action center rather than each agent handling its own. Skills are judged on granularity, discoverability and whether they come from a central repository. The best performer is the agent with the most effective tool use, not the most tools.",
      "At runtime, tasks must be written down, inputs logged, required context present, and success criteria created together with each task. Without that link you can't judge whether, say, a contract renewal succeeded. Outputs are validated with a single success score.",
      "Agent A scored 83% overall and beat B in every cluster except skills. B had more skills but lacked explainability, success tracking and governance compliance. Over time, a 'vanity dashboard' can sit flat at 92% containment all year while the weekly whole score slides from 83 to 60.",
      "Druid's 2026 AI adoption benchmark (15 months of anonymised production agent data, free to download) gives containment by industry: financial services 80%, healthcare 87%, HR and IT 93%, higher education 99%. The speakers say the 80/20 intern split came from this data."
    ],
    "takeaways": [
      "Stop judging agents on containment alone. Separate designed handoffs (e.g. revenue-signal routing to a human) from failure escalations, and don't compare containment across industries against one universal number.",
      "Define the agent mandate up front: scope, a measurable ROI target, the allowed context and systems (a few named endpoints, not an MCP server with thousands of tools), who may use the agent, and a budget cap.",
      "Score each component as missing / measurable / managed, and weight governance and business outcomes above technical metrics, so the decision rests on facts rather than whether the agent 'feels ready'.",
      "Have agents attach success criteria to every task and log inputs, reasoning and decisions, so outcomes can be judged afterwards. Route handoffs through a central action center.",
      "Treat evaluation as continuous: re-run the scorecard every week on live traffic against a human baseline to catch drift, and report what the agent took off people's plates (e.g. bankers freed for refinance conversations) rather than which jobs it replaces."
    ],
    "topics": [
      "agents",
      "evals",
      "metrics",
      "governance",
      "conversational-ai",
      "human-in-the-loop",
      "roi"
    ],
    "tools": [
      "Druid AI",
      "Druid 2026 AI Adoption Benchmark",
      "MCP",
      "SOC 2",
      "OpenAI",
      "Microsoft"
    ],
    "quotes": [
      {
        "text": "But surprisingly, offer to intern B is the wrong offer.",
        "at": "02:17",
        "url": "https://www.youtube.com/watch?v=PwCSYy-oGsM&t=137s"
      },
      {
        "text": "You will see at the final judgment that the performer agent is not the one with the largest number of tools.",
        "at": "14:53",
        "url": "https://www.youtube.com/watch?v=PwCSYy-oGsM&t=893s"
      },
      {
        "text": "when you hire a person, the interview ends, right? But when you hire an agent, it never ends.",
        "at": "18:02",
        "url": "https://www.youtube.com/watch?v=PwCSYy-oGsM&t=1082s"
      },
      {
        "text": "higher ed isn't better than banking. banking just design more handoff in their agenting AI design",
        "at": "20:28",
        "url": "https://www.youtube.com/watch?v=PwCSYy-oGsM&t=1228s"
      }
    ],
    "words": 3046
  },
  "summary_url": "/#PwCSYy-oGsM",
  "transcript": {
    "html": "/transcripts/PwCSYy-oGsM.html",
    "txt": "/transcripts/PwCSYy-oGsM.txt",
    "vtt": "/transcripts/PwCSYy-oGsM.vtt"
  }
}