{
  "video": {
    "id": "5UgnatzpZ9E",
    "title": "Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest",
    "duration": 1081,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "5UgnatzpZ9E",
    "title": "Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest",
    "one_liner": "NatWest's lesson from taking its Kora assistant from scripted answers to AI for 19 million customers is that building the agent is the cheap part; the reusable, hard part is the evaluation spine, including checking that your LLM judge can actually discriminate, since one with a 0.97 mean had almost zero correlation with human scores.",
    "summary": "Jonathan Osborne, a lead AI engineer at NatWest Group, describes moving about a third of Kora's journeys (roughly 100,000 conversations a day overall) to RAG-backed answers and action-taking agents, and what it took to get approval to go live. He argues the risk is blast radius rather than model quality, so the team first shrank the attack surface with a routing funnel, then treated quality as a vector of safety, task and experience scores rather than one number. Human validation and production data from an internal complaints agent exposed misleading headline scores and an over-generous LLM judge. The result is continuous evaluation in Comet (Opik and MPM), hard CI/CD regression gates, and a shadow-then-brand-by-brand rollout with feature-flag kill switches, though he notes Kora's new agents only go live next month.",
    "key_points": [
      "Agents add four failure modes a scripted bot doesn't have: confidently wrong (hallucination), talked into things (prompt injection), right but unhelpful (accurate answers that go round in circles), and nondeterministic (same inputs, different outputs), which undermines traditional assertions and years of existing regression tests.",
      "The routing funnel narrows what reaches a model: a contextual filter removes common events like button clicks (about 20%), deterministic pre-classification rules route with no ML, then a gated relevancy and rephrasing step, a non-generative intent classifier (a neural network over embeddings), and policy rules. It exists for access control, auditability, resilience and governance, with decisions carrying a code and rules held in versioned config changeable without a full redeployment.",
      "Accuracy alone is insufficient: a router that is 94% accurate but 100% confident in the 6% of cases it gets wrong is more dangerous than one that is 85% confident and knows it. Answers are therefore scored on safety, task and experience, each monitored and alarmed independently.",
      "Human validation of the payment agent covered 111 conversations over 37 payment journeys, four brands and three independent scorers per conversation. The 83% headline pass rate hid a 54% spread: 97% on happy paths, 76% on edge cases and 43% on topic changes. The two biggest causes of failure turned out to be engineering issues, not the model.",
      "Human reviewers themselves disagree: 95% agreement on bias and fairness but only 64% on intent understanding, so simply matching human validators can't be the target for an LLM judge on subjective metrics.",
      "On about 14,000 complaints-agent cases scored by both humans and an LLM judge, faithfulness averaged 0.97 for the judge with very little spread versus 0.83 for humans with a much wider spread, and correlation was almost zero. Changing only the judge model also sharply cut the margin above the threshold, making the judge model part of your configuration. Moving the judge to a different model family from the agent gave it different blind spots and pulled the scores closer together.",
      "The evaluation pipeline captures traces via OpenTelemetry and OpenInference, samples 5% at random plus interesting conversations (failed handoffs, particularly long ones, angry users), scores offline every four hours, and stores everything in Comet. It runs in an air-gapped environment with the judge reached through an internal GenAI gateway, Comet consumed as a signed partner app via AWS, and zero tolerance for PII leaving the network. CI/CD has a hard regression gate against a known good baseline.",
      "Generative and classical components fail differently: generative AI regresses when prompts change (traced in Opik, with each LangGraph node as a span), while the intent classifier drifts as conversation evolves (monitored in MPM via input feature drift, F1 against human labels and RAG thresholds for retraining). Input feature drift is a canary that moves before F1 declines.",
      "Shadow deployment classifies live utterances while the scripted solution still responds, breaking the regulated-industry deadlock where you can't get production data without approval but model risk colleagues increasingly want to see behaviour on production data. Rollout then goes brand by brand with percentage-based stages and go/no-go KPIs, some automated and some manual, plus global and per-backend feature flags that revert in seconds.",
      "In the complaints agent, human feedback is a gate: a handler can't rerun an investigation until they give feedback. That built the roughly 14,000 colleague labels, where many projects struggle to get a tenth of that number."
    ],
    "takeaways": [
      "Shrink the attack surface before hardening it: use filters, deterministic rules and policy checks, and ask what actually needs a model to solve.",
      "Treat quality as a vector (safety, task, experience) with independent alarms, and break headline scores down by scenario type such as happy path, edge case and topic change.",
      "Calibrate LLM judges against human reviewers by comparing spreads and correlation, not just averages. Assume judges skew positive, treat the judge model as breaking-change configuration, and consider a different model family from the agent.",
      "Run evaluation as production infrastructure: continuous sampling of traces, offline scoring, and a hard CI/CD regression gate on prompt and context changes against a known good baseline.",
      "Deploy in shadow mode first to get production data at almost zero risk, then roll out in stages with go/no-go KPIs and feature flags that can revert to the scripted solution in seconds. Where you can, make human feedback a required step in the workflow to build a labelled data set."
    ],
    "topics": [
      "agents",
      "evals",
      "llm-as-judge",
      "regulated-banking",
      "guardrails",
      "observability",
      "shadow-deployment",
      "drift-monitoring"
    ],
    "tools": [
      "NatWest Group",
      "Kora",
      "Comet",
      "Opik",
      "Comet MPM (Model Performance Monitor)",
      "OpenTelemetry",
      "OpenInference",
      "LangGraph",
      "AWS"
    ],
    "quotes": [
      {
        "text": "if we build an agent that can answer any question, it can also answer any question wrong.",
        "at": "03:26",
        "url": "https://www.youtube.com/watch?v=5UgnatzpZ9E&t=206s"
      },
      {
        "text": "We didn't just take a language model, slap some context into it, and point it at 19 million customers.",
        "at": "06:24",
        "url": "https://www.youtube.com/watch?v=5UgnatzpZ9E&t=384s"
      },
      {
        "text": "a judge which doesn't discriminate between the actually good cases and the bad cases is basically worthless.",
        "at": "10:15",
        "url": "https://www.youtube.com/watch?v=5UgnatzpZ9E&t=615s"
      },
      {
        "text": "continuous evaluation is part of your production infrastructure. It's not a notebook that you run a couple of times when your model goes live.",
        "at": "12:36",
        "url": "https://www.youtube.com/watch?v=5UgnatzpZ9E&t=756s"
      }
    ],
    "words": 4077
  },
  "summary_url": "/#5UgnatzpZ9E",
  "transcript": {
    "html": "/transcripts/5UgnatzpZ9E.html",
    "txt": "/transcripts/5UgnatzpZ9E.txt",
    "vtt": "/transcripts/5UgnatzpZ9E.vtt"
  }
}