{
  "video": {
    "id": "McknwOzbmyg",
    "title": "Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia",
    "duration": 1155,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "McknwOzbmyg",
    "title": "Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia",
    "one_liner": "Ufonia's research engineer explains how they prove a clinical voice agent is safe without A/B testing on patients: simulate the patients (Matrix/Patbot), judge the transcripts with a clinician-validated LLM judge, and optimize prompts with GEPA against a hazard cost matrix — because \"you don't ship the model, you ship the evidence.\"",
    "summary": "Jared Joselowitz works on the safety and evaluation stack behind Dora, Ufonia's voice AI agent that phones patients for post-op follow-ups and pre-op checks — ~200,000 real clinical calls across 20 UK hospitals, contracted to reach a million patients in two years. He argues the standard derisking playbook (ship to 5%, watch dashboards, roll back) is unavailable in healthcare: you can't randomize patients into a worse variant, you can't un-say a call, and a dashboard going red means a patient was already harmed. Instead, borrowing from self-driving's millions of simulated miles, Ufonia built Matrix: an LLM 'Patbot' plays a scenario-conditioned patient against Dora, and 'Bev judge' grades the dialogues pass/fail against clinician-defined hazards. Improvement comes not from hand prompt-engineering but from GEPA-style prompt optimization against a cost matrix that weights missed red flags as catastrophic and false alarms as merely annoying, with simulation as the inner loop and staged, clinician-supervised exposure as the outer loop.",
    "key_points": [
      "Three constraints define the problem: you can't A/B test on patients (\"randomizing patients into a worse variant is unethical and often illegal\"), you can't roll back a call that already happened, and a vendor's \"92% on some benchmark\" model card is not a defense at a post-incident review.",
      "Because Dora asks about symptoms, gives advice and answers questions, it counts as a regulated medical device; regulation reduces to three questions — what does your software do, what could go wrong, how do you ensure it doesn't — so they start from the harm. They have 20–40 documented hazards: missing a red flag like sudden vision loss or severe pain, hallucinating an answer to a medical question, or ignoring patient distress and carrying on.",
      "Scale numbers: ~200,000 real clinical calls across 20 UK hospitals, contracted to a million patients within two years, plus a 2026 US launch live in 2 clinics with 6 more signed across 4 states. At that scale a 5% canary is hundreds or thousands of patients receiving unproven care.",
      "Matrix (paper on arXiv) recreates a clinical conversation with no real patient: an LLM 'Patbot' conditioned on a specific scenario (e.g. asking whether the agent is human or AI) talks to Dora, grounded in real clinical workflows rather than abstract situations. Simulated patients over hired actors because actors don't scale with fast iteration.",
      "Patbot was validated twice — a pure script-adherence check (does it do what it was told?) that filtered out weaker instruction-following models, then a PPI (patient and public involvement) study showing real patients four side-by-side pairs of a real doctor–patient conversation and a Dora–Patbot one; in 3 of 4 sets the majority picked the simulated patient as the real person. The lesson: there is no single realistic patient, so simulate diverse personas.",
      "'Bev judge' is an LLM judge taking the dialogue, expected behaviors and clinician-defined hazardous scenarios, returning pass/fail with a structured reason and which hazards triggered. Validated on a 240-example corpus with ground truth against 10 clinicians from 10 specialties: the judge is on par with or slightly better than expert clinicians; top model at paper time (Gemini 2.5 Pro) hit F1 0.96 with near-perfect sensitivity — deliberately preferring to over-call hazards than under-call them.",
      "Manual prompt engineering was abandoned because prompt brittleness is real: formatting changes alone have swung a benchmark by 76 percentage points, and reordering few-shot examples flips a model from near-random (~50%) to near state-of-the-art. They use GEPA (genetic Pareto, from the DSPy authors): define a metric, find failures, have a strong LLM reflect on them and rewrite the prompt, keeping a Pareto frontier until budget is exhausted — hours-to-days of hand tuning becomes 30–60 minutes, reproducible and with an audit trail.",
      "'Good' is a cost matrix, not flat accuracy: catching a red flag is good, missing one is catastrophic, over-calling one is mildly annoying (a few extra questions), so the feedback metric rewards sensitivity. Want a different objective? Recompile the prompt.",
      "The replacement for ship-watch-rollback is a flywheel: real call data plus manufactured synthetic edge cases (rare symptoms, mistranscriptions) → prompt optimization → Matrix as a simulation safety gate → gated deploy, with every new call feeding back in. Then staged exposure: user testing, supervised clinical evaluation with real patients and clinicians in the loop at every step, monitored deployment, and autonomy granted in proportion to accumulated evidence.",
      "Voice adds new failure modes — back channeling and interruptions. An agent mid-safety-advice (\"you must avoid bright lights\") gets cut off by an out-of-scope question and weaker models simply drop the safety advice. Matrix catches this: the failure modes change, the framework doesn't."
    ],
    "takeaways": [
      "Start from harm, not features: write down the specific hazards for your product (Ufonia has 20–40) and make every dataset, pinned prompt and judge verdict trace back to the exact hazard it addresses — the traceable evidence, not the model, is the deliverable you ship to a regulator.",
      "Manufacture your rare-but-dangerous cases with a scenario-conditioned simulated user instead of waiting for them to occur in production, and validate that the simulator is realistic (script adherence first, then a human side-by-side discrimination study) before trusting anything it produces.",
      "Use an LLM judge to make grading scale, but validate it against a labeled corpus and multiple domain experts before you rely on it — and optimize the judge for sensitivity if a miss is catastrophic and a false alarm is merely annoying.",
      "Replace hand prompt-engineering with an optimizer like GEPA and encode your real cost function in the metric; prompt brittleness (76-point benchmark swings from formatting alone) makes manual tuning unreproducible, and the work becomes a data/metric problem defined jointly with domain experts.",
      "Treat simulation as the inner loop that earns the right to try carefully on real users, not as proof — pair it with staged, supervised rollout where the system's autonomy grows only as fast as its evidence."
    ],
    "topics": [
      "evals",
      "llm-as-judge",
      "simulation",
      "voice-agents",
      "healthcare-ai",
      "prompt-optimization",
      "safety",
      "regulation"
    ],
    "tools": [
      "Ufonia",
      "Dora",
      "Matrix",
      "Patbot",
      "Bev judge",
      "GEPA",
      "DSPy",
      "Gemini 2.5 Pro",
      "arXiv",
      "Waymo"
    ],
    "quotes": [
      {
        "text": "randomizing patients into a worse variant is unethical and often illegal",
        "at": "01:07",
        "url": "https://www.youtube.com/watch?v=McknwOzbmyg&t=67s"
      },
      {
        "text": "roll back you can't really roll back the call has already happened the person has already been harmed by watching the dashboards the dashboards are just going red means that a patient was actually hurt",
        "at": "05:10",
        "url": "https://www.youtube.com/watch?v=McknwOzbmyg&t=310s"
      },
      {
        "text": "So simulation is necessary but it's not sufficient.",
        "at": "15:51",
        "url": "https://www.youtube.com/watch?v=McknwOzbmyg&t=951s"
      },
      {
        "text": "you don't ship the model, you ship the evidence when trying to regulate",
        "at": "16:48",
        "url": "https://www.youtube.com/watch?v=McknwOzbmyg&t=1008s"
      },
      {
        "text": "The failure modes change the framework doesn't.",
        "at": "18:16",
        "url": "https://www.youtube.com/watch?v=McknwOzbmyg&t=1096s"
      }
    ],
    "words": 4144
  },
  "summary_url": "/#McknwOzbmyg",
  "transcript": {
    "html": "/transcripts/McknwOzbmyg.html",
    "txt": "/transcripts/McknwOzbmyg.txt",
    "vtt": "/transcripts/McknwOzbmyg.vtt"
  }
}