{
  "video": {
    "id": "YXEqC05WEI0",
    "title": "Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health",
    "duration": 1309,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "YXEqC05WEI0",
    "title": "Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health",
    "one_liner": "Hinge Health's AI/ML lead argues that member-facing health AI is made safe by architecture and deterministic code above the model — not by prompts — plus continuous judges on live traffic and an explicit five-rule framework for deciding what ships when stakeholders disagree.",
    "summary": "Rashi Agrawal opens with production failures — the man hospitalised for three weeks after an AI assistant told him to replace salt with sodium bromide, Mount Sinai finding a consumer health AI under-triaging life-threatening emergencies 50% of the time, and ECRI naming AI chatbot misuse the #1 health technology hazard of 2026 — to argue this is the baseline, not a frontier problem. The first half lays out three layers: strip PHI at the pipeline boundary rather than redacting at runtime, put must-not-fail behaviour in a deterministic code layer that runs above the model on every turn, and treat safety as continuous evaluation on live traffic rather than a pre-launch gate. The second half is a decisioning framework for when architecture isn't enough and five stakeholders see five different risks in the same bug five days before launch. She closes on judge discipline: in a non-deterministic system the judge is non-deterministic too, so verify the scorer before changing the agent.",
    "key_points": [
      "40 million people use frontier models to triage their own health issues; a 60-year-old man following an AI's advice to swap salt for sodium bromide hit bromide levels 200× the safe limit and spent 3 weeks in hospital with paranoia and hallucinations.",
      "Mount Sinai's first independent safety test of a consumer health AI found it under-triaged life-threatening emergencies (diabetic ketoacidosis, respiratory failure) 50% of the time, saying 'see a doctor in a day or two' when the answer was the ER now; ECRI named AI chatbot misuse the #1 health technology hazard of 2026.",
      "PHI is an architecture problem, not a runtime one: strip it at ingestion before it reaches the data lake so there is nothing to redact in a dashboard; keep production and non-production completely separate with no pipes; gate access on both role and geographic region so engineers outside the regulated region cannot reach raw PHI; treat HIPAA, FDA good machine learning practice and state laws like Texas's as grounding inputs, not afterthoughts.",
      "A model is not a guardrail, and a model with a system prompt is not a guardrail either — the labs themselves publish an authority hierarchy (root, system, developer, user, guideline) where every layer above user is one prompt injection away from being overridden.",
      "Three things that belong in the deterministic code layer above the model: emergency escalation (self-harm, suicidal ideation, acute emergency routed to 911/988 without the model ever seeing the turn), intent routing (a clinical question quietly routed to the generic tech-support agent is unrecoverable), and identity verification (authentication is a security boundary; prompts are not).",
      "Continuous evaluation runs on three signal sources: 30–40+ automated judges scoring live traffic (not a golden dataset) on clinical accuracy, safety, escalation, relevance, drift and refusal; thumbs up/down member feedback on every single message as the truth signal that catches tone; and random sample traces with 100% sampling on high-stakes cases. Judges scale; pattern interpretation doesn't — the bottleneck is people to read the signal.",
      "Five decision rules: worst case always wins (a bug lightly annoying 100% of users is less severe than one causing serious harm in 0.1%); severity is not capacity (only fix, delay, or accept with explicit sign-off — never quietly downgrade); asymmetric default (hold for safety bugs, ship for polish bugs when unsure); calibrate to revealed risk tolerance — what's already live without escalation is the floor, so you can't call it a launch blocker for a new feature; humans are the constraint.",
      "Verify the judge before changing the agent: when a clinical accuracy judge steady at 4.9 for weeks drops to 4.5 and stays there, ask if the judge is right first. Caffeine example — agent giving FDA guidance (400 mg for most adults, less if pregnant or on certain medications) flagged as hallucination means fix the judge; agent saying 1,000 mg/day is fine means fix the agent. 'Fixing a judge prompt is not cheating. Judges are software, too.'"
    ],
    "takeaways": [
      "Move must-not-fail behaviour out of the prompt into a code layer that executes first on every turn — emergency escalation, high-stakes intent routing and identity verification should be decided before the model is invoked, with the model handling the long tail.",
      "Strip PHI at the pipeline boundary at ingestion rather than redacting downstream, separate prod and non-prod with no pipes at all, and scope access by role and region so the system is incapable of the failure rather than merely policy-bound against it.",
      "Stand up continuously refreshing automated judges on live production traffic across many dimensions, collect per-message thumbs up/down, and sample 100% of high-stakes traces — then staff people to actually read the signal, because that's the real bottleneck, not compute.",
      "When a judge score moves, investigate the judge before touching the agent; treat judges as evolving software and add a new judge every time a new production failure appears, since prompt fixes buy less each round and the failure rate never reaches zero.",
      "Triage by asking 'what's the worst version of this?' instead of 'how often does this happen?', default to the safer mistake for safety bugs and to shipping for polish bugs, calibrate the launch bar to what's already live in production without escalation, and treat fast follows as committed debt rather than backlog."
    ],
    "topics": [
      "ai-safety",
      "healthcare-ai",
      "guardrails",
      "evals",
      "llm-as-judge",
      "compliance",
      "prompt-injection",
      "production-monitoring"
    ],
    "tools": [
      "Hinge Health",
      "Mount Sinai",
      "ECRI",
      "HIPAA",
      "FDA Good Machine Learning Practice",
      "Texas state AI law (TRAIGA)",
      "911 / 988",
      "LinkedIn"
    ],
    "quotes": [
      {
        "text": "What can never be wrong cannot be left to probability.",
        "at": "03:03",
        "url": "https://www.youtube.com/watch?v=YXEqC05WEI0&t=183s"
      },
      {
        "text": "If the labs themselves don't trust the prompt as a security boundary, neither should you.",
        "at": "07:44",
        "url": "https://www.youtube.com/watch?v=YXEqC05WEI0&t=464s"
      },
      {
        "text": "The bottleneck is not the compute, the models, the capability. It's actually having enough people to read the signal and act on it.",
        "at": "11:36",
        "url": "https://www.youtube.com/watch?v=YXEqC05WEI0&t=696s"
      },
      {
        "text": "In a non-deterministic system, the judge is also non-deterministic. Before you trust the score, verify the scorer.",
        "at": "18:07",
        "url": "https://www.youtube.com/watch?v=YXEqC05WEI0&t=1087s"
      }
    ],
    "words": 3385
  },
  "summary_url": "/#YXEqC05WEI0",
  "transcript": {
    "html": "/transcripts/YXEqC05WEI0.html",
    "txt": "/transcripts/YXEqC05WEI0.txt",
    "vtt": "/transcripts/YXEqC05WEI0.vtt"
  }
}