Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health
https://www.youtube.com/watch?v=YXEqC05WEI0
AI Engineer · 21 min · 302 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] Hello and good morning.
[00:14] Chetana gave us a great overview of you know what a bridge does.
[00:18] Today I'm here to talk more from a practitioner's view of you know how we are building health care AI within Hinge Health.
[00:24] So hi, I'm Rashi Agrawal.
[00:26] I lead AI and ML at Hinge Health and today I will be talking about guardrails that are needed to build member-facing health care AI.
[00:39] I want to talk a little bit about the state of health care AI right now.
[00:42] We do have a lot of frontier models which are running and believe it or not, 40 million people actually use these models for triaging their health care issues.
[00:52] But there is a caveat and these are some of the headlines that have been happening in the past few in the past one year or so.
[01:00] Poisoned by a chatbot.
[01:02] Let's start with this one person.
[01:04] A 60-year-old healthy man asked a popular AI assistant how to cut salt from his diet.
[01:11] The LLM told him to swap it with bromine sodium bromide.
[01:15] He did it for 3 months.
[01:17] He landed in the ER with paranoia and hallucinations.
[01:21] Bromide levels 200 times the safe limit.
[01:25] 3 weeks in the hospital.
[01:27] For what?
[01:27] For following diet advice?
[01:30] Let's look at another pattern.
[01:32] The first independent safety test of a consumer health AI out of Mount Sinai found that this health AI is under triaging life-threatening emergency 50% of the times.
[01:46] Diabetic ketoacidosis, respiratory failure.
[01:50] And it told the people to go see a doctor in a day or two.
[01:54] The right answer was ER right now.
[01:58] And this is in French.
[02:00] In February, ECRI the patient safety group that hospitals trust to rank their top risks named AI chatbot misuse as the number one health technology hazard of 2026.
[02:16] Number one on the list that they publish every year.
[02:19] So this is not this is not really a frontier problem.
[02:23] This is the production baseline that we're working with right now.
[02:27] So the question comes, how do you ship AI to somebody who's already trusted you with their health?
[02:34] The next 20 minutes are all about that.
[02:39] It starts with three non-negotiable foundations.
[02:42] One, the constraint is the architecture.
[02:46] Most AI safety failures in health care are not model failures.
[02:50] They are architectural decisions that were made before even a single token was generated.
[02:57] Two, deterministic rules belong above the model, not inside it.
[03:03] What can never be wrong cannot be left to probability.
[03:08] And three, safety is a continuous evaluation layer, not a one-time gate.
[03:15] Launch of your product is where the real risk starts, not where it ends.
[03:20] That's the first part of what I want to talk about today.
[03:23] The second half is what happens when the architecture is not enough and a human has to make a decision of what ships versus what holds.
[03:33] [snorts]
[03:35] Let's start with layer one.
[03:38] Protecting PHI takes both policy and architecture.
[03:43] Policy tells you what to protect and architecture makes sure that it actually happens.
[03:49] The first thing that changes when you start shipping member facing health AI is where PHI lives.
[03:57] Most teams treat PHI as a runtime problem.
[04:00] Something to redact when a log gets written to a dashboard.
[04:05] That's the reactive version.
[04:08] The architecture version strips PHI at the pipeline boundary.
[04:13] At ingestion before it ever reaches the data lake.
[04:18] By the time the data is stored, the PHI is gone.
[04:21] So, a developer opens a dashboard there's nothing to redact.
[04:26] The PHI was never there.
[04:28] The rest of the architecture works in a similar way.
[04:32] Production and non-production stay completely separate.
[04:36] No pipes in between because even a single pipe is all that it takes for member data to leak into a dev environment.
[04:44] And HIPAA laws are very stringent, especially in health care.
[04:47] You know, the regulatory bar is much, much higher.
[04:50] So, you have to be very careful about the architecture that you're designing.
[04:54] Another big thing access depends on two things.
[04:58] Your role and your geographic region.
[05:01] We all work with the teams which are geographically distributed, but not everybody has access to PHI.
[05:07] That is a certification, a policy that is applied to specific regions only.
[05:12] An engineer outside the regulated region cannot reach raw PHI at all.
[05:18] And the compliance rules, HIPAA, FDA's good machine learning practice, state laws like Texas, Triaga, they are not afterthoughts.
[05:27] They are the grounding input in how you actually design your systems.
[05:32] You cannot slap on HIPAA on top of, you know, an underlying system or an architecture.
[05:37] You start with it and let the architecture grow around it.
[05:42] When PHI is protected at the architecture level, you're not just trusting that the policies will get followed.
[05:49] You're actually relying on a system that's incapable of certain failures.
[05:56] Let's move to the layer two.
[05:58] Probabilistic systems are great at generation.
[06:01] We all know that.
[06:03] However, they are unreliable for things that cannot can never be wrong.
[06:08] So, the rule is very simple.
[06:10] Must not fail behavior belongs above your prompt, above the model.
[06:16] And what does above the prompt actually mean?
[06:19] It means that there is a code layer that runs first on every turn before the model even runs.
[06:27] The code layer is what makes your irreversible decisions.
[06:31] The decision of, you know, whether this is an emergency escalation, should they be routed to 911, should a clinician step into the loop.
[06:38] All of those are irreversible decisions which need to lie at a deterministic code level layer.
[06:44] The model handles the long tail of your conversations and interactions with your members.
[06:51] The picture to hold in your head is a stack.
[06:54] Code on top, model below.
[06:57] Every turn goes through the code layer first.
[07:02] Most turns do reach the model, but the model never gets a vote on high stake calls.
[07:13] Here's how you can think about it in a different way.
[07:15] A model is not a guardrail.
[07:18] A model with a system prompt is also not a guardrail.
[07:23] Code that runs above the model is closer.
[07:27] Even the labs that be build these frontier models publish the authority hierarchy.
[07:33] Root, system, developer, user, guideline.
[07:38] Every layer above user is one prompt injection away from being overridden.
[07:44] If the labs themselves don't trust the prompt as a security boundary, neither should you.
[07:51] [snorts]
[07:52] So, what does live in this code layer?
[07:54] Let's examine it a little bit.
[07:56] Let's take three examples.
[07:58] First, very very relevant to health care, which is emergency escalation.
[08:03] If a member mentions self-harm, suicidal ideation, or an acute medical emergency, the system must route to 911 or 988.
[08:14] The model should not even see this turn.
[08:17] Code runs first, decides, and routes, and makes a decision right away.
[08:22] Another example, intent routing.
[08:26] Which capability in your underlying multi-agentic system, multi-agentic architecture handles a conversation turn?
[08:33] Is it clinical?
[08:35] Is it tech support?
[08:36] Is it education from the millions of, you know, credited articles?
[08:40] Is it exercise recommendation?
[08:43] The model can help to classify, but high-stakes path must must again take a deterministic route at the top itself.
[08:53] You You don't want like a clinical question quietly being routed to your generic tech support agent.
[09:00] That's unrecoverable.
[09:04] Third, identity verification.
[09:08] Anything that touches member data has to check that the right member is at the other end.
[09:15] That's an authentication check.
[09:17] And authentication is a security bound boundary.
[09:20] Prompts are not.
[09:22] The underlying pattern across all three, code runs first.
[09:27] Code makes the irreversible decisions.
[09:30] The model handles what's left.
[09:35] Last but not least, layer three.
[09:38] As we all know, safety is not a gate you pass once.
[09:43] It is a continuous layer that runs the whole time.
[09:47] Most teams treat evals as a pre-launch checklist.
[09:51] You run your tests, you ship, you move on.
[09:54] That's necessary, of course, but that's hardly enough.
[09:58] What actually holds up in production is judges that continuously keep scoring real conversations as they happen.
[10:06] Not a saved golden data set.
[10:09] Live traffic.
[10:11] Scored on a lot of dimensions all the time.
[10:16] These signals come from three sources, and each one catches something different.
[10:21] First, automated judges.
[10:23] 30, 40, name it, you know, as as much as you can scale.
[10:27] Automated judges with multiple dimensions, always refreshing.
[10:32] Clinical accuracy, safety, escalation, relevance, drift, refusal, etc., etc. I can keep going on.
[10:39] But you you get the point.
[10:41] These are the automated judges that are always going to catch regressions and any even sensitive drops in quality.
[10:49] Second, your gold mine of information.
[10:52] That's going to be member feedback.
[10:54] Thumbs up, thumbs down on each and every single message.
[10:58] That's the truth signal.
[10:59] That's your member communicating with you.
[11:02] And it's the only one that comes straight from the person that you're serving it to.
[11:07] It catches tone problems and things that judges miss.
[11:12] [snorts]
[11:13] Third, sample traces.
[11:15] Random samples spread across capabilities with high-stake cases checked every single time.
[11:22] 100% sampling on those.
[11:25] Ultimately, people need to read these signals.
[11:30] People are going to catch what no single metric is going to catch.
[11:34] And here's the part that nobody really warns you about.
[11:37] The bottleneck is not the compute, the models, the capability.
[11:41] It's actually having enough people to read the signal and act on it.
[11:49] One more thing about layer three.
[11:51] Some failures, you can't just prompt away.
[11:54] You ship the fix, it comes back under new conditions.
[11:58] New prompts, new tools, the model shifts.
[12:01] You ship the fix again.
[12:03] Each round buys you less and less.
[12:06] The rate never hits zero.
[12:09] At this point, monitoring is not a last resort.
[12:12] It is the first resort, which is always on.
[12:16] A new failure that you see in production simply means you now have a new judge.
[12:21] Your underlying architecture and your system needs to be able to keep scaling with new judges, new monitoring as you keep scaling your, you know, consumers.
[12:30] And
[12:30] [snorts]
[12:30] that's the point.
[12:32] Monitoring is how you know that the architecture is still holding.
[12:37] [snorts]
[12:39] But monitoring also tells you when the architecture is not enough.
[12:44] And when the architecture is not enough, a human has to decide.
[12:48] And this is the second part of my talk where I want to focus on the decisioning frameworks.
[12:54] [snorts]
[12:55] Let's take an example.
[12:57] You're about to ship, you know, uh consumer AI again in the healthcare space, and you have a feature, a specific capability that you're about to launch.
[13:05] And there is one issue left on the board 5 days before your launch.
[13:10] And you have multiple different stakeholders.
[13:13] Five stakeholders look at the same issue.
[13:15] Each one sees a different risk.
[13:18] And they don't agree what to do about it.
[13:22] Clinical sees member safety risk.
[13:24] They want to hold the launch.
[13:27] Legal sees regulatory exposure.
[13:29] Compliance sees audit risk.
[13:32] Product sees adoption risk.
[13:36] The If the If it ships broken, the feature won't land.
[13:40] And engineering sees velocity risk.
[13:43] They can't fix it without slipping the date they want to ship.
[13:47] Five rational people, five different risks, and five very different fixes.
[13:54] So, what do you do?
[13:55] Do you hold the launch and fix, or do you actually ship?
[14:01] The next slide is the framework I actually use for making these decisions.
[14:05] Five rules.
[14:07] This is how I think about decisions when stakeholders disagree.
[14:12] Rule one.
[14:13] Worst case always wins.
[14:16] Severity is set by the worst possible plausible outcome, not the average.
[14:21] And this is extremely relevant in health care.
[14:24] A bug that lightly annoys 100% of users is way less severe than one that could cause serious harm in 0.1% of cases.
[14:36] This is non-negotiable.
[14:38] The worst case matters more than the average case, always.
[14:43] So, when you're triaging, don't ask, "How often does this happen?"
[14:48] Ask, "What's the worst version of this?"
[14:51] That sets the severity.
[14:54] Rule two.
[14:56] Severity is not capacity.
[14:58] This one keeps politics out of it.
[15:01] As we all know, as we ship features, there's always a little bit of contention between timelines, features, deliverables.
[15:07] But, a bug's severity comes from the harm that it causes.
[15:12] Not who owns it, not whether your team has the capacity to fix it, not how hard the fix is.
[15:20] You have three options in front of you at this point.
[15:23] Fix, delay the launch, or accept the risk with explicit sign-off.
[15:29] Those are the three.
[15:31] You never quietly downgrade a bug just because you can't get to it.
[15:37] Rule three, asymmetric default.
[15:41] When you don't know what to do, always pick the safer mistake.
[15:47] And there are two spectrums to it.
[15:48] One is safety bugs and polish, the other side is polish bugs.
[15:52] For safety bugs, the math is one-sided.
[15:56] Shipping a real safety bug is much worse than delaying for a false alarm.
[16:02] So, for safety bugs, when you're not sure, always hold and fix.
[16:07] On the other side, for polish bugs, the math runs the other way.
[16:13] Delaying a launch costs more than shipping a small flaw.
[16:18] So, when you're not sure, ship in case of polish bugs.
[16:22] Ultimately, the framework doesn't decide for you.
[16:25] It just tells you which way to lean.
[16:28] Rule four, revealed risk tolerance, not stated risk tolerance.
[16:35] Your launch bar is what your org already accepts in production, not what it says it will accept.
[16:43] If a behavior has been live in your existing product for weeks, months, without escalation, without member complaints, without leadership concern, you cannot you cannot call it a launch blocker just for a new thing.
[16:57] Your stated risk tolerance might be no bugs in production, but your revealed risk tolerance is what's actually shipping today.
[17:07] Calibrate to the revealed one.
[17:10] That's the floor.
[17:14] Rule five, humans are the constraint.
[17:18] Judges scale.
[17:20] Pattern interpretation doesn't.
[17:22] Always, always design for human in the loop.
[17:26] Judges code traces automatically.
[17:29] Dashboards refresh every few hours.
[17:32] None of that is hard anymore.
[17:34] But what's hard is having enough people to read the signal and act on it.
[17:41] One more piece around this.
[17:43] Fast follows are committed debt, not an optional backlog.
[17:48] If you didn't ship it at launch, it's not a wish list item.
[17:51] It's already committed.
[17:55] The five rules tell you how to decide.
[17:58] But they all assume one thing.
[18:00] That your underlying signal is true.
[18:03] So here's the discipline that needs to come first.
[18:07] In a non-deterministic system, the judge is also non-deterministic.
[18:13] Before you trust the score, verify the scorer.
[18:17] And here's what it looks like in practice.
[18:21] Say you're watching a clinical accuracy judge in production.
[18:24] The score has been steady for at 4.9 for weeks.
[18:29] Today, it drops to 4.5. And tomorrow, it stays at 4.5. The immediate instinct is, let's start changing the prompts.
[18:38] The agent is broken.
[18:39] Let's fix the agent.
[18:41] That's reactive.
[18:42] And it's risky.
[18:44] You fix one thing and you break another.
[18:48] Worse, you're changing the agent based on a signal that might not be true.
[18:54] And the discipline needs to be different.
[18:57] First, ask whether the judge is right.
[19:02] We can solidify that with a with a concrete example.
[19:06] Uh let's take it side by side.
[19:08] In scenario A, same question, member asks about caffeine.
[19:13] The agent gives FDA standard guidance.
[19:16] 400 mg for most adults, less if pregnant or on certain medications.
[19:22] The judge flags it as a hallucination.
[19:25] Because the agent mentioned pregnancy and medications without checking.
[19:32] But that's just clinical context.
[19:35] The judge is over calling in this case.
[19:38] Fix the judge in this scenario.
[19:41] For the same question, scenario B, the agent says 1,000 mg a day is fine.
[19:47] That's well above the safety limits.
[19:50] The judge correctly flags it.
[19:52] And the agent is wrong.
[19:54] In this case, fix the agent.
[19:56] The rule is always ask, is the judge right before changing the agent's response?
[20:03] Fixing a judge prompt is not cheating.
[20:06] Judges are software, too.
[20:08] And they need to continuously evolve.
[20:10] This is what production discipline looks like when the system is not deterministic.
[20:18] Here's the whole talk in one slide.
[20:20] If you screenshot one thing, this would be it.
[20:23] Six takeaways, three from architecture, three from decisioning.
[20:27] On the architecture side, the pattern is very simple.
[20:31] Don't X what you can Y. Don't policy what you can architect.
[20:36] Don't prompt what you can code.
[20:38] Don't gate what you can monitor.
[20:42] On the decisioning side, the pattern is how humans decide when the system cannot.
[20:48] Score by the worst case and default to the safer mistake.
[20:53] Calibrate to your org.
[20:55] And always design for the human in the loop.
[20:58] Fast follows are debt, not backlog.
[21:03] Yes, building guardrails first is slower than bolting them on later.
[21:09] But that's the design, not limitation.
[21:12] We are not building a generic low-stakes chatbot.
[21:15] We are building a system that has to be worthy of someone's health.
[21:20] The architecture is how, the decisioning is when, and member trust is why.
[21:27] Thank you.
[21:28] Let's continue the conversation on LinkedIn.
[21:30] Thank you.