Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health

AI Engineer · 21 min · 302 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello and good morning.
00:14Chetana gave us a great overview of you know what a bridge does.
00:18Today I'm here to talk more from a practitioner's view of you know how we are building health care AI within Hinge Health.
00:24So hi, I'm Rashi Agrawal.
00:26I lead AI and ML at Hinge Health and today I will be talking about guardrails that are needed to build member-facing health care AI.
00:39I want to talk a little bit about the state of health care AI right now.
00:42We do have a lot of frontier models which are running and believe it or not, 40 million people actually use these models for triaging their health care issues.
00:52But there is a caveat and these are some of the headlines that have been happening in the past few in the past one year or so.
01:00Poisoned by a chatbot.
01:02Let's start with this one person.
01:04A 60-year-old healthy man asked a popular AI assistant how to cut salt from his diet.
01:11The LLM told him to swap it with bromine sodium bromide.
01:15He did it for 3 months.
01:17He landed in the ER with paranoia and hallucinations.
01:21Bromide levels 200 times the safe limit.
01:253 weeks in the hospital.
01:27For what?
01:27For following diet advice?
01:30Let's look at another pattern.
01:32The first independent safety test of a consumer health AI out of Mount Sinai found that this health AI is under triaging life-threatening emergency 50% of the times.
01:46Diabetic ketoacidosis, respiratory failure.
01:50And it told the people to go see a doctor in a day or two.
01:54The right answer was ER right now.
01:58And this is in French.
02:00In February, ECRI the patient safety group that hospitals trust to rank their top risks named AI chatbot misuse as the number one health technology hazard of 2026.
02:16Number one on the list that they publish every year.
02:19So this is not this is not really a frontier problem.
02:23This is the production baseline that we're working with right now.
02:27So the question comes, how do you ship AI to somebody who's already trusted you with their health?
02:34The next 20 minutes are all about that.
02:39It starts with three non-negotiable foundations.
02:42One, the constraint is the architecture.
02:46Most AI safety failures in health care are not model failures.
02:50They are architectural decisions that were made before even a single token was generated.
02:57Two, deterministic rules belong above the model, not inside it.
03:03What can never be wrong cannot be left to probability.
03:08And three, safety is a continuous evaluation layer, not a one-time gate.
03:15Launch of your product is where the real risk starts, not where it ends.
03:20That's the first part of what I want to talk about today.
03:23The second half is what happens when the architecture is not enough and a human has to make a decision of what ships versus what holds.
03:33[snorts]
03:35Let's start with layer one.
03:38Protecting PHI takes both policy and architecture.
03:43Policy tells you what to protect and architecture makes sure that it actually happens.
03:49The first thing that changes when you start shipping member facing health AI is where PHI lives.
03:57Most teams treat PHI as a runtime problem.
04:00Something to redact when a log gets written to a dashboard.
04:05That's the reactive version.
04:08The architecture version strips PHI at the pipeline boundary.
04:13At ingestion before it ever reaches the data lake.
04:18By the time the data is stored, the PHI is gone.
04:21So, a developer opens a dashboard there's nothing to redact.
04:26The PHI was never there.
04:28The rest of the architecture works in a similar way.
04:32Production and non-production stay completely separate.
04:36No pipes in between because even a single pipe is all that it takes for member data to leak into a dev environment.
04:44And HIPAA laws are very stringent, especially in health care.
04:47You know, the regulatory bar is much, much higher.
04:50So, you have to be very careful about the architecture that you're designing.
04:54Another big thing access depends on two things.
04:58Your role and your geographic region.
05:01We all work with the teams which are geographically distributed, but not everybody has access to PHI.
05:07That is a certification, a policy that is applied to specific regions only.
05:12An engineer outside the regulated region cannot reach raw PHI at all.
05:18And the compliance rules, HIPAA, FDA's good machine learning practice, state laws like Texas, Triaga, they are not afterthoughts.
05:27They are the grounding input in how you actually design your systems.
05:32You cannot slap on HIPAA on top of, you know, an underlying system or an architecture.
05:37You start with it and let the architecture grow around it.
05:42When PHI is protected at the architecture level, you're not just trusting that the policies will get followed.
05:49You're actually relying on a system that's incapable of certain failures.
05:56Let's move to the layer two.
05:58Probabilistic systems are great at generation.
06:01We all know that.
06:03However, they are unreliable for things that cannot can never be wrong.
06:08So, the rule is very simple.
06:10Must not fail behavior belongs above your prompt, above the model.
06:16And what does above the prompt actually mean?
06:19It means that there is a code layer that runs first on every turn before the model even runs.
06:27The code layer is what makes your irreversible decisions.
06:31The decision of, you know, whether this is an emergency escalation, should they be routed to 911, should a clinician step into the loop.
06:38All of those are irreversible decisions which need to lie at a deterministic code level layer.
06:44The model handles the long tail of your conversations and interactions with your members.
06:51The picture to hold in your head is a stack.
06:54Code on top, model below.
06:57Every turn goes through the code layer first.
07:02Most turns do reach the model, but the model never gets a vote on high stake calls.
07:13Here's how you can think about it in a different way.
07:15A model is not a guardrail.
07:18A model with a system prompt is also not a guardrail.
07:23Code that runs above the model is closer.
07:27Even the labs that be build these frontier models publish the authority hierarchy.
07:33Root, system, developer, user, guideline.
07:38Every layer above user is one prompt injection away from being overridden.
07:44If the labs themselves don't trust the prompt as a security boundary, neither should you.
07:51[snorts]
07:52So, what does live in this code layer?
07:54Let's examine it a little bit.
07:56Let's take three examples.
07:58First, very very relevant to health care, which is emergency escalation.
08:03If a member mentions self-harm, suicidal ideation, or an acute medical emergency, the system must route to 911 or 988.
08:14The model should not even see this turn.
08:17Code runs first, decides, and routes, and makes a decision right away.
08:22Another example, intent routing.
08:26Which capability in your underlying multi-agentic system, multi-agentic architecture handles a conversation turn?
08:33Is it clinical?
08:35Is it tech support?
08:36Is it education from the millions of, you know, credited articles?
08:40Is it exercise recommendation?
08:43The model can help to classify, but high-stakes path must must again take a deterministic route at the top itself.
08:53You You don't want like a clinical question quietly being routed to your generic tech support agent.
09:00That's unrecoverable.
09:04Third, identity verification.
09:08Anything that touches member data has to check that the right member is at the other end.
09:15That's an authentication check.
09:17And authentication is a security bound boundary.
09:20Prompts are not.
09:22The underlying pattern across all three, code runs first.
09:27Code makes the irreversible decisions.
09:30The model handles what's left.
09:35Last but not least, layer three.
09:38As we all know, safety is not a gate you pass once.
09:43It is a continuous layer that runs the whole time.
09:47Most teams treat evals as a pre-launch checklist.
09:51You run your tests, you ship, you move on.
09:54That's necessary, of course, but that's hardly enough.
09:58What actually holds up in production is judges that continuously keep scoring real conversations as they happen.
10:06Not a saved golden data set.
10:09Live traffic.
10:11Scored on a lot of dimensions all the time.
10:16These signals come from three sources, and each one catches something different.
10:21First, automated judges.
10:2330, 40, name it, you know, as as much as you can scale.
10:27Automated judges with multiple dimensions, always refreshing.
10:32Clinical accuracy, safety, escalation, relevance, drift, refusal, etc., etc. I can keep going on.
10:39But you you get the point.
10:41These are the automated judges that are always going to catch regressions and any even sensitive drops in quality.
10:49Second, your gold mine of information.
10:52That's going to be member feedback.
10:54Thumbs up, thumbs down on each and every single message.
10:58That's the truth signal.
10:59That's your member communicating with you.
11:02And it's the only one that comes straight from the person that you're serving it to.
11:07It catches tone problems and things that judges miss.
11:12[snorts]
11:13Third, sample traces.
11:15Random samples spread across capabilities with high-stake cases checked every single time.
11:22100% sampling on those.
11:25Ultimately, people need to read these signals.
11:30People are going to catch what no single metric is going to catch.
11:34And here's the part that nobody really warns you about.
11:37The bottleneck is not the compute, the models, the capability.
11:41It's actually having enough people to read the signal and act on it.
11:49One more thing about layer three.
11:51Some failures, you can't just prompt away.
11:54You ship the fix, it comes back under new conditions.
11:58New prompts, new tools, the model shifts.
12:01You ship the fix again.
12:03Each round buys you less and less.
12:06The rate never hits zero.
12:09At this point, monitoring is not a last resort.
12:12It is the first resort, which is always on.
12:16A new failure that you see in production simply means you now have a new judge.
12:21Your underlying architecture and your system needs to be able to keep scaling with new judges, new monitoring as you keep scaling your, you know, consumers.
12:30And
12:30[snorts]
12:30that's the point.
12:32Monitoring is how you know that the architecture is still holding.
12:37[snorts]
12:39But monitoring also tells you when the architecture is not enough.
12:44And when the architecture is not enough, a human has to decide.
12:48And this is the second part of my talk where I want to focus on the decisioning frameworks.
12:54[snorts]
12:55Let's take an example.
12:57You're about to ship, you know, uh consumer AI again in the healthcare space, and you have a feature, a specific capability that you're about to launch.
13:05And there is one issue left on the board 5 days before your launch.
13:10And you have multiple different stakeholders.
13:13Five stakeholders look at the same issue.
13:15Each one sees a different risk.
13:18And they don't agree what to do about it.
13:22Clinical sees member safety risk.
13:24They want to hold the launch.
13:27Legal sees regulatory exposure.
13:29Compliance sees audit risk.
13:32Product sees adoption risk.
13:36The If the If it ships broken, the feature won't land.
13:40And engineering sees velocity risk.
13:43They can't fix it without slipping the date they want to ship.
13:47Five rational people, five different risks, and five very different fixes.
13:54So, what do you do?
13:55Do you hold the launch and fix, or do you actually ship?
14:01The next slide is the framework I actually use for making these decisions.
14:05Five rules.
14:07This is how I think about decisions when stakeholders disagree.
14:12Rule one.
14:13Worst case always wins.
14:16Severity is set by the worst possible plausible outcome, not the average.
14:21And this is extremely relevant in health care.
14:24A bug that lightly annoys 100% of users is way less severe than one that could cause serious harm in 0.1% of cases.
14:36This is non-negotiable.
14:38The worst case matters more than the average case, always.
14:43So, when you're triaging, don't ask, "How often does this happen?"
14:48Ask, "What's the worst version of this?"
14:51That sets the severity.
14:54Rule two.
14:56Severity is not capacity.
14:58This one keeps politics out of it.
15:01As we all know, as we ship features, there's always a little bit of contention between timelines, features, deliverables.
15:07But, a bug's severity comes from the harm that it causes.
15:12Not who owns it, not whether your team has the capacity to fix it, not how hard the fix is.
15:20You have three options in front of you at this point.
15:23Fix, delay the launch, or accept the risk with explicit sign-off.
15:29Those are the three.
15:31You never quietly downgrade a bug just because you can't get to it.
15:37Rule three, asymmetric default.
15:41When you don't know what to do, always pick the safer mistake.
15:47And there are two spectrums to it.
15:48One is safety bugs and polish, the other side is polish bugs.
15:52For safety bugs, the math is one-sided.
15:56Shipping a real safety bug is much worse than delaying for a false alarm.
16:02So, for safety bugs, when you're not sure, always hold and fix.
16:07On the other side, for polish bugs, the math runs the other way.
16:13Delaying a launch costs more than shipping a small flaw.
16:18So, when you're not sure, ship in case of polish bugs.
16:22Ultimately, the framework doesn't decide for you.
16:25It just tells you which way to lean.
16:28Rule four, revealed risk tolerance, not stated risk tolerance.
16:35Your launch bar is what your org already accepts in production, not what it says it will accept.
16:43If a behavior has been live in your existing product for weeks, months, without escalation, without member complaints, without leadership concern, you cannot you cannot call it a launch blocker just for a new thing.
16:57Your stated risk tolerance might be no bugs in production, but your revealed risk tolerance is what's actually shipping today.
17:07Calibrate to the revealed one.
17:10That's the floor.
17:14Rule five, humans are the constraint.
17:18Judges scale.
17:20Pattern interpretation doesn't.
17:22Always, always design for human in the loop.
17:26Judges code traces automatically.
17:29Dashboards refresh every few hours.
17:32None of that is hard anymore.
17:34But what's hard is having enough people to read the signal and act on it.
17:41One more piece around this.
17:43Fast follows are committed debt, not an optional backlog.
17:48If you didn't ship it at launch, it's not a wish list item.
17:51It's already committed.
17:55The five rules tell you how to decide.
17:58But they all assume one thing.
18:00That your underlying signal is true.
18:03So here's the discipline that needs to come first.
18:07In a non-deterministic system, the judge is also non-deterministic.
18:13Before you trust the score, verify the scorer.
18:17And here's what it looks like in practice.
18:21Say you're watching a clinical accuracy judge in production.
18:24The score has been steady for at 4.9 for weeks.
18:29Today, it drops to 4.5. And tomorrow, it stays at 4.5. The immediate instinct is, let's start changing the prompts.
18:38The agent is broken.
18:39Let's fix the agent.
18:41That's reactive.
18:42And it's risky.
18:44You fix one thing and you break another.
18:48Worse, you're changing the agent based on a signal that might not be true.
18:54And the discipline needs to be different.
18:57First, ask whether the judge is right.
19:02We can solidify that with a with a concrete example.
19:06Uh let's take it side by side.
19:08In scenario A, same question, member asks about caffeine.
19:13The agent gives FDA standard guidance.
19:16400 mg for most adults, less if pregnant or on certain medications.
19:22The judge flags it as a hallucination.
19:25Because the agent mentioned pregnancy and medications without checking.
19:32But that's just clinical context.
19:35The judge is over calling in this case.
19:38Fix the judge in this scenario.
19:41For the same question, scenario B, the agent says 1,000 mg a day is fine.
19:47That's well above the safety limits.
19:50The judge correctly flags it.
19:52And the agent is wrong.
19:54In this case, fix the agent.
19:56The rule is always ask, is the judge right before changing the agent's response?
20:03Fixing a judge prompt is not cheating.
20:06Judges are software, too.
20:08And they need to continuously evolve.
20:10This is what production discipline looks like when the system is not deterministic.
20:18Here's the whole talk in one slide.
20:20If you screenshot one thing, this would be it.
20:23Six takeaways, three from architecture, three from decisioning.
20:27On the architecture side, the pattern is very simple.
20:31Don't X what you can Y. Don't policy what you can architect.
20:36Don't prompt what you can code.
20:38Don't gate what you can monitor.
20:42On the decisioning side, the pattern is how humans decide when the system cannot.
20:48Score by the worst case and default to the safer mistake.
20:53Calibrate to your org.
20:55And always design for the human in the loop.
20:58Fast follows are debt, not backlog.
21:03Yes, building guardrails first is slower than bolting them on later.
21:09But that's the design, not limitation.
21:12We are not building a generic low-stakes chatbot.
21:15We are building a system that has to be worthy of someone's health.
21:20The architecture is how, the decisioning is when, and member trust is why.
21:27Thank you.
21:28Let's continue the conversation on LinkedIn.
21:30Thank you.