WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=YXEqC05WEI0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/YXEqC05WEI0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.720 --> 00:00:14.320
Hello and good morning.

s3
00:00:14.320 --> 00:00:18.480
Chetana gave us a great overview of you know what a bridge does.

s4
00:00:18.480 --> 00:00:24.840
Today I'm here to talk more from a practitioner's view of you know how we are building health care AI within Hinge Health.

s5
00:00:24.840 --> 00:00:26.840
So hi, I'm Rashi Agrawal.

s6
00:00:26.840 --> 00:00:37.160
I lead AI and ML at Hinge Health and today I will be talking about guardrails that are needed to build member-facing health care AI.

s7
00:00:39.000 --> 00:00:42.560
I want to talk a little bit about the state of health care AI right now.

s8
00:00:42.560 --> 00:00:52.920
We do have a lot of frontier models which are running and believe it or not, 40 million people actually use these models for triaging their health care issues.

s9
00:00:52.920 --> 00:01:00.040
But there is a caveat and these are some of the headlines that have been happening in the past few in the past one year or so.

s10
00:01:00.040 --> 00:01:02.520
Poisoned by a chatbot.

s11
00:01:02.520 --> 00:01:04.400
Let's start with this one person.

s12
00:01:04.400 --> 00:01:11.440
A 60-year-old healthy man asked a popular AI assistant how to cut salt from his diet.

s13
00:01:11.440 --> 00:01:15.880
The LLM told him to swap it with bromine sodium bromide.

s14
00:01:15.880 --> 00:01:17.920
He did it for 3 months.

s15
00:01:17.920 --> 00:01:21.600
He landed in the ER with paranoia and hallucinations.

s16
00:01:21.600 --> 00:01:25.080
Bromide levels 200 times the safe limit.

s17
00:01:25.080 --> 00:01:27.040
3 weeks in the hospital.

s18
00:01:27.040 --> 00:01:27.680
For what?

s19
00:01:27.680 --> 00:01:30.440
For following diet advice?

s20
00:01:30.720 --> 00:01:32.880
Let's look at another pattern.

s21
00:01:32.880 --> 00:01:46.200
The first independent safety test of a consumer health AI out of Mount Sinai found that this health AI is under triaging life-threatening emergency 50% of the times.

s22
00:01:46.200 --> 00:01:50.000
Diabetic ketoacidosis, respiratory failure.

s23
00:01:50.000 --> 00:01:54.800
And it told the people to go see a doctor in a day or two.

s24
00:01:54.800 --> 00:01:58.160
The right answer was ER right now.

s25
00:01:58.160 --> 00:02:00.160
And this is in French.

s26
00:02:00.160 --> 00:02:16.360
In February, ECRI the patient safety group that hospitals trust to rank their top risks named AI chatbot misuse as the number one health technology hazard of 2026.

s27
00:02:16.360 --> 00:02:19.600
Number one on the list that they publish every year.

s28
00:02:19.600 --> 00:02:23.400
So this is not this is not really a frontier problem.

s29
00:02:23.400 --> 00:02:27.400
This is the production baseline that we're working with right now.

s30
00:02:27.400 --> 00:02:34.480
So the question comes, how do you ship AI to somebody who's already trusted you with their health?

s31
00:02:34.480 --> 00:02:38.040
The next 20 minutes are all about that.

s32
00:02:39.440 --> 00:02:42.440
It starts with three non-negotiable foundations.

s33
00:02:42.440 --> 00:02:46.440
One, the constraint is the architecture.

s34
00:02:46.440 --> 00:02:50.440
Most AI safety failures in health care are not model failures.

s35
00:02:50.440 --> 00:02:57.160
They are architectural decisions that were made before even a single token was generated.

s36
00:02:57.160 --> 00:03:03.160
Two, deterministic rules belong above the model, not inside it.

s37
00:03:03.160 --> 00:03:08.239
What can never be wrong cannot be left to probability.

s38
00:03:08.239 --> 00:03:15.120
And three, safety is a continuous evaluation layer, not a one-time gate.

s39
00:03:15.120 --> 00:03:20.440
Launch of your product is where the real risk starts, not where it ends.

s40
00:03:20.440 --> 00:03:23.360
That's the first part of what I want to talk about today.

s41
00:03:23.360 --> 00:03:33.532
The second half is what happens when the architecture is not enough and a human has to make a decision of what ships versus what holds.

s42
00:03:33.532 --> 00:03:35.120
[snorts]

s43
00:03:35.120 --> 00:03:38.000
Let's start with layer one.

s44
00:03:38.000 --> 00:03:43.360
Protecting PHI takes both policy and architecture.

s45
00:03:43.360 --> 00:03:49.880
Policy tells you what to protect and architecture makes sure that it actually happens.

s46
00:03:49.880 --> 00:03:57.080
The first thing that changes when you start shipping member facing health AI is where PHI lives.

s47
00:03:57.080 --> 00:04:00.960
Most teams treat PHI as a runtime problem.

s48
00:04:00.960 --> 00:04:05.160
Something to redact when a log gets written to a dashboard.

s49
00:04:05.160 --> 00:04:08.040
That's the reactive version.

s50
00:04:08.560 --> 00:04:13.400
The architecture version strips PHI at the pipeline boundary.

s51
00:04:13.400 --> 00:04:18.000
At ingestion before it ever reaches the data lake.

s52
00:04:18.000 --> 00:04:21.760
By the time the data is stored, the PHI is gone.

s53
00:04:21.760 --> 00:04:26.280
So, a developer opens a dashboard there's nothing to redact.

s54
00:04:26.280 --> 00:04:28.880
The PHI was never there.

s55
00:04:28.880 --> 00:04:32.880
The rest of the architecture works in a similar way.

s56
00:04:32.880 --> 00:04:36.640
Production and non-production stay completely separate.

s57
00:04:36.640 --> 00:04:44.080
No pipes in between because even a single pipe is all that it takes for member data to leak into a dev environment.

s58
00:04:44.080 --> 00:04:47.919
And HIPAA laws are very stringent, especially in health care.

s59
00:04:47.919 --> 00:04:50.320
You know, the regulatory bar is much, much higher.

s60
00:04:50.320 --> 00:04:54.400
So, you have to be very careful about the architecture that you're designing.

s61
00:04:54.400 --> 00:04:58.560
Another big thing access depends on two things.

s62
00:04:58.560 --> 00:05:01.560
Your role and your geographic region.

s63
00:05:01.560 --> 00:05:07.040
We all work with the teams which are geographically distributed, but not everybody has access to PHI.

s64
00:05:07.040 --> 00:05:12.240
That is a certification, a policy that is applied to specific regions only.

s65
00:05:12.240 --> 00:05:18.000
An engineer outside the regulated region cannot reach raw PHI at all.

s66
00:05:18.000 --> 00:05:27.120
And the compliance rules, HIPAA, FDA's good machine learning practice, state laws like Texas, Triaga, they are not afterthoughts.

s67
00:05:27.120 --> 00:05:32.760
They are the grounding input in how you actually design your systems.

s68
00:05:32.760 --> 00:05:37.240
You cannot slap on HIPAA on top of, you know, an underlying system or an architecture.

s69
00:05:37.240 --> 00:05:42.520
You start with it and let the architecture grow around it.

s70
00:05:42.520 --> 00:05:49.360
When PHI is protected at the architecture level, you're not just trusting that the policies will get followed.

s71
00:05:49.360 --> 00:05:55.120
You're actually relying on a system that's incapable of certain failures.

s72
00:05:56.520 --> 00:05:58.800
Let's move to the layer two.

s73
00:05:58.800 --> 00:06:01.760
Probabilistic systems are great at generation.

s74
00:06:01.760 --> 00:06:03.120
We all know that.

s75
00:06:03.120 --> 00:06:08.520
However, they are unreliable for things that cannot can never be wrong.

s76
00:06:08.520 --> 00:06:10.880
So, the rule is very simple.

s77
00:06:10.880 --> 00:06:16.560
Must not fail behavior belongs above your prompt, above the model.

s78
00:06:16.560 --> 00:06:19.480
And what does above the prompt actually mean?

s79
00:06:19.480 --> 00:06:27.880
It means that there is a code layer that runs first on every turn before the model even runs.

s80
00:06:27.880 --> 00:06:31.560
The code layer is what makes your irreversible decisions.

s81
00:06:31.560 --> 00:06:38.400
The decision of, you know, whether this is an emergency escalation, should they be routed to 911, should a clinician step into the loop.

s82
00:06:38.400 --> 00:06:44.040
All of those are irreversible decisions which need to lie at a deterministic code level layer.

s83
00:06:44.040 --> 00:06:50.560
The model handles the long tail of your conversations and interactions with your members.

s84
00:06:51.280 --> 00:06:54.800
The picture to hold in your head is a stack.

s85
00:06:54.800 --> 00:06:57.760
Code on top, model below.

s86
00:06:57.760 --> 00:07:01.880
Every turn goes through the code layer first.

s87
00:07:02.919 --> 00:07:10.600
Most turns do reach the model, but the model never gets a vote on high stake calls.

s88
00:07:13.000 --> 00:07:15.720
Here's how you can think about it in a different way.

s89
00:07:15.720 --> 00:07:18.960
A model is not a guardrail.

s90
00:07:18.960 --> 00:07:23.400
A model with a system prompt is also not a guardrail.

s91
00:07:23.400 --> 00:07:27.320
Code that runs above the model is closer.

s92
00:07:27.320 --> 00:07:33.080
Even the labs that be build these frontier models publish the authority hierarchy.

s93
00:07:33.080 --> 00:07:38.040
Root, system, developer, user, guideline.

s94
00:07:38.080 --> 00:07:44.720
Every layer above user is one prompt injection away from being overridden.

s95
00:07:44.720 --> 00:07:51.240
If the labs themselves don't trust the prompt as a security boundary, neither should you.

s96
00:07:51.605 --> 00:07:52.360
[snorts]

s97
00:07:52.360 --> 00:07:54.600
So, what does live in this code layer?

s98
00:07:54.600 --> 00:07:56.360
Let's examine it a little bit.

s99
00:07:56.360 --> 00:07:58.320
Let's take three examples.

s100
00:07:58.320 --> 00:08:03.920
First, very very relevant to health care, which is emergency escalation.

s101
00:08:03.920 --> 00:08:14.400
If a member mentions self-harm, suicidal ideation, or an acute medical emergency, the system must route to 911 or 988.

s102
00:08:14.400 --> 00:08:17.360
The model should not even see this turn.

s103
00:08:17.360 --> 00:08:22.840
Code runs first, decides, and routes, and makes a decision right away.

s104
00:08:22.880 --> 00:08:26.160
Another example, intent routing.

s105
00:08:26.160 --> 00:08:33.760
Which capability in your underlying multi-agentic system, multi-agentic architecture handles a conversation turn?

s106
00:08:33.760 --> 00:08:35.080
Is it clinical?

s107
00:08:35.080 --> 00:08:36.360
Is it tech support?

s108
00:08:36.360 --> 00:08:40.360
Is it education from the millions of, you know, credited articles?

s109
00:08:40.360 --> 00:08:43.360
Is it exercise recommendation?

s110
00:08:43.440 --> 00:08:53.240
The model can help to classify, but high-stakes path must must again take a deterministic route at the top itself.

s111
00:08:53.240 --> 00:09:00.200
You You don't want like a clinical question quietly being routed to your generic tech support agent.

s112
00:09:00.200 --> 00:09:02.800
That's unrecoverable.

s113
00:09:04.320 --> 00:09:08.200
Third, identity verification.

s114
00:09:08.240 --> 00:09:15.400
Anything that touches member data has to check that the right member is at the other end.

s115
00:09:15.400 --> 00:09:17.360
That's an authentication check.

s116
00:09:17.360 --> 00:09:20.400
And authentication is a security bound boundary.

s117
00:09:20.400 --> 00:09:22.720
Prompts are not.

s118
00:09:22.720 --> 00:09:27.320
The underlying pattern across all three, code runs first.

s119
00:09:27.320 --> 00:09:30.160
Code makes the irreversible decisions.

s120
00:09:30.160 --> 00:09:33.720
The model handles what's left.

s121
00:09:35.720 --> 00:09:38.960
Last but not least, layer three.

s122
00:09:38.960 --> 00:09:43.080
As we all know, safety is not a gate you pass once.

s123
00:09:43.080 --> 00:09:47.240
It is a continuous layer that runs the whole time.

s124
00:09:47.240 --> 00:09:51.280
Most teams treat evals as a pre-launch checklist.

s125
00:09:51.280 --> 00:09:54.320
You run your tests, you ship, you move on.

s126
00:09:54.320 --> 00:09:58.840
That's necessary, of course, but that's hardly enough.

s127
00:09:58.840 --> 00:10:06.560
What actually holds up in production is judges that continuously keep scoring real conversations as they happen.

s128
00:10:06.560 --> 00:10:09.160
Not a saved golden data set.

s129
00:10:09.160 --> 00:10:11.360
Live traffic.

s130
00:10:11.360 --> 00:10:15.560
Scored on a lot of dimensions all the time.

s131
00:10:16.000 --> 00:10:21.320
These signals come from three sources, and each one catches something different.

s132
00:10:21.320 --> 00:10:23.560
First, automated judges.

s133
00:10:23.560 --> 00:10:27.320
30, 40, name it, you know, as as much as you can scale.

s134
00:10:27.320 --> 00:10:32.680
Automated judges with multiple dimensions, always refreshing.

s135
00:10:32.680 --> 00:10:39.160
Clinical accuracy, safety, escalation, relevance, drift, refusal, etc., etc. I can keep going on.

s136
00:10:39.160 --> 00:10:41.440
But you you get the point.

s137
00:10:41.440 --> 00:10:49.720
These are the automated judges that are always going to catch regressions and any even sensitive drops in quality.

s138
00:10:49.839 --> 00:10:52.760
Second, your gold mine of information.

s139
00:10:52.760 --> 00:10:54.920
That's going to be member feedback.

s140
00:10:54.920 --> 00:10:58.240
Thumbs up, thumbs down on each and every single message.

s141
00:10:58.240 --> 00:10:59.760
That's the truth signal.

s142
00:10:59.760 --> 00:11:02.320
That's your member communicating with you.

s143
00:11:02.320 --> 00:11:07.800
And it's the only one that comes straight from the person that you're serving it to.

s144
00:11:07.800 --> 00:11:12.520
It catches tone problems and things that judges miss.

s145
00:11:12.692 --> 00:11:13.400
[snorts]

s146
00:11:13.400 --> 00:11:15.800
Third, sample traces.

s147
00:11:15.800 --> 00:11:22.240
Random samples spread across capabilities with high-stake cases checked every single time.

s148
00:11:22.240 --> 00:11:25.000
100% sampling on those.

s149
00:11:25.000 --> 00:11:29.640
Ultimately, people need to read these signals.

s150
00:11:30.800 --> 00:11:34.640
People are going to catch what no single metric is going to catch.

s151
00:11:34.640 --> 00:11:37.640
And here's the part that nobody really warns you about.

s152
00:11:37.640 --> 00:11:41.840
The bottleneck is not the compute, the models, the capability.

s153
00:11:41.840 --> 00:11:48.240
It's actually having enough people to read the signal and act on it.

s154
00:11:49.400 --> 00:11:51.600
One more thing about layer three.

s155
00:11:51.600 --> 00:11:54.440
Some failures, you can't just prompt away.

s156
00:11:54.440 --> 00:11:58.360
You ship the fix, it comes back under new conditions.

s157
00:11:58.360 --> 00:12:01.360
New prompts, new tools, the model shifts.

s158
00:12:01.360 --> 00:12:03.320
You ship the fix again.

s159
00:12:03.320 --> 00:12:06.280
Each round buys you less and less.

s160
00:12:06.280 --> 00:12:09.240
The rate never hits zero.

s161
00:12:09.240 --> 00:12:12.360
At this point, monitoring is not a last resort.

s162
00:12:12.360 --> 00:12:16.280
It is the first resort, which is always on.

s163
00:12:16.280 --> 00:12:21.560
A new failure that you see in production simply means you now have a new judge.

s164
00:12:21.560 --> 00:12:30.600
Your underlying architecture and your system needs to be able to keep scaling with new judges, new monitoring as you keep scaling your, you know, consumers.

s165
00:12:30.600 --> 00:12:30.638
And

s166
00:12:30.638 --> 00:12:30.760
[snorts]

s167
00:12:30.760 --> 00:12:32.520
that's the point.

s168
00:12:32.520 --> 00:12:37.188
Monitoring is how you know that the architecture is still holding.

s169
00:12:37.188 --> 00:12:39.188
[snorts]

s170
00:12:39.480 --> 00:12:44.160
But monitoring also tells you when the architecture is not enough.

s171
00:12:44.160 --> 00:12:48.880
And when the architecture is not enough, a human has to decide.

s172
00:12:48.880 --> 00:12:54.218
And this is the second part of my talk where I want to focus on the decisioning frameworks.

s173
00:12:54.218 --> 00:12:55.960
[snorts]

s174
00:12:55.960 --> 00:12:57.400
Let's take an example.

s175
00:12:57.400 --> 00:13:05.760
You're about to ship, you know, uh consumer AI again in the healthcare space, and you have a feature, a specific capability that you're about to launch.

s176
00:13:05.760 --> 00:13:10.839
And there is one issue left on the board 5 days before your launch.

s177
00:13:10.839 --> 00:13:13.400
And you have multiple different stakeholders.

s178
00:13:13.400 --> 00:13:15.880
Five stakeholders look at the same issue.

s179
00:13:15.880 --> 00:13:18.760
Each one sees a different risk.

s180
00:13:18.760 --> 00:13:22.280
And they don't agree what to do about it.

s181
00:13:22.280 --> 00:13:24.800
Clinical sees member safety risk.

s182
00:13:24.800 --> 00:13:27.000
They want to hold the launch.

s183
00:13:27.000 --> 00:13:29.920
Legal sees regulatory exposure.

s184
00:13:29.920 --> 00:13:32.760
Compliance sees audit risk.

s185
00:13:32.760 --> 00:13:36.000
Product sees adoption risk.

s186
00:13:36.000 --> 00:13:40.280
The If the If it ships broken, the feature won't land.

s187
00:13:40.280 --> 00:13:43.520
And engineering sees velocity risk.

s188
00:13:43.520 --> 00:13:47.760
They can't fix it without slipping the date they want to ship.

s189
00:13:47.760 --> 00:13:54.800
Five rational people, five different risks, and five very different fixes.

s190
00:13:54.800 --> 00:13:55.880
So, what do you do?

s191
00:13:55.880 --> 00:14:01.160
Do you hold the launch and fix, or do you actually ship?

s192
00:14:01.160 --> 00:14:05.720
The next slide is the framework I actually use for making these decisions.

s193
00:14:05.920 --> 00:14:07.720
Five rules.

s194
00:14:07.720 --> 00:14:12.000
This is how I think about decisions when stakeholders disagree.

s195
00:14:12.000 --> 00:14:13.560
Rule one.

s196
00:14:13.560 --> 00:14:16.080
Worst case always wins.

s197
00:14:16.080 --> 00:14:21.240
Severity is set by the worst possible plausible outcome, not the average.

s198
00:14:21.240 --> 00:14:24.360
And this is extremely relevant in health care.

s199
00:14:24.360 --> 00:14:36.480
A bug that lightly annoys 100% of users is way less severe than one that could cause serious harm in 0.1% of cases.

s200
00:14:36.480 --> 00:14:38.520
This is non-negotiable.

s201
00:14:38.520 --> 00:14:43.760
The worst case matters more than the average case, always.

s202
00:14:43.760 --> 00:14:48.240
So, when you're triaging, don't ask, "How often does this happen?"

s203
00:14:48.240 --> 00:14:51.880
Ask, "What's the worst version of this?"

s204
00:14:51.880 --> 00:14:54.560
That sets the severity.

s205
00:14:54.640 --> 00:14:56.640
Rule two.

s206
00:14:56.640 --> 00:14:58.839
Severity is not capacity.

s207
00:14:58.839 --> 00:15:01.280
This one keeps politics out of it.

s208
00:15:01.280 --> 00:15:07.839
As we all know, as we ship features, there's always a little bit of contention between timelines, features, deliverables.

s209
00:15:07.839 --> 00:15:12.880
But, a bug's severity comes from the harm that it causes.

s210
00:15:12.880 --> 00:15:20.520
Not who owns it, not whether your team has the capacity to fix it, not how hard the fix is.

s211
00:15:20.520 --> 00:15:23.120
You have three options in front of you at this point.

s212
00:15:23.120 --> 00:15:29.360
Fix, delay the launch, or accept the risk with explicit sign-off.

s213
00:15:29.360 --> 00:15:31.520
Those are the three.

s214
00:15:31.520 --> 00:15:36.520
You never quietly downgrade a bug just because you can't get to it.

s215
00:15:37.960 --> 00:15:41.840
Rule three, asymmetric default.

s216
00:15:41.840 --> 00:15:46.680
When you don't know what to do, always pick the safer mistake.

s217
00:15:47.000 --> 00:15:48.880
And there are two spectrums to it.

s218
00:15:48.880 --> 00:15:52.680
One is safety bugs and polish, the other side is polish bugs.

s219
00:15:52.680 --> 00:15:56.400
For safety bugs, the math is one-sided.

s220
00:15:56.400 --> 00:16:02.280
Shipping a real safety bug is much worse than delaying for a false alarm.

s221
00:16:02.280 --> 00:16:07.960
So, for safety bugs, when you're not sure, always hold and fix.

s222
00:16:07.960 --> 00:16:13.160
On the other side, for polish bugs, the math runs the other way.

s223
00:16:13.160 --> 00:16:18.400
Delaying a launch costs more than shipping a small flaw.

s224
00:16:18.400 --> 00:16:22.920
So, when you're not sure, ship in case of polish bugs.

s225
00:16:22.920 --> 00:16:25.400
Ultimately, the framework doesn't decide for you.

s226
00:16:25.400 --> 00:16:28.760
It just tells you which way to lean.

s227
00:16:28.760 --> 00:16:35.280
Rule four, revealed risk tolerance, not stated risk tolerance.

s228
00:16:35.280 --> 00:16:43.680
Your launch bar is what your org already accepts in production, not what it says it will accept.

s229
00:16:43.680 --> 00:16:57.640
If a behavior has been live in your existing product for weeks, months, without escalation, without member complaints, without leadership concern, you cannot you cannot call it a launch blocker just for a new thing.

s230
00:16:57.640 --> 00:17:07.839
Your stated risk tolerance might be no bugs in production, but your revealed risk tolerance is what's actually shipping today.

s231
00:17:07.839 --> 00:17:10.040
Calibrate to the revealed one.

s232
00:17:10.040 --> 00:17:12.520
That's the floor.

s233
00:17:14.520 --> 00:17:18.520
Rule five, humans are the constraint.

s234
00:17:18.520 --> 00:17:20.120
Judges scale.

s235
00:17:20.120 --> 00:17:22.760
Pattern interpretation doesn't.

s236
00:17:22.760 --> 00:17:26.880
Always, always design for human in the loop.

s237
00:17:26.880 --> 00:17:29.320
Judges code traces automatically.

s238
00:17:29.320 --> 00:17:32.000
Dashboards refresh every few hours.

s239
00:17:32.000 --> 00:17:34.400
None of that is hard anymore.

s240
00:17:34.400 --> 00:17:41.400
But what's hard is having enough people to read the signal and act on it.

s241
00:17:41.400 --> 00:17:43.680
One more piece around this.

s242
00:17:43.680 --> 00:17:48.280
Fast follows are committed debt, not an optional backlog.

s243
00:17:48.280 --> 00:17:51.680
If you didn't ship it at launch, it's not a wish list item.

s244
00:17:51.680 --> 00:17:54.280
It's already committed.

s245
00:17:55.920 --> 00:17:58.720
The five rules tell you how to decide.

s246
00:17:58.720 --> 00:18:00.960
But they all assume one thing.

s247
00:18:00.960 --> 00:18:03.920
That your underlying signal is true.

s248
00:18:03.920 --> 00:18:07.520
So here's the discipline that needs to come first.

s249
00:18:07.520 --> 00:18:12.960
In a non-deterministic system, the judge is also non-deterministic.

s250
00:18:13.320 --> 00:18:17.560
Before you trust the score, verify the scorer.

s251
00:18:17.560 --> 00:18:20.920
And here's what it looks like in practice.

s252
00:18:21.400 --> 00:18:24.720
Say you're watching a clinical accuracy judge in production.

s253
00:18:24.720 --> 00:18:29.160
The score has been steady for at 4.9 for weeks.

s254
00:18:29.160 --> 00:18:38.320
Today, it drops to 4.5. And tomorrow, it stays at 4.5. The immediate instinct is, let's start changing the prompts.

s255
00:18:38.320 --> 00:18:39.400
The agent is broken.

s256
00:18:39.400 --> 00:18:41.000
Let's fix the agent.

s257
00:18:41.000 --> 00:18:42.560
That's reactive.

s258
00:18:42.560 --> 00:18:44.920
And it's risky.

s259
00:18:44.960 --> 00:18:48.280
You fix one thing and you break another.

s260
00:18:48.280 --> 00:18:54.520
Worse, you're changing the agent based on a signal that might not be true.

s261
00:18:54.520 --> 00:18:57.120
And the discipline needs to be different.

s262
00:18:57.120 --> 00:19:02.560
First, ask whether the judge is right.

s263
00:19:02.920 --> 00:19:06.480
We can solidify that with a with a concrete example.

s264
00:19:06.480 --> 00:19:08.280
Uh let's take it side by side.

s265
00:19:08.280 --> 00:19:13.080
In scenario A, same question, member asks about caffeine.

s266
00:19:13.080 --> 00:19:16.160
The agent gives FDA standard guidance.

s267
00:19:16.160 --> 00:19:22.720
400 mg for most adults, less if pregnant or on certain medications.

s268
00:19:22.720 --> 00:19:25.160
The judge flags it as a hallucination.

s269
00:19:25.160 --> 00:19:32.040
Because the agent mentioned pregnancy and medications without checking.

s270
00:19:32.360 --> 00:19:35.120
But that's just clinical context.

s271
00:19:35.120 --> 00:19:38.200
The judge is over calling in this case.

s272
00:19:38.200 --> 00:19:41.560
Fix the judge in this scenario.

s273
00:19:41.560 --> 00:19:47.880
For the same question, scenario B, the agent says 1,000 mg a day is fine.

s274
00:19:47.880 --> 00:19:50.560
That's well above the safety limits.

s275
00:19:50.560 --> 00:19:52.800
The judge correctly flags it.

s276
00:19:52.800 --> 00:19:54.080
And the agent is wrong.

s277
00:19:54.080 --> 00:19:56.440
In this case, fix the agent.

s278
00:19:56.440 --> 00:20:03.680
The rule is always ask, is the judge right before changing the agent's response?

s279
00:20:03.680 --> 00:20:06.320
Fixing a judge prompt is not cheating.

s280
00:20:06.320 --> 00:20:08.200
Judges are software, too.

s281
00:20:08.200 --> 00:20:10.880
And they need to continuously evolve.

s282
00:20:10.880 --> 00:20:16.880
This is what production discipline looks like when the system is not deterministic.

s283
00:20:18.160 --> 00:20:20.480
Here's the whole talk in one slide.

s284
00:20:20.480 --> 00:20:23.080
If you screenshot one thing, this would be it.

s285
00:20:23.080 --> 00:20:27.320
Six takeaways, three from architecture, three from decisioning.

s286
00:20:27.320 --> 00:20:31.000
On the architecture side, the pattern is very simple.

s287
00:20:31.000 --> 00:20:36.280
Don't X what you can Y. Don't policy what you can architect.

s288
00:20:36.280 --> 00:20:38.760
Don't prompt what you can code.

s289
00:20:38.760 --> 00:20:42.040
Don't gate what you can monitor.

s290
00:20:42.520 --> 00:20:48.680
On the decisioning side, the pattern is how humans decide when the system cannot.

s291
00:20:48.680 --> 00:20:53.040
Score by the worst case and default to the safer mistake.

s292
00:20:53.040 --> 00:20:55.240
Calibrate to your org.

s293
00:20:55.240 --> 00:20:58.160
And always design for the human in the loop.

s294
00:20:58.160 --> 00:21:02.560
Fast follows are debt, not backlog.

s295
00:21:03.560 --> 00:21:09.400
Yes, building guardrails first is slower than bolting them on later.

s296
00:21:09.400 --> 00:21:12.560
But that's the design, not limitation.

s297
00:21:12.560 --> 00:21:15.520
We are not building a generic low-stakes chatbot.

s298
00:21:15.520 --> 00:21:20.320
We are building a system that has to be worthy of someone's health.

s299
00:21:20.320 --> 00:21:27.160
The architecture is how, the decisioning is when, and member trust is why.

s300
00:21:27.160 --> 00:21:28.080
Thank you.

s301
00:21:28.080 --> 00:21:30.240
Let's continue the conversation on LinkedIn.

s302
00:21:30.240 --> 00:21:32.480
Thank you.