WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=McknwOzbmyg

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/McknwOzbmyg.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.800 --> 00:00:13.840
Uh hello everyone.

s3
00:00:13.840 --> 00:00:15.839
It's really nice to see you all.

s4
00:00:15.839 --> 00:00:23.279
Um my name is Jared and I'm going to share some of the work that we do on shipping healthcare AI safely basically.

s5
00:00:23.279 --> 00:00:24.800
Um so just a little bit about me.

s6
00:00:24.800 --> 00:00:34.079
Um I come from South Africa where I actually studied electrical engineering um before making the very unique decision to transition to AI a few years ago.

s7
00:00:34.079 --> 00:00:47.200
Um I now work as a research engineer for Euphonia which is basically a healthcare company based in the UK and um the work we do I work within the science team is we build the safety and evaluation stack behind DORA

s8
00:00:47.200 --> 00:00:57.840
which is a clinical conversational agent and my job and our job within the science team is proving that the product is safe before a patient ever actually hears it.

s9
00:00:58.079 --> 00:01:02.239
So shipping to patients takes away the normal safety nets you would normally ship with.

s10
00:01:02.239 --> 00:01:06.000
Um, three of them could be that you can't actually AB test on patients.

s11
00:01:06.000 --> 00:01:11.600
Of course, randomizing patients into a worse variant is unethical and often illegal.

s12
00:01:11.600 --> 00:01:13.600
Um, you can't undo a call.

s13
00:01:13.600 --> 00:01:17.200
One store says it, it's been said and there is no roll back.

s14
00:01:17.200 --> 00:01:20.640
And very importantly, the model card won't save you.

s15
00:01:20.640 --> 00:01:25.119
Um, you can't claim like some model vendors said that they have 92% on some benchmark.

s16
00:01:25.119 --> 00:01:27.920
Um, it's not a defense at a post instant review.

s17
00:01:27.920 --> 00:01:33.520
So everything that we do at Euphonia comes from living within these three constraints.

s18
00:01:34.000 --> 00:01:35.200
So what is Dora?

s19
00:01:35.200 --> 00:01:36.320
Um a little bit about Dora.

s20
00:01:36.320 --> 00:01:43.119
Dora is a a voice AI agent that phones patients and has real clinical conversations like posttop follow-ups or pre-op checks.

s21
00:01:43.119 --> 00:01:46.479
Um these are calls that a clinician would usually have to make themselves.

s22
00:01:46.479 --> 00:01:55.040
But Dora doesn't so Dora obviously doesn't replace doctors, but it just takes a hugely time consuming task off their plate and gives them their time back.

s23
00:01:55.040 --> 00:02:04.799
Um, so far we've done around 200,000 real clinical calls within the UK across 20 hospitals and we are contracted to scale to a million patients in the next two years.

s24
00:02:04.799 --> 00:02:14.560
Um, we also um released our product in the US this year and we are live in in in two clinics and we've signed up for six more clinics across four different states.

s25
00:02:15.200 --> 00:02:22.560
So I'm just going to for for a bit of better context I just want to do a quick demo of what the experience with Dora is actually like.

s26
00:02:22.560 --> 00:02:24.879
That's not uncommon after cataract surgery.

s27
00:02:24.879 --> 00:02:30.080
Are you having trouble seeing clearly at a distance, up close, or both?

s28
00:02:30.080 --> 00:02:32.800
Um, at a distance is fine.

s29
00:02:32.800 --> 00:02:35.200
It's just uh up close.

s30
00:02:35.200 --> 00:02:40.480
That's um it's a bit it's I'm having a bit of trouble with that.

s31
00:02:40.480 --> 00:02:47.760
So, when exactly did you first notice it was a bit blurry when trying to read or see things up close?

s32
00:02:47.760 --> 00:02:57.040
Um, well, really just after the surgery, and I was wondering how long it would last.

s33
00:02:57.360 --> 00:02:59.360
So, you're asking about the duration.

s34
00:02:59.360 --> 00:03:00.480
That's understandable.

s35
00:03:00.480 --> 00:03:04.560
Blurry vision should improve rapidly in the first few days after surgery.

s36
00:03:04.560 --> 00:03:05.120
Okay.

s37
00:03:05.120 --> 00:03:09.760
Um, also, um, is it all right if I go swimming again?

s38
00:03:09.760 --> 00:03:12.720
You should avoid swimming for a month after surgery.

s39
00:03:12.720 --> 00:03:16.239
That's just to be on the safe side and help your eye heal properly.

s40
00:03:16.239 --> 00:03:16.879
All right.

s41
00:03:16.879 --> 00:03:18.720
Okay.

s42
00:03:18.720 --> 00:03:20.720
Awesome.

s43
00:03:22.640 --> 00:03:28.959
So, as you just heard, sometimes Dora asks about symptoms, it gives advice, sometimes it it answers questions.

s44
00:03:28.959 --> 00:03:31.920
And when it does this, this makes it what's called a medical device.

s45
00:03:31.920 --> 00:03:34.640
And therefore, it has to, of course, be regulated.

s46
00:03:34.640 --> 00:03:36.799
Um, and we have to prove that it's safe.

s47
00:03:36.799 --> 00:03:40.560
And regulation essentially bottles down to three main questions.

s48
00:03:40.560 --> 00:03:42.080
What does your software do?

s49
00:03:42.080 --> 00:03:43.120
What could go wrong?

s50
00:03:43.120 --> 00:03:45.200
And how do you ensure that it doesn't?

s51
00:03:45.200 --> 00:03:47.599
For normal software, it's quite tractable the question.

s52
00:03:47.599 --> 00:03:53.120
But for a model that talks to actual patients, what could go wrong is quite huge.

s53
00:03:53.120 --> 00:03:54.319
So where do we start?

s54
00:03:54.319 --> 00:03:55.920
We start from what could go wrong.

s55
00:03:55.920 --> 00:03:56.799
We start from the harm.

s56
00:03:56.799 --> 00:03:58.640
What could actually harm a patient?

s57
00:03:58.640 --> 00:04:00.959
And let's look at some examples.

s58
00:04:00.959 --> 00:04:06.720
Um Dora could miss a red flag symptom such as sudden vision loss or severe pain.

s59
00:04:06.720 --> 00:04:11.120
A patient could ask a medical question and Dora invents an answer, hallucinates something.

s60
00:04:11.120 --> 00:04:16.639
The patient could be distressed and Dora just ignores it and carries on without actually acknowledging the distress.

s61
00:04:16.639 --> 00:04:25.919
There's many many many documented hazards of these 20 30 40 and we have to ensure that none of them actually happen in real life.

s62
00:04:25.919 --> 00:04:30.400
So how would we actually normally catch a problem like this before it actually spreads?

s63
00:04:30.400 --> 00:04:33.840
We would lean usually on the playbook that most software ships on.

s64
00:04:33.840 --> 00:04:36.240
You ship to a small percentage of people.

s65
00:04:36.240 --> 00:04:37.520
You watch the dashboard.

s66
00:04:37.520 --> 00:04:40.320
You roll back if it breaks and you iterate from there.

s67
00:04:40.320 --> 00:04:41.680
This is a very good playbook.

s68
00:04:41.680 --> 00:04:42.400
It's reactive.

s69
00:04:42.400 --> 00:04:42.880
It's fast.

s70
00:04:42.880 --> 00:04:43.759
It's very safe.

s71
00:04:43.759 --> 00:04:46.960
And it's how the industry usually derisks a launch.

s72
00:04:46.960 --> 00:04:52.000
But there's a hidden assumption here that it only works because you can afford to be wrong for an instant.

s73
00:04:52.000 --> 00:04:53.600
A bad change hits a few users.

s74
00:04:53.600 --> 00:04:54.720
You can quickly catch it.

s75
00:04:54.720 --> 00:04:57.759
You can roll back and no one's actually literally harmed.

s76
00:04:57.759 --> 00:05:03.520
This is the one assumption um is why that it breaks when when the actual user is a patient.

s77
00:05:03.520 --> 00:05:15.840
for 5% that could be a hundred if not thousands of of patients that have got unproven changes and un undue care roll back you can't really roll back the call has already happened the person has already been harmed

s78
00:05:15.840 --> 00:05:28.320
by watching the dashboards the dashboards are just going to going red means that a patient was actually hurt so the reactive loop is actually gone now so how do you iterate at all when you can't touch a patient until you're sure

s79
00:05:28.320 --> 00:05:34.320
well for this we started looking at other at at examples from other high reliable industries ries.

s80
00:05:34.320 --> 00:05:36.479
The most obvious one is self-driving cars.

s81
00:05:36.479 --> 00:05:37.759
Obviously, we're in SF now.

s82
00:05:37.759 --> 00:05:39.360
There's a lot of Whimos driving around.

s83
00:05:39.360 --> 00:05:42.720
They've only just come to London, unfortunately, very very late to the to the party.

s84
00:05:42.720 --> 00:05:45.759
But what did what did self-driving cars do?

s85
00:05:45.759 --> 00:05:49.199
Well, they didn't just drive around crashing into walls and say, "We won't do that again."

s86
00:05:49.199 --> 00:05:51.680
And then doing another RL loop.

s87
00:05:51.680 --> 00:05:58.639
They put millions of miles of simulations first before they actually got any um passages in into the car.

s88
00:05:58.639 --> 00:06:01.120
For us, we believe in the same thing.

s89
00:06:01.120 --> 00:06:02.400
Simulation is only real.

s90
00:06:02.400 --> 00:06:04.479
the real ethical option we can go with.

s91
00:06:04.479 --> 00:06:10.160
You can't run all the hazard the hazards I just mentioned on real people as a first grasp.

s92
00:06:10.160 --> 00:06:19.600
So for our clinical history taking we built a sim simulation framework called matrix and I'm going to work through how it works and how we use it to prove that our product is safe.

s93
00:06:19.600 --> 00:06:24.800
And the paper's on archive if you want to read it along with some of the other research that we do.

s94
00:06:24.800 --> 00:06:31.440
At its core, Matrix recreates a real clinical work conversation but with no real patient in it.

s95
00:06:31.440 --> 00:06:33.440
We use an LLM to play the patient.

s96
00:06:33.440 --> 00:06:35.360
We call it patbot.

s97
00:06:35.360 --> 00:06:36.479
And what does it do?

s98
00:06:36.479 --> 00:06:41.520
We use a simulated patient and not a hired actor because hired actors don't scale.

s99
00:06:41.520 --> 00:06:50.080
If we want to iterate very fast and simulate different things at the same time while also updating our system, um, hiring actors would just be too slow of a process.

s100
00:06:50.080 --> 00:06:54.560
So, as a first version, we just use a simulated patient.

s101
00:06:54.639 --> 00:06:59.120
The simulated patient is conditioned on the actual scenario we want to test.

s102
00:06:59.120 --> 00:07:04.160
The scenario defines exactly what the patient should try and do when talking to our agent.

s103
00:07:04.160 --> 00:07:09.599
For example, asking whether the agent is a human or a or an AI.

s104
00:07:09.599 --> 00:07:15.360
Patbot then has a conversation with Dora, our target system, and then generate simulator dialogues.

s105
00:07:15.360 --> 00:07:19.919
Very importantly, this all happens under a very specific clinical use case context.

s106
00:07:19.919 --> 00:07:25.599
So, the scenarios are grounded in real clinical workflows and not abstract situations.

s107
00:07:25.919 --> 00:07:31.039
So, how do we actually make sure that the the patient is realistic?

s108
00:07:31.280 --> 00:07:35.039
If if Patbot is is sounding robotic, the tests aren't really worth much.

s109
00:07:35.039 --> 00:07:37.360
So, we had to, of course, validate it.

s110
00:07:37.360 --> 00:07:41.599
The first thing we did was just a pure um script adherence check.

s111
00:07:41.599 --> 00:07:44.319
If we told Patbot to do something, does Patbot do it?

s112
00:07:44.319 --> 00:07:46.000
Yes or no?

s113
00:07:46.000 --> 00:07:51.599
This helped us filter out a lot of maybe weaker models that that were that didn't listen to instructions properly.

s114
00:07:51.599 --> 00:07:55.919
But just purely um following instructions does not make a realistic patient.

s115
00:07:55.919 --> 00:08:00.639
We want a patient that flows more realistically like a like a real person.

s116
00:08:00.639 --> 00:08:05.280
So we set up what's called a PPI study, a patient and public involvement study.

s117
00:08:05.280 --> 00:08:08.879
We took real patients and we showed them two sets of conversations.

s118
00:08:08.879 --> 00:08:17.120
One conversation was between a real doctor and a real patient and one conversation was between Dora and Patbot within our matrix framework.

s119
00:08:17.120 --> 00:08:24.800
and we showed them these two examples side by side and said, "Looking at the patient, can you tell which one is the real person and which one is the simulated person?"

s120
00:08:24.800 --> 00:08:26.479
So, I'm going to just wait for a few seconds here.

s121
00:08:26.479 --> 00:08:31.120
If you guys want to quickly read the two conversations, um maybe we can do a do a hands up.

s122
00:08:31.120 --> 00:08:35.200
Who thinks conversation A is the real person?

s123
00:08:35.519 --> 00:08:40.320
Who thinks who thinks conversation B is the real person?

s124
00:08:40.560 --> 00:08:47.839
Okay, I think us as as engineers sometimes are pretty good at at finding these things, but um it was actually much more difficult than we thought.

s125
00:08:47.839 --> 00:08:50.720
And we did this with four conversation sets.

s126
00:08:50.720 --> 00:08:56.800
Basically, in three out of the four, the majority of people actually thought that the simulated patient was more realistic.

s127
00:08:56.800 --> 00:09:01.279
But the most important thing that we found was of course there is no single realistic patient.

s128
00:09:01.279 --> 00:09:02.720
That doesn't really make sense.

s129
00:09:02.720 --> 00:09:06.480
Some people prefer to speak more verbosely, a lot of ums and os.

s130
00:09:06.480 --> 00:09:09.519
Some people are more straight to the point, a lot of yeses and nos.

s131
00:09:09.519 --> 00:09:12.800
But the point is that we actually want to simulate all these different scenarios.

s132
00:09:12.800 --> 00:09:16.399
We want to simulate people with very diverse personas.

s133
00:09:16.399 --> 00:09:23.040
Um, but what it did show us is that at least our patot was realistic enough um for this simulation.

s134
00:09:23.839 --> 00:09:27.279
Okay, so now you've got thousands and thousands of simulated dialogues.

s135
00:09:27.279 --> 00:09:31.360
Are us as the engineers going to go read through them one by one and see if a hazard happened?

s136
00:09:31.360 --> 00:09:32.080
Of course not.

s137
00:09:32.080 --> 00:09:39.040
It doesn't scale at all for one and number two we on clinician so we don't actually know if an actual hazard has really occurred.

s138
00:09:39.040 --> 00:09:43.120
So we use another LLM as a judge of course and we call it bev judge.

s139
00:09:43.120 --> 00:09:52.240
It takes the simulated dialogue a set of expected behaviors and the hazardous scenarios that we talk through with clinicians and it makes a judgment pass or fail.

s140
00:09:52.240 --> 00:09:55.600
If it fails it gives us a reason why it gave that answer.

s141
00:09:55.600 --> 00:10:02.240
So we give we get a structured output of which hazard were triggered and what actually went wrong in that scenario.

s142
00:10:02.800 --> 00:10:04.480
So how did we validate bev judge?

s143
00:10:04.480 --> 00:10:07.760
Uh we val we we validated bev judge against expert clinicians.

s144
00:10:07.760 --> 00:10:18.000
We had we created a corpus of 240 examples and we had a ground truth of whether a hazard existed in these conversations yes or no. Then we got 10 clinicians from 10 clinical specialties

s145
00:10:18.000 --> 00:10:20.640
to to label them for whether they had a hazard or not.

s146
00:10:20.640 --> 00:10:22.640
And we did the same thing with a judge.

s147
00:10:22.640 --> 00:10:28.959
And the results showed that our judge is at least on par if not slightly better than the the real expert clinicians.

s148
00:10:28.959 --> 00:10:33.279
The top model which as of a year ago when we wrote the paper was Gemini 2.5 Pro.

s149
00:10:33.279 --> 00:10:35.120
Now we've maybe updated the models.

s150
00:10:35.120 --> 00:10:38.800
Um it achieved an F1 score of of 0.96.

s151
00:10:38.800 --> 00:10:43.200
And even maybe more importantly it achieved almost perfect sensitivity.

s152
00:10:43.200 --> 00:10:52.640
Um sensitivity being a very important metric to to healthcare and and to clinicians of course because you want to make 100% sure almost that there's that no hazards appear in a conversation.

s153
00:10:52.640 --> 00:10:56.959
You would rather overall hazards that aren't there than underall hazards that are there.

s154
00:10:56.959 --> 00:11:05.760
So now we have an automated judge that performs at expert layer at at expert level and this is actually what makes this whole process scalable.

s155
00:11:05.839 --> 00:11:06.399
Okay.

s156
00:11:06.399 --> 00:11:12.079
So now matrix can grade thousands of conversations but grading isn't technically improving the product.

s157
00:11:12.079 --> 00:11:17.519
A pile of pass fails tells you where door breaks and where is not safe but doesn't actually make the product better.

s158
00:11:17.519 --> 00:11:20.640
So how do you do this without experimenting on the patient?

s159
00:11:20.640 --> 00:11:27.040
The answer isn't like maybe uh very long ago in our world eight eight months ago we would manually prompt engineer this.

s160
00:11:27.040 --> 00:11:33.360
We would look at which agents are going wrong or which prompts are going wrong and we would have to manually prompt engineer.

s161
00:11:33.360 --> 00:11:37.279
But we know that prompt brittleleness is real and it's it's quite absurd.

s162
00:11:37.279 --> 00:11:42.000
Formatting changes alone have been seen to swing a benchmark by 76 percentage points.

s163
00:11:42.000 --> 00:11:49.200
And reordering few shot examples flips a model from near random, so near 50% to near state-of-the-art on some benchmarks.

s164
00:11:49.200 --> 00:11:51.360
and hand tuning can't survive that.

s165
00:11:51.360 --> 00:11:52.480
It's very subjective.

s166
00:11:52.480 --> 00:11:56.640
It's not reproducible and very importantly it's extremely timeconuming.

s167
00:11:56.640 --> 00:12:02.320
So over the last year or so there's been these prompt optimizers that have started to come out and we've focused on those.

s168
00:12:02.320 --> 00:12:06.240
The one that we use the most is is JPE which stands for genetic PTOA.

s169
00:12:06.240 --> 00:12:10.880
It comes from the same people who made DSPY if anyone knows about them.

s170
00:12:10.880 --> 00:12:12.320
And how does JPE work?

s171
00:12:12.320 --> 00:12:17.200
You essentially define a metric for what good is which I'll I'll get into a bit later.

s172
00:12:17.200 --> 00:12:26.079
Then um you you pass your data through through um through through Jeppa and and it tells you which examples failed.

s173
00:12:26.079 --> 00:12:31.680
Then you get a very strong LLM to reflect on the failures and update the prompt automatically.

s174
00:12:31.680 --> 00:12:43.760
You do this over and over and over again and it it keeps what they call a PTO frontier of the best prompts until your budget has been exhausted and you've now come up with with what what Jeba comes up with the best prompt.

s175
00:12:43.760 --> 00:12:48.560
So we believe this is a much better process from both both a time consuming process.

s176
00:12:48.560 --> 00:12:54.560
You know it takes maybe manual prompt engineering would take in the order of hours to days to this is in the hour of minutes.

s177
00:12:54.560 --> 00:13:09.120
Normally between 30 and an hour minutes and an hour you get an optimized prompt and very importantly it's reproducible and there's a very clear order trail and clear feedback loop and if anything goes wrong you can just maybe it's purely now a data science problem.

s178
00:13:09.120 --> 00:13:17.440
It's mainly focused on the data, how to make your data right, the feature engineering and you define the actual metric along with the clinicians.

s179
00:13:18.399 --> 00:13:21.920
So how do you actually know what good is?

s180
00:13:21.920 --> 00:13:24.240
Um it's not a flat accuracy score.

s181
00:13:24.240 --> 00:13:27.600
You don't want just an average of how your whole data set did.

s182
00:13:27.600 --> 00:13:29.440
You give it a cost matrix.

s183
00:13:29.440 --> 00:13:31.920
So let's go back to our sensitivity metric.

s184
00:13:31.920 --> 00:13:38.000
Let's say it's very important for clinicians to understand when when and where a red flag is present.

s185
00:13:38.000 --> 00:13:41.839
If a red flag is present and you correctly catch it, that's good.

s186
00:13:41.839 --> 00:13:44.959
If you if you miss it, it is it could be catastrophic.

s187
00:13:44.959 --> 00:13:50.720
If there's no actual red flag and it overcauses um um that there's a red flag there.

s188
00:13:50.720 --> 00:13:52.320
It's just mildly annoying to the patient.

s189
00:13:52.320 --> 00:13:59.040
They may need to ask answer a couple extra questions, but it's not a cat catastrophic harm situation.

s190
00:13:59.040 --> 00:14:00.079
So, we what can we do?

s191
00:14:00.079 --> 00:14:02.240
We can optimize for sensitivity.

s192
00:14:02.240 --> 00:14:11.839
we can we can work with the feedback metric and we can just make it um give a higher reward for finding the red flags and a lower reward for um for missing them.

s193
00:14:11.839 --> 00:14:13.920
So you can optimize for certain metrics.

s194
00:14:13.920 --> 00:14:17.839
You can also optimize for something else that that a that a a clinician might want.

s195
00:14:17.839 --> 00:14:21.920
They might want to optimize for accuracy or might want to optimize for some other metric.

s196
00:14:21.920 --> 00:14:28.000
All you have to do is recompile the prompt and then you got a a new um optimized prompt.

s197
00:14:28.320 --> 00:14:32.480
So remember earlier when we said the re we feel like the reactive loop is gone.

s198
00:14:32.480 --> 00:14:34.320
You ship, watch and roll back.

s199
00:14:34.320 --> 00:14:36.320
This is what we believe replaces it.

s200
00:14:36.320 --> 00:14:39.040
We take real calls, real data.

s201
00:14:39.040 --> 00:14:46.800
We um we use then synthetic edge cases which may not come up in real calls such as like rare symptoms or or misranscriptions.

s202
00:14:46.880 --> 00:14:51.360
We get an optimized prompt through through Jeff or some other prompt optimizer.

s203
00:14:51.360 --> 00:14:56.399
Then we pass it through through something like matrix as a simulation safety gate.

s204
00:14:56.399 --> 00:15:04.000
If anything fails or anything doesn't look right, we can redo that whole process, redo the data or or re label or get more data.

s205
00:15:04.000 --> 00:15:09.120
And then after that and you're happy with that, then you can do some gated deploy which we'll get into a bit.

s206
00:15:09.120 --> 00:15:12.079
Um the most important thing here is that it's a flywheel.

s207
00:15:12.079 --> 00:15:15.440
Every single deployment and every new call produces more call data.

s208
00:15:15.440 --> 00:15:19.040
So your system is consistently improving.

s209
00:15:20.240 --> 00:15:25.120
So as we said with with our matrix framework, we use simulated patient.

s210
00:15:25.120 --> 00:15:29.920
But however realistic you think they are, of course they are not real patients.

s211
00:15:29.920 --> 00:15:33.760
Passing every test in simulation doesn't prove that Dora actually helps someone in real life.

s212
00:15:33.760 --> 00:15:37.199
The things might come up in real life that that you can't get in simulation.

s213
00:15:37.199 --> 00:15:40.320
It only earns the right to actually try carefully.

s214
00:15:40.320 --> 00:15:41.760
Simulation is the inner loop.

s215
00:15:41.760 --> 00:15:42.480
It's fast.

s216
00:15:42.480 --> 00:15:43.040
It's free.

s217
00:15:43.040 --> 00:15:47.279
You can do thousands of runs before anyone actually real is is exposed.

s218
00:15:47.279 --> 00:15:51.040
But real patients are the outer loop and that's where the only real proof is.

s219
00:15:51.040 --> 00:15:55.040
So simulation is necessary but it's not sufficient.

s220
00:15:55.040 --> 00:16:00.079
Simulation earns the right to to to test on real people and then eventually real patients.

s221
00:16:00.079 --> 00:16:01.600
But you don't just flip a switch.

s222
00:16:01.600 --> 00:16:05.519
You cross it in stages and each stage earns the the right for the next.

s223
00:16:05.519 --> 00:16:10.240
After you've done your simulations and you're happy with the results, you might do a round of user testing.

s224
00:16:10.240 --> 00:16:16.240
Then you get supervised clinical evaluation based on those tests and you and you base it on real patients.

s225
00:16:16.240 --> 00:16:20.800
You can do some voice actors, but of course the the most realistic is to get real patients.

s226
00:16:20.800 --> 00:16:25.199
But in this step, it's very important that there's clinicians at every step in the loop.

s227
00:16:25.199 --> 00:16:28.399
Then you can do some deployment, but it's still monitored.

s228
00:16:28.399 --> 00:16:32.720
And how much autonomy you allow the system to do depends on your evidence.

s229
00:16:32.720 --> 00:16:37.120
As the system gets more evidence, you can give it more independence.

s230
00:16:37.120 --> 00:16:46.560
And underneath all of this, um, every call, every data set, and every pinned, um, prompt, every judge verdict traces back to the exact hazard that it addresses.

s231
00:16:46.560 --> 00:16:48.000
That's the real deliverable.

s232
00:16:48.000 --> 00:16:53.040
The important thing is that you don't ship the model, you ship the evidence when trying to regulate.

s233
00:16:53.040 --> 00:16:56.880
So, what can you take back to your own stacks both in healthcare and other areas?

s234
00:16:56.880 --> 00:17:00.399
You first have to define exactly what harm is for your product.

s235
00:17:00.399 --> 00:17:03.600
You have to manufacture your rare but dangerous cases.

s236
00:17:03.600 --> 00:17:05.919
Don't wait for them to to happen naturally.

s237
00:17:05.919 --> 00:17:09.919
Make your evaluation metric your real cost function to optimize.

s238
00:17:09.919 --> 00:17:12.400
Pin your prompt versions and keep the traces.

s239
00:17:12.400 --> 00:17:14.880
These are the important things.

s240
00:17:14.880 --> 00:17:17.439
The important thing also is that the the work is never done.

s241
00:17:17.439 --> 00:17:22.799
As you move into new modalities or new languages, there'll always be new hazards that start to arise.

s242
00:17:22.799 --> 00:17:30.559
But with something like matrix and a a prompt optimization loop, um you can use the same framework as new modalities arise.

s243
00:17:30.559 --> 00:17:38.080
You know, when you move into voice, for example, there's things like back channeling and interruptions, which which breaks a lot of the the texts.

s244
00:17:38.080 --> 00:17:49.120
For example, an agent might be midsafety advice, like you must avoid bright lights, and when the patient maybe um um cuts in with some um out of scope question,

s245
00:17:49.120 --> 00:17:54.000
and weaker models usually just forget about the safety advice and just ask answer the next question.

s246
00:17:54.000 --> 00:17:56.799
But Matrix captures these things.

s247
00:17:56.799 --> 00:18:10.160
Um, other things that that usually go wrong that we've tested is that um the the patient could the the Dora or or an agent could be halfway through giving some safety advice and a back channel just cause the model to just completely

s248
00:18:10.160 --> 00:18:15.440
ignore the safety advice and and stop right there and and wait for the patient to to say something else.

s249
00:18:15.440 --> 00:18:18.880
The failure models the failure modes change the framework doesn't.

s250
00:18:18.880 --> 00:18:25.200
You still blackbox the system, write down the new hazards and simulate and judge them exactly the we did over text.

s251
00:18:25.200 --> 00:18:31.200
Voice is just a new module in the same safety case and we actually use matrix in the voice space to do these exact same things.

s252
00:18:31.200 --> 00:18:34.400
So whatever modality you move into, you're not starting over.

s253
00:18:34.400 --> 00:18:38.400
The same approach finds the hazards before a real user does.

s254
00:18:38.400 --> 00:18:45.360
So again, you simulate before it ever touches a patient and you use an optimization loop to actually improve the system.

s255
00:18:45.360 --> 00:18:47.520
Thank you very very much for coming to listen.

s256
00:18:47.520 --> 00:18:54.480
Um I'm very very open to to talking more about this and if you want to connect during the conference or after, there's my LinkedIn and socials.

s257
00:18:54.480 --> 00:18:55.783
Thank you very much.

s258
00:18:55.783 --> 00:18:57.783
[applause]