Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia

AI Engineer · 19 min · 258 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Uh hello everyone.
00:13It's really nice to see you all.
00:15Um my name is Jared and I'm going to share some of the work that we do on shipping healthcare AI safely basically.
00:23Um so just a little bit about me.
00:24Um I come from South Africa where I actually studied electrical engineering um before making the very unique decision to transition to AI a few years ago.
00:34Um I now work as a research engineer for Euphonia which is basically a healthcare company based in the UK and um the work we do I work within the science team is we build the safety and evaluation stack behind DORA
00:47which is a clinical conversational agent and my job and our job within the science team is proving that the product is safe before a patient ever actually hears it.
00:58So shipping to patients takes away the normal safety nets you would normally ship with.
01:02Um, three of them could be that you can't actually AB test on patients.
01:06Of course, randomizing patients into a worse variant is unethical and often illegal.
01:11Um, you can't undo a call.
01:13One store says it, it's been said and there is no roll back.
01:17And very importantly, the model card won't save you.
01:20Um, you can't claim like some model vendors said that they have 92% on some benchmark.
01:25Um, it's not a defense at a post instant review.
01:27So everything that we do at Euphonia comes from living within these three constraints.
01:34So what is Dora?
01:35Um a little bit about Dora.
01:36Dora is a a voice AI agent that phones patients and has real clinical conversations like posttop follow-ups or pre-op checks.
01:43Um these are calls that a clinician would usually have to make themselves.
01:46But Dora doesn't so Dora obviously doesn't replace doctors, but it just takes a hugely time consuming task off their plate and gives them their time back.
01:55Um, so far we've done around 200,000 real clinical calls within the UK across 20 hospitals and we are contracted to scale to a million patients in the next two years.
02:04Um, we also um released our product in the US this year and we are live in in in two clinics and we've signed up for six more clinics across four different states.
02:15So I'm just going to for for a bit of better context I just want to do a quick demo of what the experience with Dora is actually like.
02:22That's not uncommon after cataract surgery.
02:24Are you having trouble seeing clearly at a distance, up close, or both?
02:30Um, at a distance is fine.
02:32It's just uh up close.
02:35That's um it's a bit it's I'm having a bit of trouble with that.
02:40So, when exactly did you first notice it was a bit blurry when trying to read or see things up close?
02:47Um, well, really just after the surgery, and I was wondering how long it would last.
02:57So, you're asking about the duration.
02:59That's understandable.
03:00Blurry vision should improve rapidly in the first few days after surgery.
03:04Okay.
03:05Um, also, um, is it all right if I go swimming again?
03:09You should avoid swimming for a month after surgery.
03:12That's just to be on the safe side and help your eye heal properly.
03:16All right.
03:16Okay.
03:18Awesome.
03:22So, as you just heard, sometimes Dora asks about symptoms, it gives advice, sometimes it it answers questions.
03:28And when it does this, this makes it what's called a medical device.
03:31And therefore, it has to, of course, be regulated.
03:34Um, and we have to prove that it's safe.
03:36And regulation essentially bottles down to three main questions.
03:40What does your software do?
03:42What could go wrong?
03:43And how do you ensure that it doesn't?
03:45For normal software, it's quite tractable the question.
03:47But for a model that talks to actual patients, what could go wrong is quite huge.
03:53So where do we start?
03:54We start from what could go wrong.
03:55We start from the harm.
03:56What could actually harm a patient?
03:58And let's look at some examples.
04:00Um Dora could miss a red flag symptom such as sudden vision loss or severe pain.
04:06A patient could ask a medical question and Dora invents an answer, hallucinates something.
04:11The patient could be distressed and Dora just ignores it and carries on without actually acknowledging the distress.
04:16There's many many many documented hazards of these 20 30 40 and we have to ensure that none of them actually happen in real life.
04:25So how would we actually normally catch a problem like this before it actually spreads?
04:30We would lean usually on the playbook that most software ships on.
04:33You ship to a small percentage of people.
04:36You watch the dashboard.
04:37You roll back if it breaks and you iterate from there.
04:40This is a very good playbook.
04:41It's reactive.
04:42It's fast.
04:42It's very safe.
04:43And it's how the industry usually derisks a launch.
04:46But there's a hidden assumption here that it only works because you can afford to be wrong for an instant.
04:52A bad change hits a few users.
04:53You can quickly catch it.
04:54You can roll back and no one's actually literally harmed.
04:57This is the one assumption um is why that it breaks when when the actual user is a patient.
05:03for 5% that could be a hundred if not thousands of of patients that have got unproven changes and un undue care roll back you can't really roll back the call has already happened the person has already been harmed
05:15by watching the dashboards the dashboards are just going to going red means that a patient was actually hurt so the reactive loop is actually gone now so how do you iterate at all when you can't touch a patient until you're sure
05:28well for this we started looking at other at at examples from other high reliable industries ries.
05:34The most obvious one is self-driving cars.
05:36Obviously, we're in SF now.
05:37There's a lot of Whimos driving around.
05:39They've only just come to London, unfortunately, very very late to the to the party.
05:42But what did what did self-driving cars do?
05:45Well, they didn't just drive around crashing into walls and say, "We won't do that again."
05:49And then doing another RL loop.
05:51They put millions of miles of simulations first before they actually got any um passages in into the car.
05:58For us, we believe in the same thing.
06:01Simulation is only real.
06:02the real ethical option we can go with.
06:04You can't run all the hazard the hazards I just mentioned on real people as a first grasp.
06:10So for our clinical history taking we built a sim simulation framework called matrix and I'm going to work through how it works and how we use it to prove that our product is safe.
06:19And the paper's on archive if you want to read it along with some of the other research that we do.
06:24At its core, Matrix recreates a real clinical work conversation but with no real patient in it.
06:31We use an LLM to play the patient.
06:33We call it patbot.
06:35And what does it do?
06:36We use a simulated patient and not a hired actor because hired actors don't scale.
06:41If we want to iterate very fast and simulate different things at the same time while also updating our system, um, hiring actors would just be too slow of a process.
06:50So, as a first version, we just use a simulated patient.
06:54The simulated patient is conditioned on the actual scenario we want to test.
06:59The scenario defines exactly what the patient should try and do when talking to our agent.
07:04For example, asking whether the agent is a human or a or an AI.
07:09Patbot then has a conversation with Dora, our target system, and then generate simulator dialogues.
07:15Very importantly, this all happens under a very specific clinical use case context.
07:19So, the scenarios are grounded in real clinical workflows and not abstract situations.
07:25So, how do we actually make sure that the the patient is realistic?
07:31If if Patbot is is sounding robotic, the tests aren't really worth much.
07:35So, we had to, of course, validate it.
07:37The first thing we did was just a pure um script adherence check.
07:41If we told Patbot to do something, does Patbot do it?
07:44Yes or no?
07:46This helped us filter out a lot of maybe weaker models that that were that didn't listen to instructions properly.
07:51But just purely um following instructions does not make a realistic patient.
07:55We want a patient that flows more realistically like a like a real person.
08:00So we set up what's called a PPI study, a patient and public involvement study.
08:05We took real patients and we showed them two sets of conversations.
08:08One conversation was between a real doctor and a real patient and one conversation was between Dora and Patbot within our matrix framework.
08:17and we showed them these two examples side by side and said, "Looking at the patient, can you tell which one is the real person and which one is the simulated person?"
08:24So, I'm going to just wait for a few seconds here.
08:26If you guys want to quickly read the two conversations, um maybe we can do a do a hands up.
08:31Who thinks conversation A is the real person?
08:35Who thinks who thinks conversation B is the real person?
08:40Okay, I think us as as engineers sometimes are pretty good at at finding these things, but um it was actually much more difficult than we thought.
08:47And we did this with four conversation sets.
08:50Basically, in three out of the four, the majority of people actually thought that the simulated patient was more realistic.
08:56But the most important thing that we found was of course there is no single realistic patient.
09:01That doesn't really make sense.
09:02Some people prefer to speak more verbosely, a lot of ums and os.
09:06Some people are more straight to the point, a lot of yeses and nos.
09:09But the point is that we actually want to simulate all these different scenarios.
09:12We want to simulate people with very diverse personas.
09:16Um, but what it did show us is that at least our patot was realistic enough um for this simulation.
09:23Okay, so now you've got thousands and thousands of simulated dialogues.
09:27Are us as the engineers going to go read through them one by one and see if a hazard happened?
09:31Of course not.
09:32It doesn't scale at all for one and number two we on clinician so we don't actually know if an actual hazard has really occurred.
09:39So we use another LLM as a judge of course and we call it bev judge.
09:43It takes the simulated dialogue a set of expected behaviors and the hazardous scenarios that we talk through with clinicians and it makes a judgment pass or fail.
09:52If it fails it gives us a reason why it gave that answer.
09:55So we give we get a structured output of which hazard were triggered and what actually went wrong in that scenario.
10:02So how did we validate bev judge?
10:04Uh we val we we validated bev judge against expert clinicians.
10:07We had we created a corpus of 240 examples and we had a ground truth of whether a hazard existed in these conversations yes or no. Then we got 10 clinicians from 10 clinical specialties
10:18to to label them for whether they had a hazard or not.
10:20And we did the same thing with a judge.
10:22And the results showed that our judge is at least on par if not slightly better than the the real expert clinicians.
10:28The top model which as of a year ago when we wrote the paper was Gemini 2.5 Pro.
10:33Now we've maybe updated the models.
10:35Um it achieved an F1 score of of 0.96.
10:38And even maybe more importantly it achieved almost perfect sensitivity.
10:43Um sensitivity being a very important metric to to healthcare and and to clinicians of course because you want to make 100% sure almost that there's that no hazards appear in a conversation.
10:52You would rather overall hazards that aren't there than underall hazards that are there.
10:56So now we have an automated judge that performs at expert layer at at expert level and this is actually what makes this whole process scalable.
11:05Okay.
11:06So now matrix can grade thousands of conversations but grading isn't technically improving the product.
11:12A pile of pass fails tells you where door breaks and where is not safe but doesn't actually make the product better.
11:17So how do you do this without experimenting on the patient?
11:20The answer isn't like maybe uh very long ago in our world eight eight months ago we would manually prompt engineer this.
11:27We would look at which agents are going wrong or which prompts are going wrong and we would have to manually prompt engineer.
11:33But we know that prompt brittleleness is real and it's it's quite absurd.
11:37Formatting changes alone have been seen to swing a benchmark by 76 percentage points.
11:42And reordering few shot examples flips a model from near random, so near 50% to near state-of-the-art on some benchmarks.
11:49and hand tuning can't survive that.
11:51It's very subjective.
11:52It's not reproducible and very importantly it's extremely timeconuming.
11:56So over the last year or so there's been these prompt optimizers that have started to come out and we've focused on those.
12:02The one that we use the most is is JPE which stands for genetic PTOA.
12:06It comes from the same people who made DSPY if anyone knows about them.
12:10And how does JPE work?
12:12You essentially define a metric for what good is which I'll I'll get into a bit later.
12:17Then um you you pass your data through through um through through Jeppa and and it tells you which examples failed.
12:26Then you get a very strong LLM to reflect on the failures and update the prompt automatically.
12:31You do this over and over and over again and it it keeps what they call a PTO frontier of the best prompts until your budget has been exhausted and you've now come up with with what what Jeba comes up with the best prompt.
12:43So we believe this is a much better process from both both a time consuming process.
12:48You know it takes maybe manual prompt engineering would take in the order of hours to days to this is in the hour of minutes.
12:54Normally between 30 and an hour minutes and an hour you get an optimized prompt and very importantly it's reproducible and there's a very clear order trail and clear feedback loop and if anything goes wrong you can just maybe it's purely now a data science problem.
13:09It's mainly focused on the data, how to make your data right, the feature engineering and you define the actual metric along with the clinicians.
13:18So how do you actually know what good is?
13:21Um it's not a flat accuracy score.
13:24You don't want just an average of how your whole data set did.
13:27You give it a cost matrix.
13:29So let's go back to our sensitivity metric.
13:31Let's say it's very important for clinicians to understand when when and where a red flag is present.
13:38If a red flag is present and you correctly catch it, that's good.
13:41If you if you miss it, it is it could be catastrophic.
13:44If there's no actual red flag and it overcauses um um that there's a red flag there.
13:50It's just mildly annoying to the patient.
13:52They may need to ask answer a couple extra questions, but it's not a cat catastrophic harm situation.
13:59So, we what can we do?
14:00We can optimize for sensitivity.
14:02we can we can work with the feedback metric and we can just make it um give a higher reward for finding the red flags and a lower reward for um for missing them.
14:11So you can optimize for certain metrics.
14:13You can also optimize for something else that that a that a a clinician might want.
14:17They might want to optimize for accuracy or might want to optimize for some other metric.
14:21All you have to do is recompile the prompt and then you got a a new um optimized prompt.
14:28So remember earlier when we said the re we feel like the reactive loop is gone.
14:32You ship, watch and roll back.
14:34This is what we believe replaces it.
14:36We take real calls, real data.
14:39We um we use then synthetic edge cases which may not come up in real calls such as like rare symptoms or or misranscriptions.
14:46We get an optimized prompt through through Jeff or some other prompt optimizer.
14:51Then we pass it through through something like matrix as a simulation safety gate.
14:56If anything fails or anything doesn't look right, we can redo that whole process, redo the data or or re label or get more data.
15:04And then after that and you're happy with that, then you can do some gated deploy which we'll get into a bit.
15:09Um the most important thing here is that it's a flywheel.
15:12Every single deployment and every new call produces more call data.
15:15So your system is consistently improving.
15:20So as we said with with our matrix framework, we use simulated patient.
15:25But however realistic you think they are, of course they are not real patients.
15:29Passing every test in simulation doesn't prove that Dora actually helps someone in real life.
15:33The things might come up in real life that that you can't get in simulation.
15:37It only earns the right to actually try carefully.
15:40Simulation is the inner loop.
15:41It's fast.
15:42It's free.
15:43You can do thousands of runs before anyone actually real is is exposed.
15:47But real patients are the outer loop and that's where the only real proof is.
15:51So simulation is necessary but it's not sufficient.
15:55Simulation earns the right to to to test on real people and then eventually real patients.
16:00But you don't just flip a switch.
16:01You cross it in stages and each stage earns the the right for the next.
16:05After you've done your simulations and you're happy with the results, you might do a round of user testing.
16:10Then you get supervised clinical evaluation based on those tests and you and you base it on real patients.
16:16You can do some voice actors, but of course the the most realistic is to get real patients.
16:20But in this step, it's very important that there's clinicians at every step in the loop.
16:25Then you can do some deployment, but it's still monitored.
16:28And how much autonomy you allow the system to do depends on your evidence.
16:32As the system gets more evidence, you can give it more independence.
16:37And underneath all of this, um, every call, every data set, and every pinned, um, prompt, every judge verdict traces back to the exact hazard that it addresses.
16:46That's the real deliverable.
16:48The important thing is that you don't ship the model, you ship the evidence when trying to regulate.
16:53So, what can you take back to your own stacks both in healthcare and other areas?
16:56You first have to define exactly what harm is for your product.
17:00You have to manufacture your rare but dangerous cases.
17:03Don't wait for them to to happen naturally.
17:05Make your evaluation metric your real cost function to optimize.
17:09Pin your prompt versions and keep the traces.
17:12These are the important things.
17:14The important thing also is that the the work is never done.
17:17As you move into new modalities or new languages, there'll always be new hazards that start to arise.
17:22But with something like matrix and a a prompt optimization loop, um you can use the same framework as new modalities arise.
17:30You know, when you move into voice, for example, there's things like back channeling and interruptions, which which breaks a lot of the the texts.
17:38For example, an agent might be midsafety advice, like you must avoid bright lights, and when the patient maybe um um cuts in with some um out of scope question,
17:49and weaker models usually just forget about the safety advice and just ask answer the next question.
17:54But Matrix captures these things.
17:56Um, other things that that usually go wrong that we've tested is that um the the patient could the the Dora or or an agent could be halfway through giving some safety advice and a back channel just cause the model to just completely
18:10ignore the safety advice and and stop right there and and wait for the patient to to say something else.
18:15The failure models the failure modes change the framework doesn't.
18:18You still blackbox the system, write down the new hazards and simulate and judge them exactly the we did over text.
18:25Voice is just a new module in the same safety case and we actually use matrix in the voice space to do these exact same things.
18:31So whatever modality you move into, you're not starting over.
18:34The same approach finds the hazards before a real user does.
18:38So again, you simulate before it ever touches a patient and you use an optimization loop to actually improve the system.
18:45Thank you very very much for coming to listen.
18:47Um I'm very very open to to talking more about this and if you want to connect during the conference or after, there's my LinkedIn and socials.
18:54Thank you very much.
18:55[applause]