Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo

AI Engineer · 19 min · 264 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12This is a clinical note an AI wrote from a real consultation.
00:17Take a few seconds and read it.
00:19It reads like a routine headache.
00:21A new headache, likely tension type, take some paracetamol, come back if it doesn't settle.
00:27Looks completely fine, doesn't it?
00:30Here's what's missing.
00:32In the room, she also mentioned her jaw aches when she chews.
00:36A new headache, over 50 with jaw pain on chewing, that's giant cell arthritis.
00:42And untreated, it can take her sight within days.
00:45It's a same day start steroids now emergency.
00:48And that one line, it never made it into the note.
00:50On the page, it's a paracetamol headache.
00:52And nothing in the note is technically wrong.
00:55It's the dangerous part is what isn't there.
00:59And so that's what I'm going to talk about today.
01:01The dangerous failures are often the ones that actually look completely fine.
01:07Firstly, who am I?
01:09I'm Seb, medical doctor by background, and now I'm Composure, where we build AI evaluation systems for high-stakes domains.
01:19So, that one was a subtle kind of error, but sometimes it's not subtle at all.
01:25A man in his 20s sees his GP for a sore throat, tonsillitis.
01:29The AI writes that up.
01:31It gets him chest pain, suspected angina, diabetes medications he's never taken, and an address for a hospital that doesn't exist.
01:38And I I really like the LLM for this one.
01:41I think it's it's a good attempt at hospital name.
01:44Um and weeks later, he's invited to diabetic eye screening for diabetes he doesn't have.
01:50That's genuinely a real case that happened recently.
01:54Obviously, these kind of crazy ones someone notices, but it's those quiet ones that sit in the record uncalled that are the most challenging and can actually do a lot more damage.
02:06And they're not rare at all.
02:08In the largest real-world study of these notes, about 1 in 20 carried an error that was serious enough that it could cause significant harm to the patient.
02:171 in 20.
02:18That's not theoretical in testing, that's in production on real patients.
02:23And that's only the serious the ones.
02:25If you widen that lens to all errors, nearly 1 in 5 had an important omission and more than 1 in 10 had a hallucination.
02:33And AI is being deployed at scale across healthcare fast.
02:37Ambient Scribes are one of the leading cases, already in about a third of US practices and climbing.
02:43Physician AI use doubled last year and none of this is tracked.
02:48So, for most of these systems, there's no adverse event reporting at all.
02:52The errors never show up as incidents, they just sit in the record.
02:55So, errors this common that are going unseen is is quite hard for me to believe that it's not already affecting patients.
03:03It's not that we checked and it's fine, it's that we're flying blind.
03:09And this isn't just a healthcare problem, it's every high-stakes use of AI.
03:14Healthcare shows it more viscerally because here being confidently wrong can be life and death.
03:20But, everything I show you can map straight back onto other domains as well.
03:25So, here's what I want to do.
03:27I'm going to show you what exactly is going wrong, why it's going wrong, why the systems we built to catch it don't work, and a suggestion at how maybe we can start to fix that.
03:39So, first, what's going wrong and why?
03:41So, LLMs are getting good, obviously.
03:44They don't make stupid mistakes anymore most of the time.
03:47So, it's not about dumb errors.
03:48Everything here came out of three of the best production Ambient scribes on the market.
03:54Ones that we all know.
03:55We generated a load of notes across them last week.
03:58And this is exactly what's going on right now.
04:02This is every failure we found.
04:03Each dot is an error colored by type.
04:06Left to right, how much it matters.
04:07Bottom to top, whether a strong automated check catches it.
04:11And that split is the point.
04:14A handful up top get caught.
04:16But almost everything sits below the line.
04:18The ones I care about most are these on the bottom right.
04:22The high stakes and missed ones.
04:26Let me show you what a couple of those looks like.
04:29So, a woman comes in with a headache.
04:33Doctor asks, "Did it come on suddenly or build up gradually?"
04:37She says she doesn't know.
04:38It just happened.
04:39The note records that as abrupt sudden onset.
04:42And sudden onset is a red flag.
04:44You can see why it just happened could maybe be interpreted and inferred as abrupt onset.
04:50But, that's a feature that points to a bleed on the brain.
04:53She never said it.
04:54The model decided it.
04:55And now that one word drives the whole workup.
05:00Here's another.
05:01Doctor suggests running some tests.
05:03Patient says, "Can we just try try antibiotics instead?"
05:07They agree, hold off on the tests, treat, and see how it goes.
05:11Note records the opposite.
05:13Arrange tests today.
05:15It kept the plan that they talked out of, not the one they chose.
05:19Every line in the note reads fine because it's not really a hallucination at all.
05:23It's not wrong.
05:24It was there in the original.
05:26But, it's just not what they ended up deciding.
05:30So, why are these happening?
05:33There's, you know, in ambient scribes, there's first transcription and then generation.
05:38A lot of it does happen on the transcription layer.
05:42It can be words misheard for their sound-alikes.
05:45So, Humalog heard Humulin.
05:48Two insulins on completely different timelines, so swapping them could crash a blood sugar.
05:53Hyperthyroidism becomes hypothyroidism, the opposite condition.
05:57Or a drop to no on uh no evidence of cancer that becomes evidence of cancer.
06:04So, these these are really hard problems, and they are common.
06:07Not the ones I'm going to focus on, because most of what goes wrong is actually even with a perfect transcript.
06:14It's the model reading the words correctly and still doing one of three things.
06:19Either it adds something that was never said, it changes something that was, or it omits something that should be there.
06:28Now, the blatant version of each of these is is really easy to catch.
06:32The hard part in all three is the same.
06:37It's telling whether that thing that was added or changed or dropped actually matters.
06:41It's detecting that slight over inference versus the dangerous fabrication.
06:46The harmless rephrase versus the meaningful edit.
06:49A dropped line of small talk versus a dropped allergy.
06:53So, the ones that matter slip through along with all of the ones that don't.
06:59That call which different matters is taste, effectively.
07:05Not aesthetic taste, but essentially judgment.
07:09It's It's whether in this context a missed allergy might kill someone or is not important.
07:15And I think there's there's three properties that really matter about this.
07:18It's tacit, so your domain experts have it, but they can't fully write it down.
07:23It's contextual, so the same detail is critical in one note, noise in the next.
07:28And it's moving.
07:29The model changes, guidelines change, two good doctors disagree, different hospitals have different definitions.
07:35So, there's no fixed target to write down.
07:39And so, the model knows the facts, ultimately.
07:42They're extraordinarily capable, but what they lack is a sense of what matters here, for this specific example.
07:49And that's why even brilliant models make these mistakes.
07:53So, one natural move, you're never going to make that generator perfect.
07:58Generator is cheap.
07:59Generation is cheap, so stop fixing it at the source.
08:02Let it write, put a checker after it, pass only what clears the bar.
08:06And that checker should be the easier job.
08:09The generator has to get everything right and it pay attention to lots of varying instructions.
08:15Whereas the checker only has to find the one thing that's wrong and just focus on that task.
08:19You can also give it more time, more tokens, the exact failure modes to hunt for.
08:23Evaluation should be easier than generation.
08:25It's the asymmetry of verification, verifies law.
08:29That's why AI is raced ahead anyway, you can cheaply check the answer, maths and code.
08:35And doing this is exactly what the best teams do.
08:38They put a lot of energy into evaluation.
08:40It starts with the gold standard, which is expert humans reviewing notes, which obviously works offline, but you can't put a human on every note in production.
08:50So, they automate it.
08:52They build a serious system and some of the best versions of this that I've seen are you take the transcript and the note and context, put in front of the judge,
09:05a detailed rubric for faithfulness with worked, pass and fail examples.
09:10The rubric maybe auto-optimized with GPA or something like that.
09:13Maybe you have some deterministic NLP to sort of count up medical concepts that are differing between the two.
09:20That's a powerful system.
09:22And yet, I pulled all of those errors earlier out of Ambient Scribes in an afternoon.
09:29So, if the evaluation is this good, how are these errors still getting through?
09:34So, I built this system and ran those same notes through it.
09:39And it scored most of them fine.
09:42It flagged a handful of them and signed off the rest.
09:46[clears throat]
09:47But one in five of those clean passes still had some sort of serious error buried in it.
09:52And often that was an omission.
09:55The things that should have been there and actually quietly weren't.
09:59And that's the best version of a judge I've seen in a lot of teams and it waved them through.
10:06Why did it do that?
10:07It's not stupid.
10:09It's a frontier model, serious engineering behind it, more than clever enough to read the whole encounter and catch every obvious error.
10:16And it's not blind, either.
10:17And and that's part of the trap.
10:19If you take a note that says start amoxicillin, when the real decision was actually to wait and see, it's faithful to the words, amoxicillin did come up, but it's a lie about the intent.
10:30A good judge might catch that, might.
10:34But whether it flags that versus the other doesn't have other things that it could comment on depends on it knowing what decision matters most.
10:42And so it's not blind, it just can't tell what counts, essentially.
10:48So the note passes confidently and you put a judge like that in front of your system, you've not added a safety net, you've added a second silent failure that just nods along with the first.
10:59And here's the root of it.
11:01So in math or code, the verifier comes with free, a unit test, a compiler.
11:08But for is this note safe and complete, there's no unit test.
11:12You have to build the verifier yourself and verification is only easier than generation for the easy bit, i.e. spot the difference between transcription note.
11:22But that's not the hard bit.
11:23The hard bit is knowing of all those differences you've seen, which matter.
11:26And that's harder than writing that plausibly good general note in the first place.
11:31Because that standard of good was never written down anywhere that the judge can read it.
11:36A rubric that you pre-specify is only the taste you could write down.
11:40The taste that matters is the part that you couldn't.
11:43And so here's here's a bit more detail on what what matters looks like.
11:48Two patients, both with blood in their urine, both notes dropped the same kind of line where they'd been on holiday.
11:55One had been to France, the other to Lake Malawi.
11:57Same English emission, same shape, same mistake.
12:01Well, not really, because blood in the urine obviously warranted away and you're going to have to investigate it, but the France trip is irrelevant.
12:10The Lake Malawi trip is the diagnosis.
12:12Fresh water in sub-Saharan Africa means schistosomiasis until proven otherwise and it completely changes what the management plan is.
12:19So that same dropped line in one note is pure noise, in the other it's the answer.
12:24And which one it is, you simply just can't write all of that down in advance.
12:29So if you can't write it down, you can't write taste down, how do you get that into your evaluator and your whole application system?
12:38Well, we've answered a version of this before.
12:41RLHF exists because you can't write the reward function for good.
12:44You learn it from examples by showing it.
12:47The only question is where you keep what you've learned.
12:50And there's three places.
12:52You can either specify it up front, you can stuff the prompt, write the perfect rubric.
12:57We just watched that fail essentially.
13:01You can bake into the weights, fine-tuning or continual learning, but for a standard that's still moving and a score that has to be explainable, the weights, I think, are the wrong place to keep that.
13:13They go stale, they can't tell you why and you can't change them without a retrain.
13:19So there's the third option, which I'll show you, which is you essentially just keep the taste as the examples themselves.
13:25Past judgments, expert corrections, references, and for each output, you retrieve the ones that bear on it into the judges context, add one and it's live on the next call.
13:35You can point at exactly what moved the score.
13:38For this problem, it's both better and also cheaper to do.
13:44So, that's the way to do that is one repeating loop, three steps.
13:48Discover the failure modes from real outputs, capture how your experts judge them, calibrate every output against that, and when the standard moves, the loop moves with it.
13:59So, in more detail, discover.
14:00You don't write that rubric in a vacuum.
14:03You have to put the system in production and look at the real outputs.
14:06Cluster what goes wrong and the failure modes surface on their own.
14:09You name them.
14:10This is your failure mode ontology.
14:12Discover from your data, not guess on a whiteboard.
14:15And you can't shortcut it.
14:16The ways that a real system goes wrong are effectively unbounded and synthetic test cases only cover the failures you already imagined.
14:25The ones that hurt you are often the ones that you didn't.
14:27And you'll only find those in real outputs.
14:30So, this ontology is your map, what to capture judgment on, what to retrieve against, including the failures that you never thought to check for.
14:39After that, it's capture and then calibrate.
14:42So, those discovered modes, they're not a checklist that the judge runs, but they organize everything.
14:47What you What you ask your experts about, how you index the cases that you'll retrieve, and capturing is a simple part.
14:55You put real outputs in front of your experts.
14:57Clinicians spend a focused few hours leaving comments.
15:00A session doesn't have to be a month-long labeling project to start with.
15:04And you collect their judgment.
15:06Not just a score, but the reasoning and corrections.
15:08And over time, you build up that record of how your experts actually judge.
15:12You then calibrate.
15:14That's the the the generic part of this you can write down once easily.
15:18For example, be faithful or don't drop anything important.
15:22But what you can't write down is what counts as a serious miss for this specific note.
15:25That's contextual.
15:27And it shifts from note to note.
15:29So, what we recommend is you assemble that on the fly.
15:34For each output, your judging agent pulls in everything that bears on this one case.
15:38It's memory of the most similar outputs that it's judged before and how they scored, the expert corrections that apply, the reference documents and guidelines.
15:47Just context engineering per output.
15:50And crucially not just one pre-specified rubric in a vacuum, and not a model that you have to retrain every week, but a full sort of case-specific standard assembled for this output.
16:03And it's a loop as well.
16:05Every output you judge, every correction, sharpens the next.
16:09And when a brand new failure mode appears, Discovery surface it, and it flows straight back in.
16:15And so, to make that a little bit more concrete, that headache that I opened with, the one that was really a possible blindness emergency, here's the kinds of things that you would want to pull in for that note.
16:25The nearest cases that your experts have judged, not this exact patient, but the same shape, maybe a red flag filed as routine.
16:33Uh the corrections that apply, like a new headache over 50, um suggests something that you need to check red flags on, and some criteria and guidelines, and you pull
16:44all of that in.
16:46It hasn't memorized this case.
16:48It's a capable model.
16:50And handed the right context to reason from, held against that, the dropped red flag stands out.
16:55It was never actually hard to catch.
16:56It just didn't know what mattered.
16:59And so, if you take that same data set generated notes from the start and pass it through these three judging systems, the first, a strong off-the-shelf judge with a rubric frontier model,
17:11um it's better than a coin flip, but it misses most of what matters.
17:17The second, that sort of serious system that we talked about before, rubric, deeper, maybe some detona stick checks, better again, but still missing quite a lot of what counts.
17:28The third, the judge running this loop, discovered failure modes, calibrated for output against what experts judged, is performing a lot better on this specific data set.
17:39Same notes.
17:41The only thing that changes is what the judge was shown.
17:43And the difference here, it's not more compute or a better prompt, it's that the first two fight taste and lose.
17:48They guess the criteria, they freeze one standard, and they go stale.
17:53This repeating evolving loop does the opposite.
17:55It discovers the modes, fits the standard to each mode, and keeps learning.
18:01So, you might not write chemical notes, but if you ship anything where being confidently wrong has a cost, the contract review that misses the clauses that change the deal,
18:12the support agent that promises a refund you don't offer, the same thing is true for all of those.
18:17It's watched, if at all, by a judge with no taste for what matters in your domain.
18:23So, three things.
18:26Discover your failure modes from real outputs, don't guess them.
18:29Capture your experts' judgment on them, the standard that they can't write down.
18:34Calibrate every output against the cases that they've already judged, not a static rubric, not a retrained model.
18:40Then keep that loop running.
18:42And if you take one thing away, easiest place to start is your experts leaving free-form comments on real outputs.
18:50That's the real That's the raw material for everything else.
18:54Your judge can verify anything that you write down in advance, but the standard of good never could be.
19:00And so, stop trying to write it all down in advance, and just start capturing it case by case, and evolving it.
19:07That's why evaluation can't be a thing you build once and freeze.
19:11The standard it checks against doesn't exist on paper.
19:15It has to be discovered from real outputs captured from the people who hold it and kept alive as it moves.
19:22Evaluation isn't something you have, it's something that you do continuously over time.
19:27Thank you.
19:29[applause]