Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo
https://www.youtube.com/watch?v=yqF6XhzbWBk
AI Engineer · 19 min · 264 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] This is a clinical note an AI wrote from a real consultation.
[00:17] Take a few seconds and read it.
[00:19] It reads like a routine headache.
[00:21] A new headache, likely tension type, take some paracetamol, come back if it doesn't settle.
[00:27] Looks completely fine, doesn't it?
[00:30] Here's what's missing.
[00:32] In the room, she also mentioned her jaw aches when she chews.
[00:36] A new headache, over 50 with jaw pain on chewing, that's giant cell arthritis.
[00:42] And untreated, it can take her sight within days.
[00:45] It's a same day start steroids now emergency.
[00:48] And that one line, it never made it into the note.
[00:50] On the page, it's a paracetamol headache.
[00:52] And nothing in the note is technically wrong.
[00:55] It's the dangerous part is what isn't there.
[00:59] And so that's what I'm going to talk about today.
[01:01] The dangerous failures are often the ones that actually look completely fine.
[01:07] Firstly, who am I?
[01:09] I'm Seb, medical doctor by background, and now I'm Composure, where we build AI evaluation systems for high-stakes domains.
[01:19] So, that one was a subtle kind of error, but sometimes it's not subtle at all.
[01:25] A man in his 20s sees his GP for a sore throat, tonsillitis.
[01:29] The AI writes that up.
[01:31] It gets him chest pain, suspected angina, diabetes medications he's never taken, and an address for a hospital that doesn't exist.
[01:38] And I I really like the LLM for this one.
[01:41] I think it's it's a good attempt at hospital name.
[01:44] Um and weeks later, he's invited to diabetic eye screening for diabetes he doesn't have.
[01:50] That's genuinely a real case that happened recently.
[01:54] Obviously, these kind of crazy ones someone notices, but it's those quiet ones that sit in the record uncalled that are the most challenging and can actually do a lot more damage.
[02:06] And they're not rare at all.
[02:08] In the largest real-world study of these notes, about 1 in 20 carried an error that was serious enough that it could cause significant harm to the patient.
[02:17] 1 in 20.
[02:18] That's not theoretical in testing, that's in production on real patients.
[02:23] And that's only the serious the ones.
[02:25] If you widen that lens to all errors, nearly 1 in 5 had an important omission and more than 1 in 10 had a hallucination.
[02:33] And AI is being deployed at scale across healthcare fast.
[02:37] Ambient Scribes are one of the leading cases, already in about a third of US practices and climbing.
[02:43] Physician AI use doubled last year and none of this is tracked.
[02:48] So, for most of these systems, there's no adverse event reporting at all.
[02:52] The errors never show up as incidents, they just sit in the record.
[02:55] So, errors this common that are going unseen is is quite hard for me to believe that it's not already affecting patients.
[03:03] It's not that we checked and it's fine, it's that we're flying blind.
[03:09] And this isn't just a healthcare problem, it's every high-stakes use of AI.
[03:14] Healthcare shows it more viscerally because here being confidently wrong can be life and death.
[03:20] But, everything I show you can map straight back onto other domains as well.
[03:25] So, here's what I want to do.
[03:27] I'm going to show you what exactly is going wrong, why it's going wrong, why the systems we built to catch it don't work, and a suggestion at how maybe we can start to fix that.
[03:39] So, first, what's going wrong and why?
[03:41] So, LLMs are getting good, obviously.
[03:44] They don't make stupid mistakes anymore most of the time.
[03:47] So, it's not about dumb errors.
[03:48] Everything here came out of three of the best production Ambient scribes on the market.
[03:54] Ones that we all know.
[03:55] We generated a load of notes across them last week.
[03:58] And this is exactly what's going on right now.
[04:02] This is every failure we found.
[04:03] Each dot is an error colored by type.
[04:06] Left to right, how much it matters.
[04:07] Bottom to top, whether a strong automated check catches it.
[04:11] And that split is the point.
[04:14] A handful up top get caught.
[04:16] But almost everything sits below the line.
[04:18] The ones I care about most are these on the bottom right.
[04:22] The high stakes and missed ones.
[04:26] Let me show you what a couple of those looks like.
[04:29] So, a woman comes in with a headache.
[04:33] Doctor asks, "Did it come on suddenly or build up gradually?"
[04:37] She says she doesn't know.
[04:38] It just happened.
[04:39] The note records that as abrupt sudden onset.
[04:42] And sudden onset is a red flag.
[04:44] You can see why it just happened could maybe be interpreted and inferred as abrupt onset.
[04:50] But, that's a feature that points to a bleed on the brain.
[04:53] She never said it.
[04:54] The model decided it.
[04:55] And now that one word drives the whole workup.
[05:00] Here's another.
[05:01] Doctor suggests running some tests.
[05:03] Patient says, "Can we just try try antibiotics instead?"
[05:07] They agree, hold off on the tests, treat, and see how it goes.
[05:11] Note records the opposite.
[05:13] Arrange tests today.
[05:15] It kept the plan that they talked out of, not the one they chose.
[05:19] Every line in the note reads fine because it's not really a hallucination at all.
[05:23] It's not wrong.
[05:24] It was there in the original.
[05:26] But, it's just not what they ended up deciding.
[05:30] So, why are these happening?
[05:33] There's, you know, in ambient scribes, there's first transcription and then generation.
[05:38] A lot of it does happen on the transcription layer.
[05:42] It can be words misheard for their sound-alikes.
[05:45] So, Humalog heard Humulin.
[05:48] Two insulins on completely different timelines, so swapping them could crash a blood sugar.
[05:53] Hyperthyroidism becomes hypothyroidism, the opposite condition.
[05:57] Or a drop to no on uh no evidence of cancer that becomes evidence of cancer.
[06:04] So, these these are really hard problems, and they are common.
[06:07] Not the ones I'm going to focus on, because most of what goes wrong is actually even with a perfect transcript.
[06:14] It's the model reading the words correctly and still doing one of three things.
[06:19] Either it adds something that was never said, it changes something that was, or it omits something that should be there.
[06:28] Now, the blatant version of each of these is is really easy to catch.
[06:32] The hard part in all three is the same.
[06:37] It's telling whether that thing that was added or changed or dropped actually matters.
[06:41] It's detecting that slight over inference versus the dangerous fabrication.
[06:46] The harmless rephrase versus the meaningful edit.
[06:49] A dropped line of small talk versus a dropped allergy.
[06:53] So, the ones that matter slip through along with all of the ones that don't.
[06:59] That call which different matters is taste, effectively.
[07:05] Not aesthetic taste, but essentially judgment.
[07:09] It's It's whether in this context a missed allergy might kill someone or is not important.
[07:15] And I think there's there's three properties that really matter about this.
[07:18] It's tacit, so your domain experts have it, but they can't fully write it down.
[07:23] It's contextual, so the same detail is critical in one note, noise in the next.
[07:28] And it's moving.
[07:29] The model changes, guidelines change, two good doctors disagree, different hospitals have different definitions.
[07:35] So, there's no fixed target to write down.
[07:39] And so, the model knows the facts, ultimately.
[07:42] They're extraordinarily capable, but what they lack is a sense of what matters here, for this specific example.
[07:49] And that's why even brilliant models make these mistakes.
[07:53] So, one natural move, you're never going to make that generator perfect.
[07:58] Generator is cheap.
[07:59] Generation is cheap, so stop fixing it at the source.
[08:02] Let it write, put a checker after it, pass only what clears the bar.
[08:06] And that checker should be the easier job.
[08:09] The generator has to get everything right and it pay attention to lots of varying instructions.
[08:15] Whereas the checker only has to find the one thing that's wrong and just focus on that task.
[08:19] You can also give it more time, more tokens, the exact failure modes to hunt for.
[08:23] Evaluation should be easier than generation.
[08:25] It's the asymmetry of verification, verifies law.
[08:29] That's why AI is raced ahead anyway, you can cheaply check the answer, maths and code.
[08:35] And doing this is exactly what the best teams do.
[08:38] They put a lot of energy into evaluation.
[08:40] It starts with the gold standard, which is expert humans reviewing notes, which obviously works offline, but you can't put a human on every note in production.
[08:50] So, they automate it.
[08:52] They build a serious system and some of the best versions of this that I've seen are you take the transcript and the note and context, put in front of the judge,
[09:05] a detailed rubric for faithfulness with worked, pass and fail examples.
[09:10] The rubric maybe auto-optimized with GPA or something like that.
[09:13] Maybe you have some deterministic NLP to sort of count up medical concepts that are differing between the two.
[09:20] That's a powerful system.
[09:22] And yet, I pulled all of those errors earlier out of Ambient Scribes in an afternoon.
[09:29] So, if the evaluation is this good, how are these errors still getting through?
[09:34] So, I built this system and ran those same notes through it.
[09:39] And it scored most of them fine.
[09:42] It flagged a handful of them and signed off the rest.
[09:46] [clears throat]
[09:47] But one in five of those clean passes still had some sort of serious error buried in it.
[09:52] And often that was an omission.
[09:55] The things that should have been there and actually quietly weren't.
[09:59] And that's the best version of a judge I've seen in a lot of teams and it waved them through.
[10:06] Why did it do that?
[10:07] It's not stupid.
[10:09] It's a frontier model, serious engineering behind it, more than clever enough to read the whole encounter and catch every obvious error.
[10:16] And it's not blind, either.
[10:17] And and that's part of the trap.
[10:19] If you take a note that says start amoxicillin, when the real decision was actually to wait and see, it's faithful to the words, amoxicillin did come up, but it's a lie about the intent.
[10:30] A good judge might catch that, might.
[10:34] But whether it flags that versus the other doesn't have other things that it could comment on depends on it knowing what decision matters most.
[10:42] And so it's not blind, it just can't tell what counts, essentially.
[10:48] So the note passes confidently and you put a judge like that in front of your system, you've not added a safety net, you've added a second silent failure that just nods along with the first.
[10:59] And here's the root of it.
[11:01] So in math or code, the verifier comes with free, a unit test, a compiler.
[11:08] But for is this note safe and complete, there's no unit test.
[11:12] You have to build the verifier yourself and verification is only easier than generation for the easy bit, i.e. spot the difference between transcription note.
[11:22] But that's not the hard bit.
[11:23] The hard bit is knowing of all those differences you've seen, which matter.
[11:26] And that's harder than writing that plausibly good general note in the first place.
[11:31] Because that standard of good was never written down anywhere that the judge can read it.
[11:36] A rubric that you pre-specify is only the taste you could write down.
[11:40] The taste that matters is the part that you couldn't.
[11:43] And so here's here's a bit more detail on what what matters looks like.
[11:48] Two patients, both with blood in their urine, both notes dropped the same kind of line where they'd been on holiday.
[11:55] One had been to France, the other to Lake Malawi.
[11:57] Same English emission, same shape, same mistake.
[12:01] Well, not really, because blood in the urine obviously warranted away and you're going to have to investigate it, but the France trip is irrelevant.
[12:10] The Lake Malawi trip is the diagnosis.
[12:12] Fresh water in sub-Saharan Africa means schistosomiasis until proven otherwise and it completely changes what the management plan is.
[12:19] So that same dropped line in one note is pure noise, in the other it's the answer.
[12:24] And which one it is, you simply just can't write all of that down in advance.
[12:29] So if you can't write it down, you can't write taste down, how do you get that into your evaluator and your whole application system?
[12:38] Well, we've answered a version of this before.
[12:41] RLHF exists because you can't write the reward function for good.
[12:44] You learn it from examples by showing it.
[12:47] The only question is where you keep what you've learned.
[12:50] And there's three places.
[12:52] You can either specify it up front, you can stuff the prompt, write the perfect rubric.
[12:57] We just watched that fail essentially.
[13:01] You can bake into the weights, fine-tuning or continual learning, but for a standard that's still moving and a score that has to be explainable, the weights, I think, are the wrong place to keep that.
[13:13] They go stale, they can't tell you why and you can't change them without a retrain.
[13:19] So there's the third option, which I'll show you, which is you essentially just keep the taste as the examples themselves.
[13:25] Past judgments, expert corrections, references, and for each output, you retrieve the ones that bear on it into the judges context, add one and it's live on the next call.
[13:35] You can point at exactly what moved the score.
[13:38] For this problem, it's both better and also cheaper to do.
[13:44] So, that's the way to do that is one repeating loop, three steps.
[13:48] Discover the failure modes from real outputs, capture how your experts judge them, calibrate every output against that, and when the standard moves, the loop moves with it.
[13:59] So, in more detail, discover.
[14:00] You don't write that rubric in a vacuum.
[14:03] You have to put the system in production and look at the real outputs.
[14:06] Cluster what goes wrong and the failure modes surface on their own.
[14:09] You name them.
[14:10] This is your failure mode ontology.
[14:12] Discover from your data, not guess on a whiteboard.
[14:15] And you can't shortcut it.
[14:16] The ways that a real system goes wrong are effectively unbounded and synthetic test cases only cover the failures you already imagined.
[14:25] The ones that hurt you are often the ones that you didn't.
[14:27] And you'll only find those in real outputs.
[14:30] So, this ontology is your map, what to capture judgment on, what to retrieve against, including the failures that you never thought to check for.
[14:39] After that, it's capture and then calibrate.
[14:42] So, those discovered modes, they're not a checklist that the judge runs, but they organize everything.
[14:47] What you What you ask your experts about, how you index the cases that you'll retrieve, and capturing is a simple part.
[14:55] You put real outputs in front of your experts.
[14:57] Clinicians spend a focused few hours leaving comments.
[15:00] A session doesn't have to be a month-long labeling project to start with.
[15:04] And you collect their judgment.
[15:06] Not just a score, but the reasoning and corrections.
[15:08] And over time, you build up that record of how your experts actually judge.
[15:12] You then calibrate.
[15:14] That's the the the generic part of this you can write down once easily.
[15:18] For example, be faithful or don't drop anything important.
[15:22] But what you can't write down is what counts as a serious miss for this specific note.
[15:25] That's contextual.
[15:27] And it shifts from note to note.
[15:29] So, what we recommend is you assemble that on the fly.
[15:34] For each output, your judging agent pulls in everything that bears on this one case.
[15:38] It's memory of the most similar outputs that it's judged before and how they scored, the expert corrections that apply, the reference documents and guidelines.
[15:47] Just context engineering per output.
[15:50] And crucially not just one pre-specified rubric in a vacuum, and not a model that you have to retrain every week, but a full sort of case-specific standard assembled for this output.
[16:03] And it's a loop as well.
[16:05] Every output you judge, every correction, sharpens the next.
[16:09] And when a brand new failure mode appears, Discovery surface it, and it flows straight back in.
[16:15] And so, to make that a little bit more concrete, that headache that I opened with, the one that was really a possible blindness emergency, here's the kinds of things that you would want to pull in for that note.
[16:25] The nearest cases that your experts have judged, not this exact patient, but the same shape, maybe a red flag filed as routine.
[16:33] Uh the corrections that apply, like a new headache over 50, um suggests something that you need to check red flags on, and some criteria and guidelines, and you pull
[16:44] all of that in.
[16:46] It hasn't memorized this case.
[16:48] It's a capable model.
[16:50] And handed the right context to reason from, held against that, the dropped red flag stands out.
[16:55] It was never actually hard to catch.
[16:56] It just didn't know what mattered.
[16:59] And so, if you take that same data set generated notes from the start and pass it through these three judging systems, the first, a strong off-the-shelf judge with a rubric frontier model,
[17:11] um it's better than a coin flip, but it misses most of what matters.
[17:17] The second, that sort of serious system that we talked about before, rubric, deeper, maybe some detona stick checks, better again, but still missing quite a lot of what counts.
[17:28] The third, the judge running this loop, discovered failure modes, calibrated for output against what experts judged, is performing a lot better on this specific data set.
[17:39] Same notes.
[17:41] The only thing that changes is what the judge was shown.
[17:43] And the difference here, it's not more compute or a better prompt, it's that the first two fight taste and lose.
[17:48] They guess the criteria, they freeze one standard, and they go stale.
[17:53] This repeating evolving loop does the opposite.
[17:55] It discovers the modes, fits the standard to each mode, and keeps learning.
[18:01] So, you might not write chemical notes, but if you ship anything where being confidently wrong has a cost, the contract review that misses the clauses that change the deal,
[18:12] the support agent that promises a refund you don't offer, the same thing is true for all of those.
[18:17] It's watched, if at all, by a judge with no taste for what matters in your domain.
[18:23] So, three things.
[18:26] Discover your failure modes from real outputs, don't guess them.
[18:29] Capture your experts' judgment on them, the standard that they can't write down.
[18:34] Calibrate every output against the cases that they've already judged, not a static rubric, not a retrained model.
[18:40] Then keep that loop running.
[18:42] And if you take one thing away, easiest place to start is your experts leaving free-form comments on real outputs.
[18:50] That's the real That's the raw material for everything else.
[18:54] Your judge can verify anything that you write down in advance, but the standard of good never could be.
[19:00] And so, stop trying to write it all down in advance, and just start capturing it case by case, and evolving it.
[19:07] That's why evaluation can't be a thing you build once and freeze.
[19:11] The standard it checks against doesn't exist on paper.
[19:15] It has to be discovered from real outputs captured from the people who hold it and kept alive as it moves.
[19:22] Evaluation isn't something you have, it's something that you do continuously over time.
[19:27] Thank you.
[19:29] [applause]