WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XAsb7MIAzm8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XAsb7MIAzm8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:13.560
Hello everyone.

s3
00:00:13.560 --> 00:00:16.360
Welcome to Don't Be Data Poor.

s4
00:00:16.360 --> 00:00:17.520
My name's Anuj.

s5
00:00:17.520 --> 00:00:20.760
I I lead AI at Anterior.

s6
00:00:20.760 --> 00:00:28.000
Um just a bit about Anterior, we are a clinician-led AI company um built for health plans backed by Sequoia and NEA.

s7
00:00:28.000 --> 00:00:31.720
Um and what we do is we run AI transformations for health plans.

s8
00:00:31.720 --> 00:00:38.840
Um as part of which we build agents for several high-stakes healthcare administrative workflows in production.

s9
00:00:38.840 --> 00:00:49.960
Um things like prior authorization, payment integrity, HEDIS measures, etc. Um it's it's okay if you're not familiar with any of these workflows um because a lot of the work that we do

s10
00:00:49.960 --> 00:00:53.000
can actually be summarized in in the same way.

s11
00:00:53.000 --> 00:00:58.120
It's uh policy-guided decision-making over highly unstructured data.

s12
00:00:58.840 --> 00:01:00.720
And the unstructured data looks something like this, right?

s13
00:01:00.720 --> 00:01:07.480
You have uh you have these scanned fax bundles containing medical records full of patient information.

s14
00:01:07.480 --> 00:01:13.840
Um a not-so-fun fact is that I think around 70% of medical communication still happens via fax.

s15
00:01:13.840 --> 00:01:19.040
Um and fortunately or unfortunately, this is the data that we end up working with the most.

s16
00:01:19.680 --> 00:01:24.680
Um it is a very rich and information-dense data that we see here.

s17
00:01:24.680 --> 00:01:32.600
Um the data distribution here is it comes from a very long tail of rare um cases with a nuanced scenarios.

s18
00:01:32.600 --> 00:01:35.520
Uh it models an entire clinical trajectory for a patient.

s19
00:01:35.520 --> 00:01:38.520
Uh and every single person's journey is very different.

s20
00:01:38.520 --> 00:01:41.200
Uh it also presents itself in various formats.

s21
00:01:41.200 --> 00:01:50.720
So, you have like things like bad handwriting, tables, checkboxes, um key-value pairs, images, um a lot of stuff here to deal with.

s22
00:01:50.720 --> 00:01:53.600
But I personally think it's a very fascinating source of data that we see here.

s23
00:01:53.600 --> 00:01:59.560
Like it's it's it's like sort of like an observation through a very fuzzy lens over an entire person's lifespan.

s24
00:01:59.560 --> 00:02:01.880
It's really unique.

s25
00:02:02.320 --> 00:02:11.280
And I'm sure you must have heard this like enough times today already, but in healthcare the the baselines for accuracy are just exceptionally high.

s26
00:02:11.280 --> 00:02:13.000
95% is not good enough.

s27
00:02:13.000 --> 00:02:18.240
Um And at Anterior, this is why we invest very deeply in datasets and emails.

s28
00:02:18.240 --> 00:02:24.440
And these unstructured medical records are a staple source of data for these emails.

s29
00:02:24.920 --> 00:02:29.440
And we we work with this kind of data in almost every workflow that we try to automate.

s30
00:02:29.640 --> 00:02:32.000
But the problem is we can't really keep this data.

s31
00:02:32.000 --> 00:02:34.080
It's PHI, it's highly protected.

s32
00:02:34.080 --> 00:02:38.080
We can't retain it, we can't reuse it, we can't even derive information from it.

s33
00:02:38.080 --> 00:02:42.120
And most of our contracts prohibit us from from doing anything like that.

s34
00:02:42.120 --> 00:02:48.800
Um even things like redacting it, anonymizing it, and keeping derivative copies like is a strict no-no, completely off the table.

s35
00:02:48.800 --> 00:02:54.640
So, nothing really survives in any sort of dataset that we want to persist over a period of time.

s36
00:02:54.886 --> 00:02:54.959
[snorts]

s37
00:02:54.959 --> 00:03:02.720
So, so what this talk is about is like what do you do when the dataset you most need is also the data you're least allowed to keep.

s38
00:03:05.959 --> 00:03:12.720
And the and the bet that we the answer that we put our bets on is that we can kind of synthetically generate this data ourselves.

s39
00:03:13.640 --> 00:03:15.959
There's been a lot of focus on synthetic data recently.

s40
00:03:15.959 --> 00:03:23.360
You have like Frontier Labs striving to generate synthetic data for continued pre-training, for RL, for computer use, for agents.

s41
00:03:23.360 --> 00:03:27.160
So, it's it's it's a hot topic and it's it's a hot topic on our minds as well.

s42
00:03:27.160 --> 00:03:34.840
And the moment you say generate, like the first thing that comes to mind is, okay, can we can we try to use an LLM to generate synthetic data?

s43
00:03:35.360 --> 00:03:36.280
And I think you can.

s44
00:03:36.280 --> 00:03:40.920
I personally believe LLMs are a fantastic tool to generate synthetic data.

s45
00:03:40.920 --> 00:03:43.440
And several teams have already demonstrated this already.

s46
00:03:43.440 --> 00:03:52.600
There's been some papers in the healthcare space, outside the healthcare space, people have successfully used LLMs to generate synthetic data for for different purposes.

s47
00:03:52.600 --> 00:03:59.560
There are some known challenges in trying to use these elements to create data especially when you're trying to one shot the whole process.

s48
00:03:59.560 --> 00:04:04.240
Uh it's really hard to generate diverse realistic looking synthetic records.

s49
00:04:04.240 --> 00:04:08.360
And this is even more of a problem when you're trying to do when you're trying to do this at scale.

s50
00:04:08.360 --> 00:04:17.519
So, uh often times these medical records are over 300 pages long and it's like imagining if you wouldn't ask an LLM to write a novel for you in one shot, right?

s51
00:04:17.519 --> 00:04:22.640
So, it's the same reason why you wouldn't use an LLM to just one shot a synthetic record for you.

s52
00:04:22.640 --> 00:04:30.280
Um and LLMs seem to suffer from this very strange mode collapse problem when it comes to generating like diverse uh data, creative data.

s53
00:04:30.280 --> 00:04:32.720
And I think there's two main reasons for it.

s54
00:04:32.720 --> 00:04:41.120
Uh the first one is uh like I just mentioned in the talk earlier, there's very little exposure to this data source in the pre-training data corpus.

s55
00:04:41.120 --> 00:04:48.680
Um and today's objectives for pre-training and post-training are are largely uh they're only they're not incentivized for creativity or diversity really.

s56
00:04:48.680 --> 00:04:51.919
They're incentivized to be helpful systems.

s57
00:04:53.240 --> 00:05:00.080
So, with these challenges in mind, uh I'll walk you through like one of our approaches in how we uh managed to build a pipeline to generate synthetic data.

s58
00:05:00.080 --> 00:05:04.000
Um earlier I mentioned our forward tasks look something like this, right?

s59
00:05:04.000 --> 00:05:09.160
So, you have workflows and tasks that uh start with some unstructured data and a policy.

s60
00:05:09.160 --> 00:05:12.080
Um and you execute your policy against that data.

s61
00:05:12.080 --> 00:05:18.360
You follow this reasoning trace through it uh and you arrive at some sort of an outcome, which is your label.

s62
00:05:18.360 --> 00:05:21.040
So, this is our forward task.

s63
00:05:21.040 --> 00:05:24.960
Uh and the idea we had was to try and reverse this process.

s64
00:05:24.960 --> 00:05:35.320
Uh can we actually start by sampling a random label, uh figuring out a a reasoning trace for that label, and then trying to generate data backwards from that?

s65
00:05:35.320 --> 00:05:51.120
Uh the idea here being that if you can actually uh sample these two things uh with enough diversity, uh we will have we will be able to generate data that's conditioned on diverse set of inputs, allowing us to kind of circumvent the diversity problem a little bit.

s66
00:05:52.040 --> 00:05:53.520
Uh so just a quick aside on policies.

s67
00:05:53.520 --> 00:05:56.240
We've talked about policies a bit, but uh let me just clarify what these really mean, right?

s68
00:05:56.240 --> 00:06:00.720
So, this is an example policy we have for for a CPAP device for patients.

s69
00:06:00.720 --> 00:06:04.120
Uh this particular one is for a medical necessity review workflow.

s70
00:06:04.120 --> 00:06:14.160
Uh and it it sort of outlines all these diverse set of conditions uh that a patient might have um in which a CPAP device should be approved or or rejected.

s71
00:06:14.160 --> 00:06:26.960
Um so and this policy, as well as many other policies, uh you can think of these as uh essentially decision trees that outline all these sorts of conditions um um that that dictate how some outcomes are met.

s72
00:06:27.400 --> 00:06:33.440
And at Antheir, actually, we we spend a lot of time and energy in trying to model these policies explicitly as decision trees.

s73
00:06:33.440 --> 00:06:44.880
Um uh we work with uh symbolic representation uh similar to decision trees, uh and it helps us achieve a better accuracy uh and consistency score when executing them in LLM-based workflow.

s74
00:06:44.880 --> 00:06:54.600
And and the reason why I'm bringing this up is that uh by having this sort of symbolic representation of a policy, you actually have uh a way to kind of deterministically sample

s75
00:06:54.600 --> 00:06:57.960
different reasoning traces for a given outcome.

s76
00:06:59.240 --> 00:07:01.440
So, back to our idea of like reversing the process, right?

s77
00:07:01.440 --> 00:07:09.680
Uh this this sampling of reasoning traces from the policies uh what helps us get that diverse conditioning input to then generate medical records from.

s78
00:07:09.680 --> 00:07:19.520
Uh and the key idea here is that the distribution here uh that we sample from is is a much more uniform uh and effective prior distribution than what you'd normally get from an LLM.

s79
00:07:19.520 --> 00:07:27.960
Uh one added benefit of sampling this way is that, in theory, you're able to test uh for far more scenarios than you would likely get from production data sources.

s80
00:07:27.960 --> 00:07:38.640
So, what I mean by that is like say you get a sample of uh 200 cases from your customer uh um and and and and you try to like have an eval that measures performance against that, and you get a 95% score.

s81
00:07:38.640 --> 00:07:44.800
Uh it doesn't really tell you about uh what you what your performance would be in those rare edge cases that are not in that data set.

s82
00:07:44.800 --> 00:07:52.880
There'll always be rare edge cases uh that are outside the distribution just because of the fact that our data is so uh highly variant.

s83
00:07:53.200 --> 00:08:00.440
So, uh for those for those family with Cynthia, like uh they follow a similar pattern uh of sampling scenarios from a symbolic causal state representation.

s84
00:08:00.440 --> 00:08:08.240
There's a few of the folks in the space who are uh working with these symbolic representations to uh to generate diversity in synthetic data generation.

s85
00:08:09.200 --> 00:08:11.240
So, let me walk you through the rest of the pipeline.

s86
00:08:11.240 --> 00:08:11.520
All right?

s87
00:08:11.520 --> 00:08:22.320
So, uh once we have this diverse set of samples as a conditioning input, what we did was we built an LLM-based pipeline that uh follows uh a coarse-to-fine pattern

s88
00:08:22.320 --> 00:08:26.880
to progressively uh uh build up a medical record layer by layer.

s89
00:08:26.880 --> 00:08:33.720
So, here we first start with creating some patient invariants like the biological sex, the birth date, the blood group.

s90
00:08:33.760 --> 00:08:43.960
Uh we use that along with a recent trace uh with an LLM again to produce an ordered list of uh events and provider that a patient might have had, and we call this the patient journey.

s91
00:08:43.960 --> 00:08:52.160
So, this is a high-level uh you can think of it as a high-level uh overview of what a patient might have gone through in their lifespan um uh captured by a list of events

s92
00:08:52.160 --> 00:08:55.360
on a high uh in natural language.

s93
00:08:56.720 --> 00:09:03.120
And in the real world, it is actually only during these uh uh encounters provider encounters that documentation is really generated.

s94
00:09:03.120 --> 00:09:09.680
At least for the data that we get uh uh most of our data source data is generated during these provider encounters.

s95
00:09:09.680 --> 00:09:12.240
So, we model exactly that in our pipeline.

s96
00:09:12.240 --> 00:09:22.600
Uh we first generate a document plan for each encounter, and then based on that and the preceding history of the uh of the patient, we uh we fan out into generating the actual documents

s97
00:09:22.600 --> 00:09:26.440
uh um to hydrate them with actual synthetic information.

s98
00:09:26.440 --> 00:09:37.120
Uh and this coarse-to-fine layering uh is actually what allows us to keep uh uh the different prompt payloads in the pipeline uh very token efficient uh from both input and output perspective.

s99
00:09:37.120 --> 00:09:41.480
While also enabling this also helps us enable the scale across longer patient journey.

s100
00:09:41.480 --> 00:09:43.560
So, you can scale this pipeline.

s101
00:09:43.560 --> 00:09:52.000
You can have a much longer patient journey and you can just fan out and generate documents that way without overloading the context windows of your LLMs.

s102
00:09:53.800 --> 00:10:03.240
Finally, we have the sort of refinement loop in the end that we that uses a set of emails to provide feedback to improve specific parts of the generated documents.

s103
00:10:03.240 --> 00:10:09.240
For example, one of the emails we have is an LLM based check for consistency between all documents.

s104
00:10:09.240 --> 00:10:17.320
So, this makes sure that there's no contradictions or inaccuracies or conflicting information between two documents that are generated.

s105
00:10:17.320 --> 00:10:24.280
This is important because we we have a parallel fan out process that is used to generate these documents independently.

s106
00:10:24.960 --> 00:10:38.120
And because we started with the labels for this particular pipeline run, uh what we actually also have is an ability to kind of use those labels run and and and compare those against the generated

s107
00:10:38.120 --> 00:10:45.920
medical records to see if the tasks that we originally used actually matches the data is in accordance with the task inputs and outputs.

s108
00:10:45.920 --> 00:10:52.080
So, we can do this sort of round trip check to ensure that our data is actually in sync and by default get the correct labels by construction.

s109
00:10:52.080 --> 00:10:56.360
So, in theory, this is a very nice property to have.

s110
00:10:56.360 --> 00:11:04.280
Like you can basically skip the ground truth and expensive ground truth in process you need for for data for fair data.

s111
00:11:04.600 --> 00:11:11.800
One thing to clarify here is that so far all the generation has been happening just in plain text and markdown text.

s112
00:11:11.800 --> 00:11:15.520
It is possible to go from that to a rendered PDF.

s113
00:11:15.520 --> 00:11:28.160
But we don't really see much value in doing that because we have state of the art PDF parsers today available to everyone and they just allow you to convert any sort of complex PDF into a nice markdown representation.

s114
00:11:28.160 --> 00:11:34.000
So, all of this synthetic generation Um, evaluation happens in the text domain.

s115
00:11:34.080 --> 00:11:38.640
So, this is just an example of like a pipeline that we created from scratch and it's it's very easy to build.

s116
00:11:38.640 --> 00:11:40.800
It's largely fully LM based.

s117
00:11:40.800 --> 00:11:42.720
Um, but but who came up with this, right?

s118
00:11:42.720 --> 00:11:48.240
Like who who am I to uh know anything about what a good medical record looks like?

s119
00:11:48.240 --> 00:11:51.920
Uh So, how do we know if this is any good?

s120
00:11:51.920 --> 00:11:55.680
And uh I think this has been mentioned a few times today already, but like you really don't.

s121
00:11:55.680 --> 00:11:58.120
Like uh no way AI engineer would ever would.

s122
00:11:58.120 --> 00:12:03.080
Like you want your domain experts to be the ones telling you what's good, what's not good.

s123
00:12:03.080 --> 00:12:11.520
Um, and which is why we believe that uh it is of great value to empower your domain experts to own your whole data pipeline.

s124
00:12:13.800 --> 00:12:16.720
And specifically, we uh we do this in two ways, right?

s125
00:12:16.720 --> 00:12:23.760
Uh we enable our clinicians to kind of interject at each point uh in the generation process with a human-in-the-loop mechanism.

s126
00:12:23.760 --> 00:12:29.880
So, at any point, a clinician can steer the generation process to make uh a medical record in the way they want it.

s127
00:12:29.880 --> 00:12:41.920
Uh we often see our clinicians use this uh to to first look at cases that happen in production, get some interesting ideas, and then use that use those ideas along with this uh steering in this pipeline

s128
00:12:41.920 --> 00:12:46.240
to make uh cases that look similar to what we might see in production or they've seen in production.

s129
00:12:46.240 --> 00:12:49.000
And this is what makes the data generated from this really useful, right?

s130
00:12:49.000 --> 00:12:57.680
Like you can actually model your uh your failure cases um beforehand or even after they after you see them in production.

s131
00:12:58.600 --> 00:13:03.400
And secondly, I think most importantly, we let our clinicians also own the whole logic of the pipeline.

s132
00:13:03.400 --> 00:13:12.240
Um, we do this by modeling the whole pipeline as a skills-based workflow running on a generic uh agent harness that we built internally.

s133
00:13:12.480 --> 00:13:22.240
So, every every uh kind of section here you see uh all the way from the patient journey to the document generation to the document enrichment to the evals, all of these things are skills

s134
00:13:22.240 --> 00:13:25.040
uh that run on our agent harness.

s135
00:13:27.560 --> 00:13:37.200
As an example, if a clinician wanted to uh say maybe add support for a new document type, let's say for a new customer, they wanted their intake forms to look a certain way,

s136
00:13:37.200 --> 00:13:44.480
they could easily just make a new skill file for it, attach it to the pipeline, and and and voila, there there wouldn't be any engineering changes required.

s137
00:13:44.480 --> 00:13:47.640
So, it's completely clinician owned from that perspective.

s138
00:13:47.640 --> 00:13:55.080
And just in a side generally, I feel like skills are really an amazing interface between AI engineers and domain experts, especially in vertical AI.

s139
00:13:55.080 --> 00:14:04.440
We see this being we see this being modeled in several of our other workflows both for internal use cases and in production as well.

s140
00:14:07.280 --> 00:14:08.520
So, some results from this, right?

s141
00:14:08.520 --> 00:14:17.600
So, even though we only really use synthetic data for evaluation at the moment, there's already a lot of merits that we get from it.

s142
00:14:17.600 --> 00:14:21.600
Roughly 90% of our data sets are already made of synthetic data.

s143
00:14:21.600 --> 00:14:29.760
This helps us maintain a very high production accuracy score um for across many customer deployments.

s144
00:14:29.760 --> 00:14:36.960
The pipelines that we just showed you already we are able to achieve a a very high fidelity on this generated data.

s145
00:14:36.960 --> 00:14:42.720
In a blind review, clinicians were not able were only able to distinguish synthetic from real about 60% of the time.

s146
00:14:42.720 --> 00:14:47.720
So, room for improvement, but but it's but it's it's close.

s147
00:14:47.720 --> 00:14:52.240
And I'm I'm quite quite it's a quite promising avenue for us to invest more more here.

s148
00:14:52.240 --> 00:14:59.480
And and the the the the fact that is the most interesting to me and what I'm really what I'm really excited about is that all of these data sets

s149
00:14:59.480 --> 00:15:03.760
well, most of our data sets today then are created just in time for these customer deployments, right?

s150
00:15:03.760 --> 00:15:12.240
You can you when you have the ability to like create data from scratch so quickly, you can kind of you don't need to depend on on on waiting for data from your customer.

s151
00:15:12.240 --> 00:15:20.800
You can kind of just model all your edge cases, simulate them, and test your workflows before you go live with the production go live in production.

s152
00:15:22.120 --> 00:15:32.800
So, some takeaways if you're looking to build your own synthetic data pipeline in healthcare or even another domain, try reversing your inference workflow.

s153
00:15:33.040 --> 00:15:38.880
Diversity should always be sampled from a from an appropriate distribution for your use case.

s154
00:15:38.880 --> 00:15:44.640
Try to emulate the process in which data was actually generated.

s155
00:15:44.640 --> 00:15:54.360
So, like I showed you, we were trying to sort of like we were using LLMs we're trying to emulate how our medical records might actually be generated during patient encounters.

s156
00:15:54.360 --> 00:15:57.080
So, and I I would highly recommend you try doing that.

s157
00:15:57.080 --> 00:16:05.960
And the fourth most important thing I think is when you're when you're making a data pipeline like this, it's really important to give your domain experts the keys because these are the people who know

s158
00:16:05.960 --> 00:16:13.760
about your data and and and they will help you drive towards a recursive self-improvement and not the AI engineers.

s159
00:16:14.960 --> 00:16:15.360
Cool.

s160
00:16:15.360 --> 00:16:18.400
So, you don't need a PHI problem for this anywhere.

s161
00:16:18.400 --> 00:16:21.560
The data you need is ephemeral, sensitive, or even expensive to label.

s162
00:16:21.560 --> 00:16:27.680
You can think about generating data yourself and hopefully you won't be data poor.

s163
00:16:27.680 --> 00:16:29.880
Thank you, everyone.

s164
00:16:42.161 --> 00:16:44.161
[music]