WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=_cVfz88_j7A NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/_cVfz88_j7A.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.760 --> 00:00:02.080 Hi everyone. s2 00:00:02.080 --> 00:00:03.560 My name is Anant Shankar. s3 00:00:03.560 --> 00:00:12.280 I am an AI engineer at Trisca and I'll be talking about oncology automating oncology workflows from end to end. s4 00:00:12.800 --> 00:00:16.760 So at Trisca, we are automating various workflows in oncology. s5 00:00:16.760 --> 00:00:25.200 One of such workflows is prior authorizations where we file for authorizations for drugs for cancer patients. s6 00:00:25.200 --> 00:00:31.040 So I'll give a brief overview about the workflow before we move further in the call. s7 00:00:31.040 --> 00:00:36.000 So the first step is that we intake the orders that we get on a daily basis. s8 00:00:36.000 --> 00:00:46.000 The second step is we verify whether the patient is actually eligible for getting the drugs based on the amount that they have in the insurance available with them. s9 00:00:46.000 --> 00:00:50.040 This is called eligibility and benefits verification. s10 00:00:50.040 --> 00:00:57.160 Next, we determine what all drugs in the for the patient require authorization. s11 00:00:57.160 --> 00:01:01.920 So drug can basically fall into one of the following pathways. s12 00:01:01.920 --> 00:01:04.720 So one is NAR which is no auth required. s13 00:01:04.720 --> 00:01:07.880 Basically the drug does not require authorization. s14 00:01:07.880 --> 00:01:14.040 Another can be that the authorization has already happened for the drug for a time period which is called auth on file. s15 00:01:14.040 --> 00:01:20.960 And the third is auth required which means that the authorization needs to be performed for this drug. s16 00:01:20.960 --> 00:01:22.280 So yeah. s17 00:01:22.280 --> 00:01:23.960 This is the workflow. s18 00:01:23.960 --> 00:01:34.760 I was So although our bots run and perform these steps, finally a human review is required in before submitting all these orders. s19 00:01:34.760 --> 00:01:47.360 I was tasked with the with the problem to run some of these orders without any sort of human touch directly towards submission. s20 00:01:47.360 --> 00:01:58.360 Which means I need to confidently identify which all orders can be proceeded without any human verification and then build the entire flow for them as well. s21 00:01:58.360 --> 00:02:04.040 So, confidence is a key metric that we were working towards. s22 00:02:04.120 --> 00:02:06.240 So, yeah. s23 00:02:06.240 --> 00:02:13.640 How we did this was using four agents, namely EV agent, auth agent, necessity agent, and submission agent. s24 00:02:13.640 --> 00:02:17.360 EV agent is for eligibility and business verification. s25 00:02:17.360 --> 00:02:24.080 So, basically fetching the patient data and determining whether it is uh fine for moving forward. s26 00:02:24.080 --> 00:02:31.560 Auth agent determines the key type of the status of the drug, whether authorization is required or not. s27 00:02:31.560 --> 00:02:35.880 Uh and the necessity agent is the clinical brain of system of the system. s28 00:02:35.880 --> 00:02:45.959 So, for the cases where authorization is required, we determine whether it has to be done on a whether it is right for the patient based on his vitals or not. s29 00:02:45.959 --> 00:02:49.200 And finally, we move towards submission. s30 00:02:50.480 --> 00:02:56.520 So, uh starting with the first step, how do we do this without any human in the loop? s31 00:02:56.520 --> 00:03:08.880 So, one of the problems that we have here is that insurance details, insurance documents, everything is a scattered across dozens of portals, APIs, and documents. s32 00:03:08.880 --> 00:03:13.519 And it is difficult to find that information at one place. s33 00:03:13.519 --> 00:03:24.920 So, in order to in order to start with our pipeline, we need to get information from various portals as well as APIs. s34 00:03:24.920 --> 00:03:36.360 In order to do so, we built a unified service that connects to different payer sources and gives the output in a normalized uniform format, which we can use for s35 00:03:36.360 --> 00:03:37.920 processing further. s36 00:03:37.920 --> 00:03:51.800 And we also added a deterministic decision engine to flag the cases which would not move forward, thus deterministically uh fixing some of the orders without any human in the loop. s37 00:03:51.800 --> 00:03:53.720 So, here's how it works. s38 00:03:53.720 --> 00:04:00.160 So, we have a coverage orchestrator which determines whether the patient will go for the API or the RPA path. s39 00:04:00.160 --> 00:04:02.400 RPA is basically the automation. s40 00:04:02.400 --> 00:04:19.200 Uh we perform the actions for the RPA or call the API and get the output in a fixed coverage result format which we then pass to our deterministic engine to determine whether the coverage is active or not. s41 00:04:19.200 --> 00:04:24.880 If it is, we move it further in the PHN, else we stop that right there. s42 00:04:24.880 --> 00:04:34.880 Now, one of the problems we saw here was that uh if we have to build this, then we will need to make custom integrations for different sorts of portals, s43 00:04:34.880 --> 00:04:40.040 which is not not a very scalable uh process. s44 00:04:40.040 --> 00:04:45.720 But, how we tackled it was that we used LLMs in the loop loop. s45 00:04:45.720 --> 00:04:54.919 So, first of all, we made a huge repository of custom actions as well as popular actions that are required in to build RPAs. s46 00:04:54.919 --> 00:05:06.600 Then, we built a LLM-based config generation which performs these actions and builds the config for for a portal to run on. s47 00:05:06.600 --> 00:05:12.840 Uh which uh fix which reduces our development time significantly. s48 00:05:12.840 --> 00:05:20.200 And finally, uh these automations are fragile, so it may happen that they break during the run time. s49 00:05:20.200 --> 00:05:28.000 For this, we have a self-healing loop which identifies these cases during production hours and then mitigates them. s50 00:05:28.000 --> 00:05:31.840 So, that thus preventing any sort of failures. s51 00:05:31.840 --> 00:05:34.520 So, yeah, that is the first step. s52 00:05:34.520 --> 00:05:39.360 Uh once we have done this, we move further in our chain. s53 00:05:39.360 --> 00:05:44.800 So, as we can see our diagram the graph, uh the patient data is fetched. s54 00:05:44.800 --> 00:05:47.200 We perform the eligibility verification. s55 00:05:47.200 --> 00:05:49.480 We see if it is fine or not. s56 00:05:49.480 --> 00:05:53.240 If it failed, we stop there, else we move forward. s57 00:05:53.240 --> 00:05:54.160 So, yeah. s58 00:05:54.160 --> 00:06:04.880 Moving forward, uh the next step was that data uh was to see uh what all drugs require authorization. s59 00:06:04.880 --> 00:06:19.160 So, if I have to uh remove humans from the loop, one simple case that I saw was that uh if a drug has already been authorized, that is the authorization s60 00:06:19.160 --> 00:06:25.000 is on file, or if the authorization is not required, then we can partly solve the order. s61 00:06:25.000 --> 00:06:30.560 That is that we do not need human oversight on that part of the order. s62 00:06:30.560 --> 00:06:32.960 Uh so, yeah. s63 00:06:32.960 --> 00:06:44.919 Uh so, as the first step, we built a simple LLM extraction pipeline, which takes in patient's notes, performs LLM's extraction, and categorizes the drugs into these types. s64 00:06:44.919 --> 00:06:47.840 The default being that authorization is required. s65 00:06:47.840 --> 00:06:49.640 However, uh yeah. s66 00:06:49.640 --> 00:06:55.160 So, we thought that they should would work, but we faced some issues here as well. s67 00:06:55.160 --> 00:07:07.320 So, one popular issue was that uh the notes that we were using did not have enough data, which means that it was performing some some errors. s68 00:07:07.320 --> 00:07:16.520 Uh Also, LLM uh extraction is sort of an indis- uh indeter- deterministic process. s69 00:07:16.520 --> 00:07:21.360 That means that whatever outputs we get from here cannot be blindly tested. s70 00:07:21.360 --> 00:07:24.320 So, we still need a human to review all these things. s71 00:07:24.320 --> 00:07:29.560 It might improve the efficiency, but it will not eliminate the human. s72 00:07:29.560 --> 00:07:36.000 So, in order to do so, uh we thought that what if I add more evidence to it. s73 00:07:36.000 --> 00:07:43.640 So, if the LLM is saying that a drug is not uh no auth required, then I have some information backing that. s74 00:07:43.640 --> 00:07:47.600 I can say that confidently or for the auth and file cases. s75 00:07:47.600 --> 00:07:49.200 You already uh yeah. s76 00:07:49.200 --> 00:07:54.280 So, in order to do so, we we leverage two other data sources. s77 00:07:54.280 --> 00:07:56.760 So, one was the authorization letters. s78 00:07:56.760 --> 00:08:00.120 So, this was basically the previous information. s79 00:08:00.120 --> 00:08:08.800 So, across the previous runs that a given patient has, they have authorization letters available, which shows that these drugs were actually authorized. s80 00:08:08.800 --> 00:08:21.680 So, now, instead of just having one source, I have two sources that give me the same information and wherever they conquer, I can say with confidence that this drug is already been authorized. s81 00:08:21.680 --> 00:08:25.520 Second is the NAR case, the no auth required case. s82 00:08:25.520 --> 00:08:36.919 So, here uh we found some other resources wherein from the portals or from some documents, we could find out that on a monthly basis, which are the drugs that the s83 00:08:36.919 --> 00:08:38.479 payer does not work for. s84 00:08:38.479 --> 00:08:40.960 The insurance does not work for, basically. s85 00:08:40.960 --> 00:08:54.160 So, uh we use that information to build a payer rule knowledge base, which is basically a SQL database uh which was made from portal checks as well as LLM extractions. s86 00:08:54.160 --> 00:09:02.760 So, we use all this information, we reconcile the evidence, and we only extract the auth auth statuses with a higher confidence. s87 00:09:02.760 --> 00:09:13.480 So, if from this pipeline, we get whether a drug is not authorized not does not require authorization or is already authorized, we have a solid proof behind it. s88 00:09:13.480 --> 00:09:19.320 Uh and a higher confidence uh to say that this is the case. s89 00:09:19.320 --> 00:09:22.480 And of course, all of this is configurable. s90 00:09:22.480 --> 00:09:29.120 So, uh in order if some mishap happens, we always have a plug over the cases. s91 00:09:29.120 --> 00:09:30.800 So, yeah. s92 00:09:30.800 --> 00:09:42.040 Uh another a good thing that happened from here is that we noticed that certain orders were completely eliminate We could eliminate certain orders completely from these two type of drugs. s93 00:09:42.040 --> 00:09:47.640 Because it may happen that an order does not actually require require authorization at all. s94 00:09:47.640 --> 00:09:53.800 The drugs are either are not requiring authorization or are actually authorized from before. s95 00:09:53.800 --> 00:09:59.200 So, this enabled us no touch on a certain set of orders that we had. s96 00:09:59.200 --> 00:10:00.160 Yeah. s97 00:10:00.160 --> 00:10:06.920 Uh this slide gives a big uh small overview on how we build the payroll knowledge base for the NAR cases. s98 00:10:06.920 --> 00:10:14.760 So, uh we had the documents which had this information on a time-wise scale. s99 00:10:14.760 --> 00:10:23.040 We built an LLM extraction which uses LLMs uh which uh is configurable. s100 00:10:23.040 --> 00:10:36.560 So, we can configure it on different types of documents and we can perform the extraction and we can get the drug-wise constraint that for this pair this is the this is how it will be treated for a period of time. s101 00:10:36.560 --> 00:10:40.200 Another thing is that we leverage some historical information. s102 00:10:40.200 --> 00:10:47.080 That is that we know that a certain organization uh treats this drug in a certain sort of way. s103 00:10:47.080 --> 00:10:50.440 So, we stored that information and use that as well. s104 00:10:50.440 --> 00:10:56.240 Apart from this, we also use some regular portal checks to get this information. s105 00:10:56.240 --> 00:10:57.160 So, yeah. s106 00:10:57.160 --> 00:11:02.560 Uh performing this movement moved our needle forward a bit. s107 00:11:02.560 --> 00:11:09.160 So, now we had these deterministic checks along with our initial ineligibility verification. s108 00:11:09.160 --> 00:11:22.680 So, whatever were not verifiable were flagged out before and whatever moved forward, we could determine these statuses for the drugs and we were able to eliminate some of the orders from here. s109 00:11:25.240 --> 00:11:31.120 So, now that we solved for two types of drugs, let's move forward. s110 00:11:34.040 --> 00:11:40.440 So, the next problem that we faced here was that some decisions do actually need clinical reasoning. s111 00:11:40.440 --> 00:11:52.720 So, so far the drugs that we were we shifted to no touch were the ones where the information was directly available or indirectly available via another source. s112 00:11:52.720 --> 00:11:58.760 But, we did not need to perform any sort of reasoning over here or any sort of question answer. s113 00:11:58.760 --> 00:12:14.960 However, for the drugs that are actually auth required, we need to uh check whether the patient is actually eligible for them and also give sort of supportive evidence from where we are answering this s114 00:12:14.960 --> 00:12:17.480 we are giving this information. s115 00:12:17.480 --> 00:12:23.520 So, in order to do this, we built our third agent, which is the medical necessity agent. s116 00:12:23.520 --> 00:12:32.440 So, the medical necessity agent answers simple and complex clinical questions per patient uh and attaches confidence score to any answer. s117 00:12:32.440 --> 00:12:37.720 So, we escalate only the ones that actually need a clinician. s118 00:12:37.720 --> 00:12:40.200 Uh so, we did a lot of work on this space. s119 00:12:40.200 --> 00:12:44.960 We actually have another have a publication here as well, which I've mentioned. s120 00:12:44.960 --> 00:12:51.880 So, how this agent works is that we have a new order and for every order, we read three things. s121 00:12:51.880 --> 00:12:54.240 So, we read the patient notes. s122 00:12:54.240 --> 00:12:56.120 We read the policy criteria. s123 00:12:56.120 --> 00:13:04.640 So, the policy criteria is basically the criteria that needs to be met in order for this drug to be processed further. s124 00:13:04.640 --> 00:13:14.760 Uh we from this criteria, we query the patient medical graph, which is a graph of biomarkers that are extracted for a patient. s125 00:13:14.760 --> 00:13:20.400 And we determine what all biomarkers exist and what is the condition of this patient from here. s126 00:13:20.400 --> 00:13:32.160 And then we use this thing these this information and pass it to an LLM to get an answered questioner with all the supportive and contradictory facts. s127 00:13:32.160 --> 00:13:34.440 So, yeah. s128 00:13:34.440 --> 00:13:42.640 And if we have the relevant information or we are able to determine this with a higher confidence, we move this forward. s129 00:13:42.640 --> 00:13:51.560 And for the cases we do not have this enough information, we move keep that for human escalation. s130 00:13:51.560 --> 00:13:52.680 So, yeah. s131 00:13:52.680 --> 00:13:54.160 This is the next step. s132 00:13:54.160 --> 00:14:09.720 So, after this medical necessity validation, we move from the first step where we fetch the data, we did the eligibility verification, we flagged the orders that were not eligible, s133 00:14:09.720 --> 00:14:14.880 moved the rest forward, extracted the not needed and already authorized letters. s134 00:14:14.880 --> 00:14:22.400 Uh got the got the cases where we can move that without any human intervention. s135 00:14:22.400 --> 00:14:26.200 And then we did the medical necessity validation. s136 00:14:26.200 --> 00:14:30.560 Whatever was not met was also moved to review. s137 00:14:30.560 --> 00:14:31.920 So, yeah. s138 00:14:31.920 --> 00:14:37.800 Finally, we assemble all this information and then we submit it back to the payers. s139 00:14:38.080 --> 00:14:40.960 So, this is where our submission agent comes into place. s140 00:14:40.960 --> 00:14:44.200 The submission agent is pretty much similar as the EVBV agent. s141 00:14:44.200 --> 00:14:48.839 We have customized integrations for every sort of payer that we have. s142 00:14:48.839 --> 00:14:56.560 Which is built using an LLM driven config as well as our repository of tools. s143 00:14:57.160 --> 00:14:57.760 So, yeah. s144 00:14:57.760 --> 00:15:03.240 This is the entire graph that we got through the entire journey. s145 00:15:03.240 --> 00:15:08.640 Uh So, the green part shows the cases that we are going without human touch. s146 00:15:08.640 --> 00:15:22.920 So, whenever eligibility verification passes, then enough information is found to not to prevent the drug from going for further medical necessity evaluation. s147 00:15:22.920 --> 00:15:27.040 Then this medical necessity evaluation and the submission. s148 00:15:27.040 --> 00:15:29.160 So, yeah. s149 00:15:29.440 --> 00:15:40.720 Finally, I'd like to mention that these drugs are also all although built initially for this case are being used across multiple workflows. s150 00:15:40.720 --> 00:15:52.600 We have extended the functionality to a more general one where the medical necessity agent can answer questions specific to any sort of workflow for the patient. s151 00:15:52.600 --> 00:15:56.360 Similarly for the other agents as well. s152 00:15:56.360 --> 00:15:57.800 So, yeah. s153 00:15:57.800 --> 00:16:03.720 As the conclusion, I would say the no touch is growing on the share of every order. s154 00:16:03.720 --> 00:16:07.080 So, we started with deterministic checks. s155 00:16:07.080 --> 00:16:11.720 Agents only for the rules that where what rules can't decide. s156 00:16:11.720 --> 00:16:18.920 We used multi-source of evidence to beat the single source of evidence to add confidence to our cases. s157 00:16:18.920 --> 00:16:24.040 We added itself healing in order to scale our RPS. s158 00:16:24.040 --> 00:16:32.600 And finally, we added the reasoning layer in order to deal with the cases where actually authentic authorization is required. s159 00:16:32.600 --> 00:16:33.880 So, yeah. s160 00:16:33.880 --> 00:16:37.920 Thanks for the yeah, thanks for your time. s161 00:16:37.920 --> 00:16:39.240 Yeah. s162 00:16:39.240 --> 00:16:41.240 Bye-bye.