WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=mMw3arwQsL0 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/mMw3arwQsL0.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.560 --> 00:00:18.760 A big part of my research is looking into AI failures, investigating it, and then learning from it. s3 00:00:18.760 --> 00:00:21.240 And joining me is Tanmay Shah. s4 00:00:21.240 --> 00:00:22.440 Yeah, hi everyone. s5 00:00:22.440 --> 00:00:23.480 I'm Tanmay Shah. s6 00:00:23.480 --> 00:00:27.200 I'm a senior quantitative modeler and an AI researcher. s7 00:00:27.200 --> 00:00:30.600 This is my second time here, so it's great to be back. s8 00:00:30.600 --> 00:00:34.880 So, I will hand it over to Vishal to get us a started. s9 00:00:34.880 --> 00:00:37.840 So, we looked at AI incident database. s10 00:00:37.840 --> 00:00:54.960 It's a publicly available AI incident database, and we looked at the AI failures and asked one simple question, what these failures teach us about how this is how the AI system should be evaluated, s11 00:00:54.960 --> 00:01:01.080 and if we can apply those learnings to system level evaluation. s12 00:01:02.080 --> 00:01:08.280 So, the first thing that we see is that AI failures have been increasing exponentially. s13 00:01:08.280 --> 00:01:20.560 And if you look at the last four years of AI failures, particularly in the foundational model era, you see that the jump is almost vertical. s14 00:01:20.560 --> 00:01:35.120 And many of us feel that the number of AI use cases that we have in enterprises, they they are much larger than our capacity to review them each one by one. s15 00:01:35.120 --> 00:01:42.440 So, what we argue is that we need a structured way to investigate these AI failures. s16 00:01:43.360 --> 00:01:50.040 And to do that, we look at the AI failures essentially into three dimensions. s17 00:01:50.040 --> 00:01:58.520 The first dimension is the development stage, when the model and or the AI system is developed. s18 00:01:58.520 --> 00:02:06.200 So, it could be issues with respect to data quality or tool architecture or system design. s19 00:02:06.200 --> 00:02:19.760 Any of these things during the development phase can can break and once it happens, then those errors can be propagated to the next dimension, which is the deployment phase. s20 00:02:19.760 --> 00:02:33.000 Now, even if the model or the system has some faulty design issues, but if you have sufficient controls in deployment, those errors can be contained. s21 00:02:33.000 --> 00:02:45.520 And that is why it is very important to kind of understand what kind of controls do we have in deployment and whether we can contain those failures from reaching s22 00:02:45.520 --> 00:02:56.400 from deployment production platform to the final stage, which is the context in which the AI system is being used from a business standpoint. s23 00:02:56.400 --> 00:02:59.880 It could lead to reputational risk. s24 00:02:59.880 --> 00:03:08.720 It could lead to legal or financial risk and all those business context really matters for the AI system in which it is being used. s25 00:03:08.720 --> 00:03:17.200 And what we say what we propose is that the model alone or the technical strength of the model is just one part of the system. s26 00:03:17.200 --> 00:03:20.519 We have to look at the system holistically. s27 00:03:23.120 --> 00:03:37.320 So, after defining those three core dimensions, the technical strength where the system should be strong or in opposite way where the system can break, the second dimension is how do s28 00:03:37.320 --> 00:03:39.920 we ensure that the deployment controls are in place? s29 00:03:39.920 --> 00:03:44.280 And third, the business context the AI system is being used. s30 00:03:44.280 --> 00:03:51.640 We divide each of these core dimensions into several sub dimensions. s31 00:03:52.560 --> 00:03:55.600 So, in the from the technical strength point, right? s32 00:03:55.600 --> 00:04:02.080 There could be six major sub dimensions which could be affecting the technical strength of the AI system. s33 00:04:02.080 --> 00:04:08.440 It could be first related with the data quality that we use for to build the AI system. s34 00:04:08.440 --> 00:04:14.480 How the system has been designed for the appropriate task that it is supposed to accomplish. s35 00:04:14.480 --> 00:04:20.680 How is the predictive validity or outcome analysis of the out of of the AI system? s36 00:04:20.680 --> 00:04:23.880 Does that system perform as we intend? s37 00:04:23.880 --> 00:04:27.680 Um there could be fairness related um checks. s38 00:04:27.680 --> 00:04:36.560 Um failure modes essentially looks at where when, how, and for whom the uh system can break. s39 00:04:36.560 --> 00:04:40.200 Uh so, basically mapping out all the failure modes of the system. s40 00:04:40.200 --> 00:04:50.560 And finally, can we explain for uh explain the decisions for the regulators or other bodies uh so that we know how the system is arriving at its decision. s41 00:04:50.560 --> 00:04:58.040 So, these six sub dimensions are essentially part of the technical strength of the AI system. s42 00:05:00.120 --> 00:05:04.680 The next four are essentially looking at how we are using the model. s43 00:05:04.680 --> 00:05:08.919 Um what kind of authorities we have given to the AI, right? s44 00:05:08.919 --> 00:05:14.240 Is the AI for informing us or it it is out there to act on our behalf. s45 00:05:14.240 --> 00:05:18.360 What kind of decision authority are we associating with respect to AI? s46 00:05:18.360 --> 00:05:20.440 Uh second one, human mediation. s47 00:05:20.440 --> 00:05:25.160 Um how effective is human in the loop process is, right? s48 00:05:25.160 --> 00:05:26.640 That's really important. s49 00:05:26.640 --> 00:05:28.200 The third one, exposure control. s50 00:05:28.200 --> 00:05:30.240 Who's actually using the uh system? s51 00:05:30.240 --> 00:05:43.000 Is it internal employees so that they see the error and they can contain it or it is exposed to the out there in the wild external customers so that any small issue can really propagate fast. s52 00:05:43.000 --> 00:05:52.680 And finally, what kind of monitoring uh controls that we have so that we see some uh issues in production and we detect it and we can contain it. s53 00:05:54.080 --> 00:06:01.160 Finally, in the financial or the business impact phase, we see these three core sub-dimensions. s54 00:06:01.160 --> 00:06:05.120 What kind of financial impact it would make if there is an error? s55 00:06:05.120 --> 00:06:09.480 There could be legal consequences and how many users would get affected? s56 00:06:09.480 --> 00:06:13.920 That's the the last one, the scale sub-dimension. s57 00:06:15.760 --> 00:06:27.040 So, we we we look at this this as one framework which has 13 dimensions and three core questions as we explained. s58 00:06:27.520 --> 00:06:40.560 Once we define these 13 sub-dimensions and look back at AI incident retrospectively, we find that using these 13 sub-dimensions, almost 98% of AI failures can be covered. s59 00:06:40.560 --> 00:06:51.440 Meaning that AI failures are are associated with some of these dimensions, in some cases multiple of these dimensions, and in some cases few of these dimensions. s60 00:06:51.440 --> 00:06:53.880 But, these dimensions are involved. s61 00:06:53.880 --> 00:06:57.720 So, failure AI failures have structures. s62 00:06:57.720 --> 00:07:04.760 We need to map it and we need to act accordingly according to these sub-dimensions. s63 00:07:04.840 --> 00:07:12.000 Now, as I mentioned that some of these failure sub-dimensions can be present in these failures simultaneously. s64 00:07:12.000 --> 00:07:20.040 Meaning that when something goes wrong with respect to a particular failure, there may be multiple things going wrong. s65 00:07:20.040 --> 00:07:33.240 What we find is that on average, there are six failure sub-dimensions that can be present in each of these AI incident failures that we we we have investigated. s66 00:07:35.200 --> 00:07:43.960 Now, the question comes to our mind is that are all of these sub-dimensions are equal or some are more important than others. s67 00:07:43.960 --> 00:07:50.800 And the answer is that some of the sub-dimensions can be more critical than others. s68 00:07:50.800 --> 00:07:58.560 And what I'm showing you here in the in the plot on the Y-axis is the association to harm when these sub-dimensions are present. s69 00:07:58.560 --> 00:08:03.680 And on X-axis, we have the frequency of these sub-dimensions being present. s70 00:08:03.680 --> 00:08:15.720 What we see on the left upper side of the plot, sub-dimensions like human mediations or effectiveness of human in the loop, harm detection, meaning that whether we have monitoring s71 00:08:15.720 --> 00:08:17.360 control in place. s72 00:08:17.360 --> 00:08:21.280 On on X-axis, we see things like scale and system design. s73 00:08:21.280 --> 00:08:32.320 Although they are present in many cases, but they are not associated so much with harm as some of the some of some of the earlier sub-dimensions that I have mentioned. s74 00:08:32.320 --> 00:08:39.599 So, when we are really looking at the AI systems, some of these sub-dimensions can be really critical. s75 00:08:39.599 --> 00:08:54.680 So, we need to prioritize these sub-dimensions by the potential of making harm, and then allocate appropriate resources to the to the investigation of these sub-dimensions rather than looking at all the failure sub-dimensions s76 00:08:54.680 --> 00:08:56.680 equally. s77 00:08:59.440 --> 00:09:12.800 And when we club the when we group these sub-dimensions together, we see that controls that we have in deployment are more critically associated with harm, followed by the business impact, and followed by the technical failure. s78 00:09:12.800 --> 00:09:31.160 So, this plot is essentially telling us that once once we have deployment controls taken care of for the high-risk system, that gives us some level of assurance that we can take care of the failures in s79 00:09:31.160 --> 00:09:36.040 in advance in contrast to just looking at the technical failure alone. s80 00:09:36.040 --> 00:09:42.440 So, all these sub-dimensions are important, but deployment controls are far more important. s81 00:09:43.760 --> 00:09:59.160 And what in the next phase what we did was we we designed a materiality scale, meaning that if we can understand which AI use cases are more risky compared to some others, s82 00:09:59.160 --> 00:10:12.200 we can we can prioritize these AI use cases based on materiality because materiality directly rank orders with respect to the harm associated with this AI use cases. s83 00:10:12.200 --> 00:10:29.440 Meaning that we can spend suppose we know which AI cases are more more material then we can allocate more resources to much higher material AI use cases than compared to some other AI use cases which may not be as material. s84 00:10:29.640 --> 00:10:37.920 With that, I would hand over to Tanmay who would explain how we can use those learnings to build system-level evals. s85 00:10:37.920 --> 00:10:40.160 Thanks, Vishal. s86 00:10:40.160 --> 00:10:49.920 Uh so, so far as we have seen what these failure teach us, so now let us use those lessons to build system-level evaluations. s87 00:10:50.120 --> 00:11:03.000 So, first first of all, what we did is that we have decomposed risk into four components, which are how often the model fails, whether that failure reaches a real decision, s88 00:11:03.000 --> 00:11:09.880 whether it goes whether it goes uncaught, and how severe the consequences are. s89 00:11:09.880 --> 00:11:24.320 So, in this scenario, the deployment setup matters, how much authority does the system have, does someone review its output, can monitoring catch error before it causes harm? s90 00:11:24.320 --> 00:11:30.960 So, this same model can create different risk depending on how we deploy it. s91 00:11:30.960 --> 00:11:40.440 So, what we are saying is that if risk depends on the whole deployment system, our evaluation needs to cover that setup, too. s92 00:11:40.440 --> 00:11:43.600 So, that brings us to the next question. s93 00:11:43.600 --> 00:11:47.480 What should we test in an agentic system? s94 00:11:47.480 --> 00:11:52.640 So, see, in traditional machine learning, we already know how to challenge models. s95 00:11:52.640 --> 00:11:56.920 We test their assumptions, input data, and behavior under stress. s96 00:11:56.920 --> 00:12:06.480 Now, when we extend that discipline to the whole deployment system, then, like, does retrieval bring back the right information? s97 00:12:06.480 --> 00:12:08.800 Or what happens when a tool fails? s98 00:12:08.800 --> 00:12:11.160 Does human review catch mistake? s99 00:12:11.160 --> 00:12:14.680 Or does the agent stay within its permission or boundary? s100 00:12:14.680 --> 00:12:17.160 And can the system finally recover? s101 00:12:17.160 --> 00:12:19.440 So, here the logic stays the same. s102 00:12:19.440 --> 00:12:23.200 The boundary of what we test gets bigger. s103 00:12:23.200 --> 00:12:28.840 So, now the question is, how should we test an agentic system? s104 00:12:28.840 --> 00:12:32.880 So, on the right, we have the familiar traditional ML. s105 00:12:32.880 --> 00:12:41.640 We test a model on held-out data, calculate metrics like accuracy and AUC ROC, use those evidence to inform deployment. s106 00:12:41.640 --> 00:12:46.080 But, we still need to check the data and assumptions behind those metrics. s107 00:12:46.080 --> 00:12:52.320 But, on the left-hand side, we have an agentic system which adds new complexity. s108 00:12:52.320 --> 00:13:04.280 So, the deployment system, like, it has many possible inputs, tool responses, and action sequences, or trajectory, or rollout, whatever you dis- say. s109 00:13:04.280 --> 00:13:06.560 So, we test selected scenarios. s110 00:13:06.560 --> 00:13:14.800 Sometimes, another AI uh agent or AI judge generates those tests or judges the result. s111 00:13:14.800 --> 00:13:22.080 So, but that judge can make mistakes, and the test can miss important failures. s112 00:13:22.080 --> 00:13:32.960 So, this is where we need to check whether the evaluator is reliable, what the evidence covers, and whether it applies to the system we are actually deploying it. s113 00:13:32.960 --> 00:13:37.480 That's why the evaluator is the part of the system we govern. s114 00:13:37.480 --> 00:13:48.280 So, now I will hand it over to Vishal to explain how we can check whether an evaluation is reliable and what to do when it is not. s115 00:13:48.280 --> 00:13:49.560 Thank you, Tanmay. s116 00:13:49.560 --> 00:13:57.880 So, we have seen the system level evaluation for evaluator as well as you know, the whole AI system. s117 00:13:57.880 --> 00:14:12.520 The question is that how should we really And when I was looking at some of the previous research, particularly in the area of cognitive neuroscience, what I found was the um this s118 00:14:12.520 --> 00:14:19.760 the scientist Peter Wason, he's the one who coined this term confirmation bias. s119 00:14:19.760 --> 00:14:28.520 And essentially what he did was he did a bunch of experiments, but essentially uh in this case here, the series that you see 2 4 6, right? s120 00:14:28.520 --> 00:14:34.480 He was asking people to predict the next number, and more often than not, they were predicting 8 10, like even numbers. s121 00:14:34.480 --> 00:14:42.040 But those answers was those answers were incorrect because essentially the correct number was any increasing number. s122 00:14:42.040 --> 00:14:50.760 Essentially proving that all humans, like most humans, um in this case it 79% of people tested wrong. s123 00:14:50.760 --> 00:14:52.760 Um may have some sort of confirmation bias. s124 00:14:52.760 --> 00:14:58.280 And AI systems, as we know, uh are trained on human-generated data. s125 00:14:58.280 --> 00:15:10.280 So, what what he proposed was that any evaluation or any any evaluation that needs to be done has to falsify, not to confirm. s126 00:15:10.280 --> 00:15:11.839 And that that's the way to go about it. s127 00:15:11.839 --> 00:15:19.560 And we take that learning from his research and propose that eval system needs to be failure-driven. s128 00:15:19.839 --> 00:15:23.560 So, if the eval systems are failure-driven, what would that mean? s129 00:15:23.560 --> 00:15:35.320 That would mean that you would start right from looking at your system claim and testing it and trying to break it and then identify where the system claims are breaking, right? s130 00:15:35.320 --> 00:15:38.200 And then what you do for fix? s131 00:15:38.200 --> 00:15:50.840 Either you uh don't let the AI system run in you know, in the in the in the territory where uh there are failures or you come up with uh supporting systems, right? s132 00:15:50.840 --> 00:15:52.280 Which you are confident of. s133 00:15:52.280 --> 00:15:59.440 And you keep doing it iteratively uh until you get um you to the to the level of confidence, right? s134 00:15:59.440 --> 00:16:06.560 This is essentially the first principles based approach that we propose for any evaluation system. s135 00:16:06.800 --> 00:16:20.080 So, essentially you take take your AI systems, perform this build-break-fix loop, understand where the AI systems could be trusted in terms of its uh uh final outcomes, and also s136 00:16:20.080 --> 00:16:36.520 a map out uh the uncertain region and high-risk region where you can not be very sure of the system's performance, and you map out the entire failure region map and route it based on the reliability and materiality, s137 00:16:36.520 --> 00:16:51.960 meaning that particularly if you're using AI for any high-risk rated system, make sure that your AI is only being used for um the the the region that you're confident of. s138 00:16:52.920 --> 00:17:01.880 And with this, I will hand over to Tanmay who who who who would apply this approach to a real uh AI failure case. s139 00:17:03.720 --> 00:17:04.360 Okay, I sure. s140 00:17:04.360 --> 00:17:13.280 So, uh so, uh let's make this framework concrete and see how a narrow evaluation can miss failures uh that matters in the real world. s141 00:17:13.280 --> 00:17:15.319 So, I will quickly go through the story behind it. s142 00:17:15.319 --> 00:17:19.160 So, a customer needed to travel after a death in the family. s143 00:17:19.160 --> 00:17:22.720 He asked an Airlines chatbot about a discounted fare. s144 00:17:22.720 --> 00:17:28.480 The chatbot said he could buy a regular ticket and claim the discount afterwards within 90 days. s145 00:17:28.480 --> 00:17:36.520 So, he relied on that advice, but the airline later refused because its policy uh did not allow that request. s146 00:17:36.520 --> 00:17:40.800 Then finally, the tribunal held the airline responsible. s147 00:17:40.800 --> 00:17:45.160 So, through this example, we we are seeing the failure chain. s148 00:17:45.160 --> 00:17:47.280 So, it's crystal clear. s149 00:17:47.280 --> 00:17:55.160 So, incorrect advice reaches the customer, the customer acts on it, and the organization faces the consequences. s150 00:17:55.160 --> 00:18:03.880 So, if we apply our build, break, fix, and break again approach, uh that could have exposed the failure way before the deployment. s151 00:18:03.880 --> 00:18:10.840 So, we Ideally, we would start with a clear requirement like advice must match the approved policy. s152 00:18:10.840 --> 00:18:16.200 Then we would challenge the chatbot with questions about refunds after travel. s153 00:18:16.200 --> 00:18:23.400 If those tests exposed incorrect advice, then we could add a policy check or human escalation. s154 00:18:23.400 --> 00:18:30.000 Then we would test again to see whether those controls caught the errors before it reached the customer. s155 00:18:30.000 --> 00:18:32.680 So, that's our framework in practice. s156 00:18:32.680 --> 00:18:41.760 We have to define the claim, actively challenge it, and use the evidence guide changes and make a deployment decision. s157 00:18:41.760 --> 00:18:49.000 So, finally, I will hand it over to Vishal to bring the pieces together and conclude. s158 00:18:49.000 --> 00:18:51.200 Um this is our final slide. s159 00:18:51.200 --> 00:18:58.080 So, essentially, what we are saying is that your evals should be risk-based um rather than frequency-based. s160 00:18:58.080 --> 00:19:08.040 You should look at your uh entire system and then break it into the core components like technical uh failure, deployment controls, and business context. s161 00:19:08.040 --> 00:19:20.720 And then assess your AI use cases materiality with some uh scale, uh and that could be your company specific scale and risk should that risk could change. s162 00:19:20.720 --> 00:19:27.640 And then once you have that, apply the system level stress testing on the AI use case. s163 00:19:27.640 --> 00:19:33.240 And then once you are confident about the region, then only make the deployment decision. s164 00:19:33.240 --> 00:19:36.280 Otherwise, it may be too expensive. s165 00:19:36.280 --> 00:19:45.320 Our final takeaway message is evaluate the full AI system based on risk, not just one component. s166 00:19:45.320 --> 00:19:47.680 Thank you so much for your time and attention. s167 00:19:47.680 --> 00:19:51.200 You can feel free to reach out to us outside. s168 00:19:51.200 --> 00:19:54.600 We'd be happy to take any questions later. s169 00:19:55.388 --> 00:19:57.388 [applause]