WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=c_SPW__jF9w NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/c_SPW__jF9w.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.520 --> 00:00:14.880 Thank you for joining this station. s3 00:00:14.880 --> 00:00:18.040 It's a bit of an emotional one. s4 00:00:18.200 --> 00:00:20.520 My name is Nathaniel. s5 00:00:20.520 --> 00:00:23.160 And I'm a non-evaluator. s6 00:00:23.320 --> 00:00:30.000 It's been a lot of time since the last agent I've been shipping without any evaluation. s7 00:00:30.000 --> 00:00:38.040 And today I want to tell you the story how I quit this addiction of shipping an agent without any evaluation. s8 00:00:39.200 --> 00:00:40.640 So, I'm working at Air Canada. s9 00:00:40.640 --> 00:00:46.880 So, compared to the previous talk, it's more like focus on specific business. s10 00:00:46.880 --> 00:00:49.880 I joined Air Canada in June of this year. s11 00:00:49.880 --> 00:00:52.080 And Air Canada is a huge company. s12 00:00:52.080 --> 00:00:55.480 We work in operation, corporate, commercial, and loyalty. s13 00:00:55.480 --> 00:00:58.880 Each of those segments have their own challenges. s14 00:00:58.880 --> 00:01:04.160 So, my team is in charge of delivering AI solution for all of those. s15 00:01:04.160 --> 00:01:15.520 And the problem when we tackle task we create an agent for a task that is so specific, we don't have any public benchmark we can rely on. s16 00:01:15.520 --> 00:01:21.080 And that's the main main problem we I'm trying to address in this presentation. s17 00:01:21.320 --> 00:01:33.320 More importantly, it's not that we don't have a benchmark, is we are data scientists that can build agents or AI engineers that can build agents, but we are not domain experts. s18 00:01:33.320 --> 00:01:36.720 And those domain experts sit in different business units. s19 00:01:36.720 --> 00:01:39.080 And they are very busy in their own task. s20 00:01:39.080 --> 00:01:40.680 And we have a limited access to that. s21 00:01:40.680 --> 00:01:52.520 So, both of those things went together in a situation where it's very hard for us to build any evaluation or any any benchmark for our task. s22 00:01:52.720 --> 00:01:56.520 So, before going to Air Canada, I was working for a different company. s23 00:01:56.520 --> 00:02:05.400 It was a company that and the task was to build a a Q&A assistant for telecommunication engineer. s24 00:02:05.400 --> 00:02:07.120 So, I don't know if you're familiar with the domain. s25 00:02:07.120 --> 00:02:09.440 The telecommunication is a huge domain. s26 00:02:09.440 --> 00:02:12.600 There is about 10,000 documents. s27 00:02:12.600 --> 00:02:15.560 A lot of vendor documents on top of that. s28 00:02:15.560 --> 00:02:19.800 There's a logic to know which document you should you should find in order to answer the question. s29 00:02:19.800 --> 00:02:22.360 So, it's it's a bit of a complex task. s30 00:02:22.360 --> 00:02:25.640 And then 2024, everybody wanted to build RAGs. s31 00:02:25.640 --> 00:02:27.360 So, we started to do that. s32 00:02:27.360 --> 00:02:31.040 Yeah, on LinkedIn it's super easy. s33 00:02:31.040 --> 00:02:31.760 It's not. s34 00:02:31.760 --> 00:02:40.320 So, um The thing is when we tackle this project, the same problem as at Air Canada, we don't have access to expert. s35 00:02:40.320 --> 00:02:42.720 We have the limited access to experts. s36 00:02:42.720 --> 00:02:47.000 And we we could not actually grade the answer ourselves. s37 00:02:47.000 --> 00:02:50.680 So, that's the the problem I I want to talk today. s38 00:02:50.680 --> 00:02:54.520 Funny enough, uh we were in competition with other teams. s39 00:02:54.520 --> 00:02:56.320 Um, two external, one internal. s40 00:02:56.320 --> 00:02:59.160 It's not like in parallel, but in segments. s41 00:02:59.160 --> 00:03:02.560 And our solution actually worked. s42 00:03:02.560 --> 00:03:05.080 What mean worked is actually delivering the value. s43 00:03:05.080 --> 00:03:09.000 The people liked the solution, actually used it. s44 00:03:09.000 --> 00:03:15.120 Uh, but to be honest, I don't know why we actually made it work compared to the other the other team. s45 00:03:15.120 --> 00:03:22.680 So, today I'm just going to show what I actually did or my team actually did to make it work. s46 00:03:22.680 --> 00:03:29.560 Um, so the the all have the same document, same task, everything is the same, but our solution somehow worked. s47 00:03:29.560 --> 00:03:42.120 So, for those that are familiar with RAGs, um the principle is very is is to ask a question to the system, the system found the the relevant documents, based on this relevant document, s48 00:03:42.120 --> 00:03:43.400 formulate an answer. s49 00:03:43.400 --> 00:03:46.280 It's something that exists for a long time now. s50 00:03:46.280 --> 00:03:54.520 And for those that actually tried it, if you found the right page or the right documents, 19 90% of the time you have actually the right answer. s51 00:03:54.520 --> 00:03:59.800 So, the challenge to find the right documents, at least in this task. s52 00:04:00.640 --> 00:04:13.760 So, we created a very complex pipeline that get complexified over time if you publication if you found publication back in the time this everybody has an amazing technique to improve the the solution. s53 00:04:13.760 --> 00:04:15.120 We try them. s54 00:04:15.120 --> 00:04:22.480 It's actually didn't improve the So, on paper it worked for them for the publication but for us it didn't really help. s55 00:04:22.480 --> 00:04:27.320 And more importantly when we add complexity, we found that like we break something else. s56 00:04:27.320 --> 00:04:29.960 So, something that used to work now is not working anymore. s57 00:04:29.960 --> 00:04:32.320 And that's actually the the real problem. s58 00:04:32.320 --> 00:04:44.000 So, we can add and more we can get more ideas on into solving the different problems but we we keep continue we keep breaking things over and over. s59 00:04:44.000 --> 00:04:47.200 So, complexity doesn't always help. s60 00:04:47.200 --> 00:04:49.440 And the other problem is this one. s61 00:04:49.440 --> 00:04:58.360 Uh we can change an agent in few minutes, add a certain layer of query writing or whatever reasoning step. s62 00:04:58.360 --> 00:05:03.720 But to grade that, we had those crazy meeting once a week with experts. s63 00:05:03.720 --> 00:05:11.919 We put a giant giant screen with five different chatbot, ask the same question to all of them and ask the the expert to grade them. s64 00:05:11.919 --> 00:05:14.160 I prefer this one, I prefer this one. s65 00:05:14.160 --> 00:05:17.000 And funny enough they're not always agreeing. s66 00:05:17.000 --> 00:05:22.360 It's very hard when you when you put expert to explain the room you get three opinions. s67 00:05:22.360 --> 00:05:25.000 So, that's the the the problem we were facing. s68 00:05:25.000 --> 00:05:32.600 And so, our budget was we can try one solution a week and this was not enough to go fast. s69 00:05:33.200 --> 00:05:38.280 So, the before uh going further so, the the the the process was the following. s70 00:05:38.280 --> 00:05:40.720 We ask we ask a question the agent answer. s71 00:05:40.720 --> 00:05:42.000 We ask the expert to read. s72 00:05:42.000 --> 00:05:43.320 Sometimes they don't know the answer. s73 00:05:43.320 --> 00:05:44.040 They have to look up. s74 00:05:44.040 --> 00:05:46.760 It takes a lot of time. s75 00:05:46.880 --> 00:05:50.640 And we had the budget of one trial a week. s76 00:05:51.200 --> 00:05:53.920 Um, so there is nothing new here. s77 00:05:53.920 --> 00:05:58.720 This abiding that is well known is we reverse the problem. s78 00:05:58.720 --> 00:06:00.640 So we start with a page. s79 00:06:00.640 --> 00:06:06.680 We ask an LLM to write a question about this page, which is uh an easy task. s80 00:06:06.680 --> 00:06:17.040 We ask the agent to search for this answer and compare to what the agent found, compared to the to the page that has been used for to generate the question. s81 00:06:17.040 --> 00:06:19.400 We can actually use code to grade that. s82 00:06:19.400 --> 00:06:22.040 And no longer opinions, no longer anything. s83 00:06:22.040 --> 00:06:24.520 We can automate this task. s84 00:06:24.520 --> 00:06:28.200 And that allow us to go much much faster. s85 00:06:28.200 --> 00:06:33.400 So for those that work in rags, there is plenty of implementation of this thing. s86 00:06:33.400 --> 00:06:34.600 I didn't invented it. s87 00:06:34.600 --> 00:06:38.640 I just found it very very uh very smart. s88 00:06:38.640 --> 00:06:40.040 And it stick to me. s89 00:06:40.040 --> 00:06:45.720 Like why this principle cannot be applied to other task? s90 00:06:45.720 --> 00:06:49.840 So the impact of that is we went from one evaluation a week to dozens a day. s91 00:06:49.840 --> 00:06:55.840 And we start to see the the accuracy of the the the grading from the expert going up and up. s92 00:06:55.840 --> 00:06:58.400 And actually removing a lot of complexity help. s93 00:06:58.400 --> 00:07:00.840 Like we didn't need a lot of those. s94 00:07:00.840 --> 00:07:09.480 And focusing on the right uh embedding model, the right chunking, the right alpha in the BM25 uh is the right thing to do. s95 00:07:09.480 --> 00:07:14.269 And we we saw the we saw the the accuracy increasing over over weeks. s96 00:07:14.269 --> 00:07:15.320 [snorts] s97 00:07:15.320 --> 00:07:20.200 So I don't know why we succeeded, but I'm pretty sure the other team didn't had any test set. s98 00:07:20.200 --> 00:07:26.440 So they were using this kind of long loop with expert maybe a much more expert than we had access to. s99 00:07:26.440 --> 00:07:30.680 But we had it uh we had this test set. s100 00:07:30.880 --> 00:07:39.600 And my learning from this uh back in 2024 is if you cannot grade the agent, uh you can't make it better. s101 00:07:39.600 --> 00:07:44.760 So that stick to me until this day has been 2 years now. s102 00:07:44.760 --> 00:07:47.800 And I want to go back to Air Canada now. s103 00:07:47.800 --> 00:07:50.000 Where we have a different project. s104 00:07:50.000 --> 00:07:58.960 So in 2026 the project is about analyzing supplier contract. s105 00:07:58.960 --> 00:08:01.600 So you can imagine Air Canada has a lot of suppliers. s106 00:08:01.600 --> 00:08:03.760 We get a lot of those contract. s107 00:08:03.760 --> 00:08:06.560 There's expert looking at them one by one. s108 00:08:06.560 --> 00:08:09.200 Checking each of those clauses. s109 00:08:09.200 --> 00:08:12.120 And they have to analyze it and provide a feedback. s110 00:08:12.120 --> 00:08:13.640 Like this clause is not compliant. s111 00:08:13.640 --> 00:08:15.200 This clause needs to be rewritten. s112 00:08:15.200 --> 00:08:18.040 Like there's a lot of work to do that. s113 00:08:18.040 --> 00:08:29.760 It's a very long task and very costly because if we need the if we need some supplier to deliver right now and we don't, then it cause a lot of delay in the in the chain. s114 00:08:29.760 --> 00:08:35.520 And if we miss a clause, it's also very costly because at at the end of the day it can backfire to us. s115 00:08:35.520 --> 00:08:37.599 So it's a tricky problem. s116 00:08:37.599 --> 00:08:46.120 Um and they came to me I mean to the to the data science team to build an agent that can actually do that for them. s117 00:08:46.120 --> 00:08:49.520 So can we automatize this task? s118 00:08:49.520 --> 00:08:56.560 And obviously there's a lot of documents, a lot of rules, a lot of things to consider when you do that. s119 00:08:56.560 --> 00:08:58.120 Uh expert are super busy. s120 00:08:58.120 --> 00:09:02.040 You can imagine the price of of a lawyer by hour. s121 00:09:02.040 --> 00:09:03.920 And there's no benchmark for that. s122 00:09:03.920 --> 00:09:10.240 We cannot go on the web and found documents I found an existing benchmark we can apply. s123 00:09:10.240 --> 00:09:15.720 So go back to the same problem as I was in 2024. s124 00:09:15.960 --> 00:09:26.680 Uh cycle were super slow and we knew that we want to be there in a in a in in a in a in a shape where we can go fast and try and try and try. s125 00:09:26.680 --> 00:09:34.560 Um and I knew that if we if we learn to this situation, we can have a better trajectory in term of accuracy. s126 00:09:34.560 --> 00:09:38.080 And that What the challenge for this uh for this project? s127 00:09:38.080 --> 00:09:49.200 So, the the problem is creating the answer from documents is a well-known and defined thing in the rags, but for this specific project, there's no guy in the world that say, "Hey, you should use this technique. s128 00:09:49.200 --> 00:09:51.320 You should use this technique." s129 00:09:51.320 --> 00:09:59.560 So, we try to think how we can reapply the same mechanism from what I learned on rag to this specific project. s130 00:09:59.560 --> 00:10:01.720 And we have plenty of project at Air Canada. s131 00:10:01.720 --> 00:10:08.440 So, we're using the same concept of getting the answer first and then try the agent to find it back. s132 00:10:08.440 --> 00:10:15.600 So, going into the details, uh it's a bit simplified, but actually the um the agent has four steps. s133 00:10:15.600 --> 00:10:18.880 Uh we need to spot the clause in the contract. s134 00:10:18.880 --> 00:10:23.480 We need to find the relevant documents to support an amendment. s135 00:10:23.480 --> 00:10:26.240 Uh we need to write the the amendment. s136 00:10:26.240 --> 00:10:32.920 And then at the end, there's a classification system that tells you should we keep it or not uh this amendment. s137 00:10:32.920 --> 00:10:39.800 So, the first step, for example, spot is classifying all the section of the contract into specific domain domains. s138 00:10:39.800 --> 00:10:45.960 And then found if there's a defect or there's a change to be to be done in this section. s139 00:10:46.040 --> 00:10:48.240 So, we did have this trick from the rag. s140 00:10:48.240 --> 00:10:49.640 So, we know how to find the documents. s141 00:10:49.640 --> 00:10:51.640 So, this is covered. s142 00:10:51.640 --> 00:10:58.280 Uh but for the other other section of the um of the agent, we didn't have anything. s143 00:10:59.320 --> 00:11:05.560 So, I going to present five different techniques that we use in this project or we try to use in this project. s144 00:11:05.560 --> 00:11:09.600 That is trying to address this specific problem. s145 00:11:09.600 --> 00:11:12.040 My goal is not to show off. s146 00:11:12.040 --> 00:11:19.520 My goal is to get you inspired in how to reapply those techniques in your own uh challenges. s147 00:11:20.240 --> 00:11:22.520 So, the first one is to use past decision and answer. s148 00:11:22.520 --> 00:11:29.240 So, we we you can retrieve past contract, compare with the one that has been changed. s149 00:11:29.240 --> 00:11:35.640 The second one is to choose something you're looking for, write the contract around or generate the contract. s150 00:11:35.640 --> 00:11:41.560 The third one is uh take a good contract that has been signed and break it. s151 00:11:41.560 --> 00:11:49.760 Um the fourth one is about things that should not change no matter what you do, you should not get more flags. s152 00:11:49.760 --> 00:11:52.600 And the fifth one is actually the most powerful. s153 00:11:52.600 --> 00:11:53.320 It's very hard. s154 00:11:53.320 --> 00:11:58.880 It's to turn an example into many using the concept of templates. s155 00:12:00.200 --> 00:12:03.440 So, immediately we jump into this approach. s156 00:12:03.440 --> 00:12:09.040 Uh we ask the expert to provide us the original contract, the one that has been signed. s157 00:12:09.040 --> 00:12:17.760 We're going to take that, do a diff like you do in the code, compare what has been changed, and the agent should should provide the same exact flags. s158 00:12:17.760 --> 00:12:19.480 That would be perfect. s159 00:12:19.480 --> 00:12:25.120 However, for many reason, our our team do not keep all the contract. s160 00:12:25.120 --> 00:12:31.040 Um they don't keep the original one, they just keep the last one, they don't have all the version of the contracts. s161 00:12:31.040 --> 00:12:34.080 It sometimes goes into negotiation, there is a lot of things. s162 00:12:34.080 --> 00:12:40.000 So, we could not have any of those pairs of contracts, so this thing doesn't work. s163 00:12:42.120 --> 00:12:44.440 Okay, second one. s164 00:12:44.440 --> 00:12:45.640 We took a rule. s165 00:12:45.640 --> 00:12:48.360 And so, that's changing a bit the topic. s166 00:12:48.360 --> 00:12:56.480 Um instead of creating um flags or creating an agent that can provide flags, we start from rules. s167 00:12:56.480 --> 00:13:01.640 And the rule is, for example, we have the the cap above $1 million. s168 00:13:01.640 --> 00:13:07.839 So, that's every contract that Arcada should follow this rule or many of those in specific domains. s169 00:13:07.839 --> 00:13:19.240 So, you know the rule, you create uh a clause without breaking this rule and generate the contract around, and you should found um the rule back the the flag, s170 00:13:19.240 --> 00:13:20.920 which is breaking the rule. s171 00:13:20.920 --> 00:13:22.680 So, that's simple. s172 00:13:22.680 --> 00:13:25.520 It doesn't require a lot of uh of work. s173 00:13:25.520 --> 00:13:26.880 A a of things can be generated. s174 00:13:26.880 --> 00:13:29.160 The problem is you need to know the rules. s175 00:13:29.160 --> 00:13:39.600 So, luckily enough, we could actually get some of those from either the expert or the documentation they had, uh but cannot guarantee the coverage of everything. s176 00:13:39.600 --> 00:13:51.000 So, we came up we also with our own rules, etc. So, we generate at the end uh 275 tests that we know can run and we know the answer. s177 00:13:51.000 --> 00:13:58.520 So, similar to the rug, we inverse it we inverted the problem, we planted the problem, and we try to find it back. s178 00:13:58.880 --> 00:14:12.240 Another one which is a very hard, actually, uh we took an example of a contract that has been signed, we remove a clause, and the agent is supposed to say, "Hey, there's something missing here." s179 00:14:12.240 --> 00:14:20.200 Um it's a very hard task because some supplier has different templates, have different kind of content. s180 00:14:20.200 --> 00:14:24.160 Knowing what should be in what contract require a bit of thinking. s181 00:14:24.160 --> 00:14:29.120 Uh and we mix those with existing contract that should not show any flag. s182 00:14:29.120 --> 00:14:32.400 So, that's the most powerful approach what that we had today. s183 00:14:32.400 --> 00:14:42.200 Um using this approach, we can generate a lot of contract, and at the end of the day, we found that our agents are missing uh most of the clause that has been removed. s184 00:14:42.200 --> 00:14:45.440 So, that's a very very strong benchmark. s185 00:14:46.080 --> 00:14:47.080 Okay. s186 00:14:47.080 --> 00:14:58.320 Uh fourth technique, um you don't change you change something that should not affect the agent or should should not change anything in the result. s187 00:14:58.320 --> 00:15:05.680 For example, the name of the supplier, the address of the supplier, something that should not have any impact on the number of flags. s188 00:15:05.680 --> 00:15:13.080 Um you can reorder the clause, so some randomness should also lead to the same results. s189 00:15:13.080 --> 00:15:16.560 Uh and you can run the same uh contract again and again. s190 00:15:16.560 --> 00:15:23.840 All of those are things that we check to make sure that the the output is consistent over all of those. s191 00:15:23.840 --> 00:15:27.880 In this example two, everything can be checked using code. s192 00:15:27.880 --> 00:15:29.920 You don't need to do any LLM as a judge. s193 00:15:29.920 --> 00:15:33.280 The previous uh speaker showed like it's very hard task. s194 00:15:33.280 --> 00:15:38.880 So, we stay away from uh LLM as a judge as much as possible. s195 00:15:38.880 --> 00:15:44.920 The fifth one, um it's the concept of template. s196 00:15:44.920 --> 00:15:47.640 Our expert have different kind of templates with blank. s197 00:15:47.640 --> 00:15:50.000 So, I don't know if you ever write a contract. s198 00:15:50.000 --> 00:15:52.760 Those blanks should be filled by the expert. s199 00:15:52.760 --> 00:16:02.240 And instead of trying to fill them, we generate um an array for each of those blanks to be filled with. s200 00:16:02.240 --> 00:16:04.440 So, that's allow us to generate plenty of contracts. s201 00:16:04.440 --> 00:16:07.880 Some tuples break some rules, some tuples doesn't. s202 00:16:07.880 --> 00:16:09.320 We know that. s203 00:16:09.320 --> 00:16:14.760 And so, we know what actually should be flagged as a as a as a breach or not. s204 00:16:14.760 --> 00:16:16.200 Again, this is a code check. s205 00:16:16.200 --> 00:16:18.880 We don't need to have any LLM involved. s206 00:16:18.880 --> 00:16:27.960 Um What What The good thing is with this one is our expert already use templates, so we can reuse them to generate some contracts. s207 00:16:29.200 --> 00:16:40.720 So, the cycle was a bit slower in this case, so 4 weeks to get a a feedback, and we can get a script to run and get the the evaluation back in 5 minutes. s208 00:16:41.000 --> 00:16:48.920 So, I just want to highlight one point, and we went through all of those um kind of categories of grading at the at the bottom. s209 00:16:48.920 --> 00:16:51.480 Exact match or code that are very fast. s210 00:16:51.480 --> 00:16:57.280 As soon as we step into LLM as a judge, uh it's can be very fancy, it can be very useful. s211 00:16:57.280 --> 00:16:58.280 It's so hard. s212 00:16:58.280 --> 00:17:05.400 Like, we got so much noise from the LLM as a judge compared to the noise with the with the agent that we building. s213 00:17:05.400 --> 00:17:10.160 Solving this problem is actually almost as hard as solving the the the right problem. s214 00:17:10.160 --> 00:17:15.120 So, finding the right prompt of the right harness for the LLM as a judge is a very hard task. s215 00:17:15.120 --> 00:17:17.600 There's some padding you can do. s216 00:17:17.600 --> 00:17:18.880 task. s217 00:17:18.880 --> 00:17:20.480 You can do fact checking. s218 00:17:20.480 --> 00:17:28.360 You can do a bunch of stuff, but actually it require so much work and we could not found the right one for this project. s219 00:17:29.320 --> 00:17:30.960 There's plenty of other techniques. s220 00:17:30.960 --> 00:17:32.520 I won't go over all of them. s221 00:17:32.520 --> 00:17:33.640 For example, trajectory. s222 00:17:33.640 --> 00:17:38.000 We have another project that require the agent to eat some hand points. s223 00:17:38.000 --> 00:17:43.920 So, trajectory you can you you know what the end point should be in some order. s224 00:17:43.920 --> 00:17:45.120 You generate a story about it. s225 00:17:45.120 --> 00:17:52.160 You ask the LM to fill to use the agent and look at the trace to to see if the the trajectory is correct. s226 00:17:52.160 --> 00:17:55.000 Test to SQL, you might already know it. s227 00:17:55.000 --> 00:17:57.040 Doesn't matter what your agent is. s228 00:17:57.040 --> 00:18:04.040 If you know how to pick the answer, if you know what you're looking for, and I showed a couple of examples. s229 00:18:04.040 --> 00:18:07.440 Generate the input and then try try to find it. s230 00:18:07.440 --> 00:18:09.720 Doesn't apply to everything. s231 00:18:09.720 --> 00:18:18.720 And sometime we had to re-engineer our solution to fit this paradigm, even if we move away slightly from the original design of the solution. s232 00:18:18.720 --> 00:18:21.760 We could with the original design we could not evaluate it. s233 00:18:21.760 --> 00:18:27.040 We switched to something that we can actually evaluate because of that. s234 00:18:27.680 --> 00:18:32.760 Um one thing to highlight is not the end of the day. s235 00:18:32.760 --> 00:18:34.280 You generate the test. s236 00:18:34.280 --> 00:18:35.800 You get a better agent. s237 00:18:35.800 --> 00:18:37.240 You get some real feedback. s238 00:18:37.240 --> 00:18:42.880 So, expert can provide some feedback and we don't use this feedback to fix the agent. s239 00:18:42.880 --> 00:18:46.160 We use this feedback to generate better test. s240 00:18:46.160 --> 00:18:47.720 Better prompt. s241 00:18:47.720 --> 00:18:53.840 Better ideas how to remove clauses, etc. etc. And one loop at a time. s242 00:18:53.840 --> 00:18:56.800 Uh that can be done by Rise. s243 00:18:56.800 --> 00:18:57.960 There's a lot of tools there. s244 00:18:57.960 --> 00:18:59.713 Thanks for the invitation. s245 00:18:59.713 --> 00:19:00.120 [snorts] s246 00:19:00.120 --> 00:19:02.880 Um and that's my conclusion slide. s247 00:19:02.880 --> 00:19:04.760 Why the answer first? s248 00:19:04.760 --> 00:19:08.240 Um I think I I explained that. s249 00:19:10.120 --> 00:19:14.560 When you generate those tests, it's not enough to know if you can ship the product. s250 00:19:14.560 --> 00:19:16.120 Just a starting point. s251 00:19:16.120 --> 00:19:18.200 It's a tool to help anticipate the developer. s252 00:19:18.200 --> 00:19:23.000 It's not the go no go for the for the solution. s253 00:19:23.000 --> 00:19:27.680 So I hope you like this this presentation. s254 00:19:27.680 --> 00:19:30.720 I know that the 24 hour chip is the hardest one. s255 00:19:30.720 --> 00:19:40.160 If you want to quit releasing agent without evaluation and I would be happy to support you and have your own feedback. s256 00:19:40.160 --> 00:19:42.640 Thank you very much.