WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=b_PmGocP4rc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/b_PmGocP4rc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.720 --> 00:00:13.880 So, hi. s3 00:00:13.880 --> 00:00:14.720 I'm Mayur. s4 00:00:14.720 --> 00:00:21.680 I've been with Character for a bit over 2 years and we'll talk about AI slop, right? s5 00:00:21.680 --> 00:00:27.360 I think that, you know, when we look at video generations as a whole, right? s6 00:00:27.360 --> 00:00:30.960 We have like two kind of uh parallel tracks. s7 00:00:30.960 --> 00:00:40.360 One is the the video generation, which became insanely good from from models like Kling and SeaDance and VEO and Sora. s8 00:00:40.360 --> 00:00:43.080 We still remember Sora. s9 00:00:43.080 --> 00:00:51.240 But but but the part that got left behind is how we evaluate the quality of the video that was generated, right? s10 00:00:51.240 --> 00:01:01.840 So, on on the one hand, um we we we still kind of squint at it and decide whether or not it's it's good, but on the other hand, we know that the generation has gotten a lot better. s11 00:01:01.840 --> 00:01:13.720 And when when we when we we look at at X or whatever social you're you're you're consuming your content on, there are a lot of guides on how to create amazing videos with s12 00:01:13.720 --> 00:01:16.400 um this model or or that. s13 00:01:16.400 --> 00:01:19.720 So, the hard part was never how to make video. s14 00:01:19.720 --> 00:01:28.440 The hard part was how do we generate um good enough video and how do we judge if the video is good enough? s15 00:01:28.440 --> 00:01:34.320 So, so now we've gotten to a world where the the generation of video is basically free, right? s16 00:01:34.320 --> 00:01:38.480 Free as especially when you compare it to how much studios would charge. s17 00:01:38.480 --> 00:01:45.320 Um and but but the problem is the most the grand majority of videos that is generated is not that good, right? s18 00:01:45.320 --> 00:01:57.880 We have like a lot of hallucinations like a third limb, uh opening and closing the the door at the same time, hovering, physics, um etc. So, unfortunately, in order to get high high high-quality s19 00:01:57.880 --> 00:02:00.160 content, we need a human to judge. s20 00:02:00.160 --> 00:02:08.920 And I I know when was the last time you've seen how someone is creating these long-form um generated video. s21 00:02:08.920 --> 00:02:13.360 It's usually a lot of shorter generations and a lot of editing. s22 00:02:13.360 --> 00:02:22.000 The problem is because we're using a lot of the tools that we built for the ticks for the text era, for the image era, for videos, right? s23 00:02:22.000 --> 00:02:28.080 We're using things like clip score, which is is great to to to judge a single frame. s24 00:02:28.080 --> 00:02:44.640 Things like LP IPS will help us kind of detect the drift between frames, but we don't have I mean but the problem is when you kind of combine all these together, all these tools are good at watching the individual frames. s25 00:02:44.640 --> 00:02:54.080 They're good at checking this this this this does this specific frame does it match the the prompt that generated it, right? s26 00:02:54.080 --> 00:02:59.600 It will check consistency between frames and it will check whether or not it match the prompt that the drove it. s27 00:02:59.600 --> 00:03:07.680 But but what it won't do it doesn't tell you if if it's if you told if you told the story that you you meant to tell, right? s28 00:03:07.680 --> 00:03:11.640 If you think about what is video, video is a storytelling medium. s29 00:03:11.640 --> 00:03:14.720 Video is just another form on how we tell a story, right? s30 00:03:14.720 --> 00:03:17.280 From for any type of story. s31 00:03:17.280 --> 00:03:21.160 So so one of one of the things we have to look at, does it tell the actual story? s32 00:03:21.160 --> 00:03:22.519 Does the physics make sense? s33 00:03:22.519 --> 00:03:29.640 Like for example, if you want a video of a character walking downstairs, does it actually walk or or hover? s34 00:03:29.800 --> 00:03:35.239 Does the character stay the same character across multiple shots? s35 00:03:35.239 --> 00:03:36.680 Does the pacing make sense? s36 00:03:36.680 --> 00:03:41.120 Like you know, for example, people take time going from one place to another. s37 00:03:41.120 --> 00:03:44.000 We need to make sure that the pacing makes sense as well. s38 00:03:44.000 --> 00:03:48.880 And especially when we add audio, we want to make sure that the audio is kind of synced with the imagery. s39 00:03:48.880 --> 00:03:59.280 Like for example, if someone is slamming a door, we want that that that that that that sound of the door being slammed to be exactly when the door is actually being slammed. s40 00:03:59.280 --> 00:04:03.200 Now, the the next iteration we all went to a while ago. s41 00:04:03.200 --> 00:04:11.680 We started using LLM as a judge for everything and we have amazing foundational models that we just throw videos at them. s42 00:04:11.680 --> 00:04:14.040 The problem with them is that A, they're slow. s43 00:04:14.040 --> 00:04:22.360 B, they're only as good as your prompt and multiple people will prompt multiple ways and the same model may respond in a very very different way. s44 00:04:22.360 --> 00:04:26.280 And sometimes the prompt we use like is it consistent? s45 00:04:26.280 --> 00:04:28.440 Does this match the the prompt? s46 00:04:28.440 --> 00:04:31.680 But but then the question we really care about is it good? s47 00:04:31.680 --> 00:04:34.760 And the answer varies. s48 00:04:34.760 --> 00:04:36.919 So, oops, sorry about that. s49 00:04:36.919 --> 00:04:47.120 So, our first iteration is like let's take all these things and build a repeatable benchmark on how we test video that we can rerun over and over and over again. s50 00:04:47.120 --> 00:04:56.760 So, so that combines both metrics as I said earlier that that knows how to view individual frames, but also consistent LLM as a judge, right? s51 00:04:56.760 --> 00:05:01.800 Where we also use human annotation to to calibrate the the LLM as a judge. s52 00:05:01.800 --> 00:05:19.360 So, for every report that we generate with that harness, we're able to have humans annotate it and basically feed that feedback back in into the the the LLM as a judge prompt to make sure that it's it's aligned with what I think or what the annotator thought is good. s53 00:05:19.360 --> 00:05:22.080 And and we use it to to score the videos. s54 00:05:22.080 --> 00:05:35.560 The problem with this approach, it's very slow, it's very expensive and especially when we we want to bring it in for our users to be able to generate a lot of video because creation is a very hard process. s55 00:05:35.560 --> 00:05:46.160 And so the problem as I said, the problem is when when So, this is a slow process and we need to bring it as close to the users as possible and also earlier into the process. s56 00:05:46.160 --> 00:06:00.800 The reason for that is if we take a look at all the metrics and there's there mistakes that we can find earlier than than later than it's a lot cheaper to correct that that particular s57 00:06:00.800 --> 00:06:01.240 mistake. s58 00:06:01.240 --> 00:06:03.520 So, for example, right? s59 00:06:03.520 --> 00:06:12.800 On on the left we have two starting frames of different different shots, right? s60 00:06:12.800 --> 00:06:23.520 But it's easy to to correct to to view it at this point and see did the character drift between frame one and and frame two because those frames will be used as starting frames to generate videos. s61 00:06:23.520 --> 00:06:33.280 So, if you can correct catch the drift at this point and correct it, then then it's much cheaper to generate the video as a whole because we we can correct it at a much s62 00:06:33.280 --> 00:06:34.360 cheaper cost. s63 00:06:34.360 --> 00:06:39.040 And the same thing applies when we look at longer form video, right? s64 00:06:39.040 --> 00:06:45.120 When we see all these three, four, five-minute long videos, they're usually a collection of a lot of shorter videos. s65 00:06:45.120 --> 00:06:57.800 And being able to catch a six-second generation that drifted and and regenerated that regenerate that before we combine the whole video will end up being a better result as a whole. s66 00:06:58.680 --> 00:07:00.514 Um s67 00:07:00.514 --> 00:07:01.080 [clears throat] s68 00:07:01.080 --> 00:07:06.520 And now the the other problem what we're trying to solve is some of these axes, right? s69 00:07:06.520 --> 00:07:09.360 Only exist across time, right? s70 00:07:09.360 --> 00:07:13.880 So, for example, when we look at the right? s71 00:07:13.880 --> 00:07:16.880 We we mentioned the story, right? s72 00:07:16.880 --> 00:07:21.880 So, does does the story that we're trying to tell with that video, does it hold in that video? s73 00:07:21.880 --> 00:07:23.720 Does the video tell the exact story? s74 00:07:23.720 --> 00:07:25.720 Does the pacing make sense, right? s75 00:07:25.720 --> 00:07:28.680 And we mentioned the the the the sound. s76 00:07:28.680 --> 00:07:30.480 So, as I said, right? s77 00:07:30.480 --> 00:07:41.600 The the you know, the underlying goal is to bring that evals closer to to the to the the online generation because the sooner we're able to catch those mistakes, we were to s78 00:07:41.600 --> 00:07:43.520 the sooner we're able to catch that drift, right? s79 00:07:43.520 --> 00:07:49.120 Then it's um it's much easier, much much cheaper to fix. s80 00:07:49.120 --> 00:07:56.520 Now, so so that now now the problem is that as I said, this is a very slow process. s81 00:07:56.520 --> 00:08:10.080 So the the solution is actually actually to take all these committee of experts and distill it into one small model that is also very very fast, but it is able to give us s82 00:08:10.080 --> 00:08:15.160 a response that is not whether or not this video is slap or not, but why is it slap? s83 00:08:15.160 --> 00:08:15.680 Right? s84 00:08:15.680 --> 00:08:19.000 Why Why is that video scored low versus the other? s85 00:08:19.000 --> 00:08:27.960 Because for example, it added an extra limb, because it didn't obey physics, because the audio was was out of sync. s86 00:08:27.960 --> 00:08:33.280 So so the the goal was a build it on top of a small VLM and why is it a VLM? s87 00:08:33.280 --> 00:08:40.760 VLM because we needed the video the model to be able to see the image, but also we needed to work fast, right? s88 00:08:40.760 --> 00:08:54.680 Because we bought it closer to the generation, where in fact it takes about the model we have trained, it takes about 3 seconds to score a three 15 second video. s89 00:08:55.280 --> 00:09:03.760 Now, the we we also tested a bigger model and the results were better, but it was significantly slower. s90 00:09:03.760 --> 00:09:16.960 And and the decision was to go with the the the smaller model because the the added value from the bigger model didn't justify the the the the slowness. s91 00:09:16.960 --> 00:09:22.800 The other very interesting realization we came to is don't score compare. s92 00:09:22.800 --> 00:09:23.520 What does that mean? s93 00:09:23.520 --> 00:09:32.720 For example, if I'll ask any person in this room to look at a particular video and and rank it from 1 to 10 on storytelling, right? s94 00:09:32.720 --> 00:09:38.440 I'm pretty sure that you know, what will be a six for you will be a five for you, will be a four for you and and and an and an eight for you. s95 00:09:38.440 --> 00:09:38.600 Right? s96 00:09:38.600 --> 00:09:47.440 But if I show you two videos and I'll ask you which one of them is telling a better story, the grand majority will probably agree that B is telling a better story than A, right? s97 00:09:47.440 --> 00:10:00.880 And if you do it enough times, then it it uh it's easy to um uh generalize the model at um towards detecting what's better uh uh versus uh not. s98 00:10:00.880 --> 00:10:01.951 So, s99 00:10:01.951 --> 00:10:02.280 [clears throat] s100 00:10:02.280 --> 00:10:04.000 we trained on pairs, right? s101 00:10:04.000 --> 00:10:08.040 Um uh A versus B as opposed to 1 through 10. s102 00:10:08.040 --> 00:10:10.520 Now, we manufactured badness. s103 00:10:10.520 --> 00:10:23.240 So, luckily, the internet is full of very high-quality videos, and it's very very easy to get good videos, and it was very fun to create bad videos A by either corrupting s104 00:10:23.240 --> 00:10:29.440 good videos or by, you know, just generating random slop. s105 00:10:29.462 --> 00:10:29.560 [clears throat] s106 00:10:29.560 --> 00:10:35.360 Um Now, we shipped V1, and it was so wrong. s107 00:10:35.360 --> 00:10:38.320 It It was wrong, but it was wrong in a very confident way. s108 00:10:38.320 --> 00:10:48.400 So, uh uh for for for example, the the frame you see here is from from a video that the model scored 9.2 on the camera work, and the camera didn't move. s109 00:10:48.400 --> 00:10:59.240 For for 4 seconds, it was like a still image of the same character, but but but the model was was very very happy with with, you know, the cinematography. s110 00:10:59.320 --> 00:11:15.839 Um so, the physics in in in in some other videos, which I'm not showing the cuz of uh time limitations, it says that the physics look great, but it said it on on ghosts hovering and people flying, etc. s111 00:11:15.839 --> 00:11:20.360 So, the I mean, so the so then the question is like, why was it wrong? s112 00:11:20.360 --> 00:11:25.400 The The reason it was wrong is because how we generated that data, right? s113 00:11:25.400 --> 00:11:29.760 It It um it scored the vibe as opposed to the the the axes. s114 00:11:29.760 --> 00:11:38.800 So, it it learned how to how to detect um cohe- coherent videos and it learned how to detect the the the the the the artificial artifacts. s115 00:11:38.800 --> 00:11:48.000 Basically, the gloss of the video as opposed to uh whether or not the video actually um told the the the the the Sorry. s116 00:11:48.000 --> 00:11:51.280 The videos actually told the the the story. s117 00:11:51.280 --> 00:11:59.360 And um And so so so the solution was to fix the data set. s118 00:11:59.360 --> 00:12:10.240 And so the way we we fixed the the the the the the data set, we actually I uh um I started pairing real footage versus um AI footage. s119 00:12:10.240 --> 00:12:18.120 Now, the risk with that and that that's the reason why I voted I voted I avoided doing it at first is because I didn't didn't want to create an AI detector, s120 00:12:18.120 --> 00:12:18.360 right? s121 00:12:18.360 --> 00:12:29.480 Because if you start creating pairs of good is is is human-generated video and bad is AI uh uh video, then then there's a very big chance of of the model overfitting s122 00:12:29.480 --> 00:12:34.520 and becoming an AI detector as opposed to a um uh uh video quality detector. s123 00:12:34.520 --> 00:12:38.080 So, the There are two things I did in order to avoid that. s124 00:12:38.080 --> 00:12:45.000 A, I made sure that the encoding is is consistent across both sides of of the equation. s125 00:12:45.000 --> 00:12:58.600 So, uh there there there there's no um artificial artifacts for for video A versus video B. Uh and I I used the exact same method of annotating both videos. s126 00:12:58.600 --> 00:13:04.320 So, so both the axes So, all the axes in those videos were annotated in the same way. s127 00:13:04.320 --> 00:13:09.120 Uh and surprise, it it turned out pretty awesome. s128 00:13:09.120 --> 00:13:21.839 Um and so so so now what what what we're able to do, especially when we you looking at at at videos, A, we we changed from a very complex pipeline, s129 00:13:21.839 --> 00:13:22.360 right? s130 00:13:22.360 --> 00:13:26.000 To an agentic workflow. s131 00:13:26.000 --> 00:13:31.600 The reason behind this is, a, the pipelines work great if you have a very very unique use case. s132 00:13:31.600 --> 00:13:39.160 But one once you put put put it in front of users, they'll have a very very distinct story that they want to tell with their own characters, with with their own s133 00:13:39.160 --> 00:13:40.480 images, and their own voice. s134 00:13:40.480 --> 00:13:42.880 So that's where it starts to drift. s135 00:13:42.880 --> 00:13:51.280 But by providing the agents with tools to validate the quality of of the outputs it's creating, it's able to adapt to changes better. s136 00:13:51.280 --> 00:13:57.160 It's also able to verify its own work and and and fix things as they go along. s137 00:13:57.160 --> 00:14:05.400 So if you're going to steal from from these from the stock of a few things, one, go relative, not absolute, right? s138 00:14:05.400 --> 00:14:16.440 As I explained earlier, the the value of comparing video A versus video B will always give you a better result going forward. s139 00:14:16.440 --> 00:14:19.280 Be score the real axis that you care about. s140 00:14:19.280 --> 00:14:27.960 So if you care about storytelling, if you care about pacing, if you care about physics, score those axes. s141 00:14:27.960 --> 00:14:31.280 Don't expect them to miraculously appear. s142 00:14:31.280 --> 00:14:36.360 And put eval inside the generation loop, right? s143 00:14:36.360 --> 00:14:47.200 Especially if your goal is is is is to have a higher quality of generation, get the the eval as close to to the generation loop loop as possible. s144 00:14:47.200 --> 00:14:49.760 Eventually evaluate it as a story. s145 00:14:49.760 --> 00:14:51.040 Videos are stories. s146 00:14:51.040 --> 00:14:55.200 Videos are just another way for us to tell stories to others. s147 00:14:55.200 --> 00:14:59.880 And Thank thank you very much. s148 00:14:59.980 --> 00:15:01.980 [applause] s149 00:15:05.720 --> 00:15:06.280 All right. s150 00:15:06.280 --> 00:15:06.920 Any questions? s151 00:15:06.920 --> 00:15:08.000 Okay, down here. s152 00:15:08.000 --> 00:15:09.640 Awesome. s153 00:15:09.640 --> 00:15:11.040 All right, I got two down here. s154 00:15:11.040 --> 00:15:13.240 Here you go. s155 00:15:16.880 --> 00:15:17.320 Hi. s156 00:15:17.320 --> 00:15:18.920 How do you eval sound? s157 00:15:18.920 --> 00:15:21.120 Sound and video matching. s158 00:15:21.120 --> 00:15:22.760 I'm I'm I'm Can you you repeat? s159 00:15:22.760 --> 00:15:24.400 How do you evaluate sound? s160 00:15:24.400 --> 00:15:26.800 Sound effects and matching with the video. s161 00:15:26.800 --> 00:15:28.120 oh yeah. s162 00:15:28.120 --> 00:15:29.800 That's a fantastic question. s163 00:15:29.800 --> 00:15:34.240 So, um so so sound is actually a a combination of few things. s164 00:15:34.240 --> 00:15:47.320 One, I'm using Atmos to to to make sure that that that the the the sound quality is high enough and is understandable. s165 00:15:47.320 --> 00:15:53.520 B, the model will will will will learn to learn to identify key frames. s166 00:15:53.520 --> 00:15:53.920 Right? s167 00:15:53.920 --> 00:16:02.360 And and and especially because when I feed something into the model, it's it can be just a video or it can be the video plus the prompt that generated s168 00:16:02.360 --> 00:16:03.240 that video. s169 00:16:03.240 --> 00:16:07.160 So, for example, if the prompt will say the door slammed, right? s170 00:16:07.160 --> 00:16:16.880 It will look for a door being slammed and and and will will will match the sound at that same frame. s171 00:16:18.320 --> 00:16:20.920 I Did I answer your question? s172 00:16:20.920 --> 00:16:22.720 How does the model recognize sound? s173 00:16:22.720 --> 00:16:35.040 Uh so, it's both by using um um uh Atmos, but and also to to correlate the So, for example, when it's looking at the frames, right? s174 00:16:35.040 --> 00:16:43.280 It's making sure that that for example, the door being slammed at frame six, frame six has a a specific timestamp. s175 00:16:43.280 --> 00:16:50.280 So, it's looking for that that spike in the sound at that timestamp. s176 00:16:50.280 --> 00:16:57.040 It doesn't know that it is that sound, but it's looking for a specific spike of sound at at that timestamp. s177 00:16:57.040 --> 00:16:58.720 What about lip syncing? s178 00:16:58.720 --> 00:17:02.466 Lip syncing is an unsolved problem yet. s179 00:17:02.466 --> 00:17:03.920 [laughter] s180 00:17:03.920 --> 00:17:04.520 One question. s181 00:17:04.520 --> 00:17:06.480 We're trying though. s182 00:17:06.480 --> 00:17:08.480 Yeah. s183 00:17:09.839 --> 00:17:14.880 Yeah, I said the the question was what about lip syncing? s184 00:17:14.880 --> 00:17:21.079 Oh, Well, wasn't me, but I guess the lip syncing answer would be interesting before I ask my question. s185 00:17:21.079 --> 00:17:25.560 Uh yeah, um as I said, it is an unsolved problem still. s186 00:17:25.560 --> 00:17:29.160 We're still working through it. s187 00:17:29.160 --> 00:17:35.240 Especially for us, some some of the characters that we're trying to do a talking head are humans, right? s188 00:17:35.240 --> 00:17:50.000 Which, you know, we can look at the different techniques to trying to identify the the lips, but some of them are just talking, you know, talking animations that have no real correlation between, you know, the the movement of of of the mouth and and and speech. s189 00:17:50.000 --> 00:17:54.080 So, unfortunately, I don't have a solution for that yet. s190 00:17:54.080 --> 00:18:02.240 So, I'm So, I'm curious about, for example, if you wanted to further enrich the data set with human evaluation. s191 00:18:02.240 --> 00:18:02.640 Yes. s192 00:18:02.640 --> 00:18:13.120 Um the question of of taste in what is good, because I think there is a big question mark about is that is that going to remain the domain of humans? s193 00:18:13.120 --> 00:18:20.280 But, I've also seen people say that, well, most humans they have terrible taste anyway in in videos and games and books. s194 00:18:20.280 --> 00:18:20.640 Fair. s195 00:18:20.640 --> 00:18:28.000 Um so, how would you construct and align sort of like any human judges? s196 00:18:28.000 --> 00:18:28.520 Yeah. s197 00:18:28.520 --> 00:18:39.880 So, So, So, So, this is actually solved at first at the the the Judge Judy part, where every report it will generate a human can go and annotate it. s198 00:18:39.880 --> 00:18:42.320 And we actually we we do that. s199 00:18:42.320 --> 00:18:48.640 We we will periodically have sessions where everyone spends 10 to 15 minutes just annotating videos. s200 00:18:48.640 --> 00:18:53.000 And and that usually happens on multiple axes. s201 00:18:53.000 --> 00:18:56.480 I won't ask everyone to annotate the same video on on 10 different things. s202 00:18:56.480 --> 00:18:57.920 It'll It'll It'll be random. s203 00:18:57.920 --> 00:19:03.760 And I use the data to to calibrate the the AI judges. s204 00:19:03.760 --> 00:19:10.560 And and the results from that is actually being served as as a data set for for training for the next version of of that model. s205 00:19:10.560 --> 00:19:16.400 So, it's a process that does take a little bit of time and and hopefully and and it does evolve over time. s206 00:19:16.400 --> 00:19:28.880 Uh but it's not immediate you know, because also taste is very subjective and and things that are great for me, you know, some that I think are fantastics some people that come and say s207 00:19:28.880 --> 00:19:39.520 are you sure they're great because you know So, yeah, it's it's it's a process and and and and I use the human feedback to calibrate the models all the time. s208 00:19:39.520 --> 00:19:42.600 Um how did you land on the Quan small VLM? s209 00:19:42.600 --> 00:19:44.000 Did you try any others? s210 00:19:44.000 --> 00:19:45.640 Uh I did. s211 00:19:45.640 --> 00:19:51.080 So, so the intent I had was to A you know, find a small enough model. s212 00:19:51.080 --> 00:19:59.200 The reason I went with Quan is because we also had a very good experience with post training Quan on other use cases. s213 00:19:59.200 --> 00:20:08.600 So, it I mean yes, I could have I did try a few others, but it just you know, everything was just there and it was good enough. s214 00:20:09.240 --> 00:20:10.800 So, my question is about scale. s215 00:20:10.800 --> 00:20:15.880 So, obviously Character AI produces thousands, millions, bajillion videos. s216 00:20:15.880 --> 00:20:19.440 What scale does this become reasonable for my domain that is not Character AI? s217 00:20:19.440 --> 00:20:22.000 So, my domain has hundreds, maybe a thousand videos. s218 00:20:22.000 --> 00:20:23.000 Mhm. s219 00:20:23.000 --> 00:20:24.080 Sure. s220 00:20:24.080 --> 00:20:34.480 So, if you're happy with with the cohort of experts and and and and it and you don't need right so I'll rephrase that. s221 00:20:34.480 --> 00:20:37.320 The scale is both for speed, right? s222 00:20:37.320 --> 00:20:45.640 As well as capacity because I can serve this model as one instance on one GPU or I can serve it as you know, 100 instances, right? s223 00:20:45.640 --> 00:20:48.400 So, so that's that determines my scale. s224 00:20:48.400 --> 00:20:56.160 The reason I chose to go towards the model is because I wanted to to speed up the creation process, right? s225 00:20:56.160 --> 00:21:03.000 It would work would have worked just as well if I didn't have this particular model, I would have used like the cohort of experts, right? s226 00:21:03.000 --> 00:21:12.000 From metrics that are available both on CPU and GPU as well as the frontier models. s227 00:21:12.000 --> 00:21:12.240 Right? s228 00:21:12.240 --> 00:21:23.440 So it's it was a balance as you know, A, how long did it take me to to to train this model and to curate the the data set and and get it to a working set, right? s229 00:21:23.440 --> 00:21:29.720 And and how much does it cost to serve it versus how much it would have cost me to do this A slower. s230 00:21:29.720 --> 00:21:32.240 Now potentially it is better, right? s231 00:21:32.240 --> 00:21:36.520 I mean like I assume that if you're going to use Fable which came back today, right? s232 00:21:36.520 --> 00:21:41.120 It will probably give you a a better result, but at what cost, right? s233 00:21:41.120 --> 00:21:42.680 If you do it for one or two, that's probably fine. s234 00:21:42.680 --> 00:21:48.720 If if you do it for thousands or tens of thousands per day, it adds up. s235 00:21:48.720 --> 00:21:54.360 So it's it's it's a matter of your your your unit economics. s236 00:21:54.600 --> 00:21:55.160 Cool. s237 00:21:55.160 --> 00:21:56.880 Over here. s238 00:21:56.880 --> 00:22:00.000 On your right, there you go. s239 00:22:00.000 --> 00:22:00.800 Last question. s240 00:22:00.800 --> 00:22:01.600 It's very bright. s241 00:22:01.600 --> 00:22:02.200 I'm sorry. s242 00:22:02.200 --> 00:22:02.640 No worries. s243 00:22:02.640 --> 00:22:03.560 No worries. s244 00:22:03.560 --> 00:22:07.160 Um My question is I looked a bit at the repo. s245 00:22:07.160 --> 00:22:11.680 You guys don't export all tail traces of the LMS judges yet. s246 00:22:11.680 --> 00:22:12.200 Correct. s247 00:22:12.200 --> 00:22:17.480 Is that something are you open to that so you can connect to other platforms? s248 00:22:17.480 --> 00:22:17.720 Sure. s249 00:22:17.720 --> 00:22:29.360 So so the the the repo itself it's a harness and you can connect any any agents or any LLMs you want. s250 00:22:29.360 --> 00:22:33.960 We actually have an internal version of this which is running it as a service, right? s251 00:22:33.960 --> 00:22:44.960 We we have an agentic harness on top of it that has all the the the the metrics we we care about, but I I I I do accept your feature request and I'll be adding s252 00:22:44.960 --> 00:22:49.120 hotel telemetry to the the harness. s253 00:22:49.120 --> 00:22:49.560 Awesome. s254 00:22:49.560 --> 00:22:50.840 Thank you very much. s255 00:22:50.840 --> 00:22:53.840 A warm welcome or round of applause for Mayor. s256 00:22:53.840 --> 00:22:54.520 Thank you. s257 00:22:54.520 --> 00:22:55.400 Thank you all. s258 00:22:55.400 --> 00:22:57.400 Thanks. s259 00:23:12.360 --> 00:23:14.360 Oh.