WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=O3FEoMYvUf8 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/O3FEoMYvUf8.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:02.680 --> 00:00:03.600 Hi everyone. s2 00:00:03.600 --> 00:00:06.200 I'm Alejandro Vidal, the founder of Mind Makers. s3 00:00:06.200 --> 00:00:16.520 And my background is psychology and computer science, which is kind of weird, but for today it's going to be extremely helpful because I'm going to show you how you can borrow ideas from psychology s4 00:00:16.520 --> 00:00:21.800 and psychometrics to improve the way that you are evaluating models right now. s5 00:00:21.800 --> 00:00:27.520 Because at this moment the state in the industry is counting the number of right answers. s6 00:00:27.520 --> 00:00:28.600 That actually has a name. s7 00:00:28.600 --> 00:00:30.480 It's classical test theory. s8 00:00:30.480 --> 00:00:33.680 And we have by far better tools to do that. s9 00:00:33.680 --> 00:00:38.280 So makes sense to borrow ideas from IQ tests and related stuff. s10 00:00:38.280 --> 00:00:40.920 So we can apply them to LLMs. s11 00:00:40.920 --> 00:00:44.000 Let me start with a very simple example here. s12 00:00:44.000 --> 00:00:47.720 Um we are using real data from epoch.ai. s13 00:00:47.720 --> 00:00:54.760 If you don't know them, their project is amazing and they have quite open data sets so you can actually use them. s14 00:00:54.760 --> 00:01:00.280 And here we have random selection of models with a real benchmark. s15 00:01:00.280 --> 00:01:03.480 As you can see here, we have an accuracy for each one of them. s16 00:01:03.480 --> 00:01:05.600 That's the current state of the art. s17 00:01:05.600 --> 00:01:14.360 So if we split each bar into different questions, each one of them is going to be a different question or a different item. s18 00:01:14.360 --> 00:01:19.440 I'm going to use item for um uh for for the same idea of question. s19 00:01:19.440 --> 00:01:22.400 In psychometrics we use item instead of question. s20 00:01:22.400 --> 00:01:27.280 Um if you sum all together, we are using a very strong assumption. s21 00:01:27.280 --> 00:01:30.280 We are saying that every question is equally important. s22 00:01:30.280 --> 00:01:34.600 They should weigh the same, which is kind of insane if you think about that. s23 00:01:34.600 --> 00:01:42.440 We have better questions, more complicated questions that may maybe we should pay more attention to. s24 00:01:42.440 --> 00:01:46.560 And also we we can have questions that are mislabeled or something like that. s25 00:01:46.560 --> 00:01:48.840 So we are going to improve this. s26 00:01:48.840 --> 00:01:54.440 Well, uh what we're going to do is we are going to use each item, each column here is going to be one item. s27 00:01:54.440 --> 00:01:58.320 And we're going to treat them as individual variables, okay? s28 00:01:58.320 --> 00:02:00.320 So, we are going to have this matrix here. s29 00:02:00.320 --> 00:02:12.640 As you can see here, uh on the top right corner, we have difficult questions for weaker models, and on the other side, we have very easy questions for strong models. s30 00:02:12.640 --> 00:02:16.520 So, makes sense that we observe this pattern, okay? s31 00:02:16.520 --> 00:02:21.760 But, we are going to estimate for each question, for each item, a difficulty level. s32 00:02:21.760 --> 00:02:32.040 That is going to be called B. The B parameter is going to be the difficulty of each one of them, and we're going to create a function for each question. s33 00:02:32.040 --> 00:02:35.959 That function maps the LLM intelligence, okay? s34 00:02:35.959 --> 00:02:40.120 To the probability of getting that answer right, okay? s35 00:02:40.120 --> 00:02:46.840 So, very easy items are going to be here, and extremely complicated items are going to be there. s36 00:02:46.840 --> 00:02:54.000 As you can see here, B is the point that uh crosses 50% chance in that curve, okay? s37 00:02:54.000 --> 00:02:56.080 Which is going to be useful later. s38 00:02:56.080 --> 00:03:04.640 Also, B is going to be distributed by a normal distribution, which is going to be also helpful to use that for interpretation, okay? s39 00:03:04.640 --> 00:03:09.800 So, with that in mind, we can actually estimate also theta. s40 00:03:09.800 --> 00:03:13.120 Theta is going to be the level of intelligence for each model. s41 00:03:13.120 --> 00:03:15.800 That's going to be that dot, that black dot. s42 00:03:15.800 --> 00:03:27.680 So, as you can see on the right side of each dot, mostly uh all of questions are going to be read, which makes sense if they they are extremely complicated or more complicated s43 00:03:27.680 --> 00:03:32.680 than the level of intelligence of that model, the model is going to fail them, okay? s44 00:03:32.680 --> 00:03:34.480 So, we are going to model that way. s45 00:03:34.480 --> 00:03:46.959 So, for example here, if I click on this button, I'm going to see that uh GPT 5.5 here is going to be able to answer that questions because GPT 5 has a theta value of s46 00:03:46.959 --> 00:03:54.800 1.2, and the difficulty of that item is minus 1.2. So, the probability of the right answer is 99. s47 00:03:54.800 --> 00:03:55.560 Okay? s48 00:03:55.560 --> 00:04:01.680 So, with that in mind, what are we doing right here is actually calibrating each question, each item. s49 00:04:01.680 --> 00:04:04.600 So, we are going to improve a lot our estimations. s50 00:04:04.600 --> 00:04:08.880 We're going to improve also our confident intervals and many other properties. s51 00:04:08.880 --> 00:04:18.959 This The other thing that I want to explain here is theta and B is going to be a a pair of numbers that are distributed with normal distributions. s52 00:04:18.959 --> 00:04:20.920 So, we can actually interpret of them. s53 00:04:20.920 --> 00:04:25.960 For example, item of B equal zero means that is going to be average. s54 00:04:25.960 --> 00:04:33.840 Half of the um models in my data set are going to be able to answer that question 50% of the time. s55 00:04:33.840 --> 00:04:41.880 So, that's going to be extremely helpful because right now to evaluate benchmarks, we need that reference compared with other models. s56 00:04:41.880 --> 00:04:46.720 So, it's better to have one uh by default with this methodology. s57 00:04:46.720 --> 00:04:53.520 Actually, this model is called item response theory, which is the evolution of classical test theory. s58 00:04:53.520 --> 00:04:54.240 Okay? s59 00:04:54.240 --> 00:04:57.120 So, on top of that, I'm going to have another parameter. s60 00:04:57.120 --> 00:05:01.560 It's going to be the a slope, that discrimination of that item. s61 00:05:01.560 --> 00:05:06.760 So, high discrimination are going to have steeper functions. s62 00:05:06.760 --> 00:05:14.320 Also, we can have random functions, items that are not related with intelligence, which I don't want. s63 00:05:14.320 --> 00:05:22.440 But, something happens and even worse, we're going to have items that have negative correlation with the actual intelligence of the model. s64 00:05:22.440 --> 00:05:35.880 But, that keep that in mind because each item is going to be a difficulty level, B, and it's going to have a discrimination parameter here, A. So, last thing that you need to understand is how can we estimate s65 00:05:35.880 --> 00:05:42.560 the intelligence level of a given model if we have IRT modeling item response theory. s66 00:05:42.560 --> 00:05:48.120 We're going to actually um go by one by one for each uh question here. s67 00:05:48.120 --> 00:05:55.520 So, I'm going to plot here on top its item curve and below you are going to see the relative likelihood of theta. s68 00:05:55.520 --> 00:05:57.040 That's the estimation of the intelligence. s69 00:05:57.040 --> 00:06:03.440 So, as you can see here we are going to use all curves to combine them in one distribution. s70 00:06:03.440 --> 00:06:08.720 Over time, obviously with more questions we're going to have have a very better estimation. s71 00:06:08.720 --> 00:06:20.600 So, if we add all of them at the end, I'm going to have my final final estimation that is extremely nice to have a distribution here because I'm can have a likelihood interval. s72 00:06:20.600 --> 00:06:25.240 With with classical test theory is more complicated to be have. s73 00:06:25.240 --> 00:06:25.760 Okay? s74 00:06:25.760 --> 00:06:40.040 So, even with that with a sir scale with B and theta, even with the likelihood interval, if you think that this not this is not useful at all, let me show you one last example, okay? s75 00:06:40.040 --> 00:06:42.120 Again, this is real data. s76 00:06:42.120 --> 00:06:44.880 So, I'm going to compare two models. s77 00:06:44.880 --> 00:06:51.040 Here is going to be on the left side Cloud Opus 4.1, yeah? s78 00:06:51.040 --> 00:06:54.680 That has 245 right answers. s79 00:06:54.680 --> 00:07:01.840 On the other side we are going to have Gemini 3 Pro, which has 247, okay? s80 00:07:01.840 --> 00:07:08.120 So, as you can see the difference here is quite small for 337 questions. s81 00:07:08.120 --> 00:07:14.520 But if you use item response theory, you can see that the difference between all of them is almost one standard deviation. s82 00:07:14.520 --> 00:07:18.440 That means Gemini 3 Pro is by far more intelligent. s83 00:07:18.440 --> 00:07:18.600 Okay? s84 00:07:18.600 --> 00:07:23.919 Which makes sense because it's a uh later model, okay? s85 00:07:23.919 --> 00:07:35.240 So, as you can see, counting the number of right answers is is not a good approach because I can create benchmarks that are not calibrated and even if I get a lot of right answers, s86 00:07:35.240 --> 00:07:37.680 I'm not more intelligent than other models. s87 00:07:37.680 --> 00:07:51.040 That could happen, for example, if Gemini is able to answer by far harder questions uh if uh Claude is able to answer more of them, maybe Claude was able to answer only the easiest one. s88 00:07:51.040 --> 00:07:53.320 So, the theta levels should be different. s89 00:07:53.320 --> 00:07:56.600 Okay, that's the that's actually what happens with IRT. s90 00:07:56.600 --> 00:08:06.600 You have better estimations, you have more parameters to define each item, which is going to be helpful later, and also you have, as you can see here, likelihood intervals. s91 00:08:06.600 --> 00:08:14.480 So, on top of everything, I'm going to show you a few applications that you can use with IRT. s92 00:08:14.480 --> 00:08:19.720 And I hope that you find them interesting. s93 00:08:19.720 --> 00:08:29.320 Okay, so the first application is going to be one of my favorite ones, actually, because you can apply this with one of the skills that I'm going to show. s94 00:08:29.320 --> 00:08:32.640 It's going to take a few minutes if you have the data. s95 00:08:32.640 --> 00:08:41.680 And [clears throat] you can actually uh pick the best items, the best questions on your benchmark, and remove the other ones that aren't working or even fix them. s96 00:08:41.680 --> 00:08:44.080 Okay, so we can audit our benchmark. s97 00:08:44.080 --> 00:08:48.680 So, remember that we have two numbers that represents each item, each question. s98 00:08:48.680 --> 00:08:53.640 We have B, difficulty, and A, which is the slope or the discrimination of the item. s99 00:08:53.640 --> 00:08:58.480 On the right side here, we have very good items, very informative ones. s100 00:08:58.480 --> 00:09:02.640 Here, close to zero, we have kind of noisy with little signal. s101 00:09:02.640 --> 00:09:06.760 And on the other side, we have items that we don't like at all, negative items. s102 00:09:06.760 --> 00:09:18.720 Here, for example, you can see that we have items that correlates uh the other way around, that better models are actually getting that answer wrong, which makes no sense. s103 00:09:18.720 --> 00:09:26.440 So, we can use this to actually find items that are significantly below zero, and we can actually flag them. s104 00:09:26.440 --> 00:09:31.880 With that, I'm going to use another LLM to actually evaluate them. s105 00:09:31.880 --> 00:09:35.560 So, here you can see two items that I detect with that technique. s106 00:09:35.560 --> 00:09:37.640 The first one is actually I don't I don't know the answer. s107 00:09:37.640 --> 00:09:40.000 I ask ChatGPT. s108 00:09:40.000 --> 00:09:44.840 And apparently the gold answer, the answer that is on the benchmark, is not right one. s109 00:09:44.840 --> 00:09:45.120 Okay? s110 00:09:45.120 --> 00:09:46.640 So, this is okay. s111 00:09:46.640 --> 00:09:51.360 But the next example is by far more interesting for me because the answer is kind of right. s112 00:09:51.360 --> 00:09:54.840 So, it's something that a lot of people can miss. s113 00:09:54.840 --> 00:09:58.000 So, here he's asking, "What is the total number of passengers?" s114 00:09:58.000 --> 00:10:07.600 The gold answer, the answer that is on the benchmark, is 583, which is the total people killed passengers plus crew. s115 00:10:07.600 --> 00:10:12.600 But the right answer, if you pay attention, I'm asking only about passengers, which is another number. s116 00:10:12.600 --> 00:10:19.640 So, again, with a very little effort, if you have the data set, you can actually find items that are mislabeled. s117 00:10:19.640 --> 00:10:20.720 This is the most common thing. s118 00:10:20.720 --> 00:10:24.440 But sometimes they are not mislabeled, they are like bad items. s119 00:10:24.440 --> 00:10:28.600 So, you should remove them or even improve them. s120 00:10:28.680 --> 00:10:30.680 This second application for me is amazing. s121 00:10:30.680 --> 00:10:35.840 It saves a lot of time, a lot of tokens, and therefore a lot of money. s122 00:10:35.840 --> 00:10:36.080 Okay? s123 00:10:36.080 --> 00:10:43.120 So, um it's quite common for organizations to have their own benchmarks to evaluate which model is better for them. s124 00:10:43.120 --> 00:10:45.800 Especially with open source models. s125 00:10:45.800 --> 00:10:52.760 With that in mind, it's quite important to reduce the size of the benchmark to find the optimal size of the benchmark. s126 00:10:52.760 --> 00:10:56.839 Before we couldn't do that because we didn't have any property of the item. s127 00:10:56.839 --> 00:11:01.720 But right now, with item response theory, we can actually pick the best items. s128 00:11:01.720 --> 00:11:02.480 Okay? s129 00:11:02.480 --> 00:11:10.760 So, this is a simplification, but we're going to say that items with high levels of discrimination are going to be the best ones. s130 00:11:10.760 --> 00:11:11.200 Okay? s131 00:11:11.200 --> 00:11:13.960 With that in mind, I can do this. s132 00:11:13.960 --> 00:11:18.760 So, again, real benchmarks with real data. s133 00:11:18.760 --> 00:11:25.320 Uh for for this benchmark, I'm going to target a 99% correlation with the original ranking. s134 00:11:25.320 --> 00:11:30.640 So, it's going to be like almost perfect for many use cases. s135 00:11:30.640 --> 00:11:46.920 And what are we going to do speak one by one starting with the best item, the item with the highest A. So, with that methodology in mind, we're going to get that around 97 items compared with 484. s136 00:11:46.920 --> 00:11:49.200 That's almost 5x. s137 00:11:49.200 --> 00:11:54.520 We are going to have the same ranking than before or almost the same ranking as before. s138 00:11:54.520 --> 00:11:58.240 To be fair, I tried to do the same thing but randomly. s139 00:11:58.240 --> 00:12:05.360 So, if you pick items by random, you're going to observe uh by far worse performance. s140 00:12:05.360 --> 00:12:07.480 So, it's extremely helpful. s141 00:12:07.480 --> 00:12:19.360 Maybe this is kind of uh weird for you because uh if we can evaluate items with by far less questions, why are actually using all the benchmark? s142 00:12:19.360 --> 00:12:23.560 So, and the answer for that is that we are not used to calibrate benchmarks. s143 00:12:23.560 --> 00:12:29.720 We are like assuming that more questions uh means better estimation, which is not true. s144 00:12:29.720 --> 00:12:33.560 For example, I can have two questions that kind of overlap. s145 00:12:33.560 --> 00:12:36.200 Their curves are are more or less the same. s146 00:12:36.200 --> 00:12:42.720 So, even if I ask two of them, I'm going to get more or less the same information about your intelligence, okay? s147 00:12:42.720 --> 00:12:47.440 And also, there are a few items that you can notice that they are not extremely informative. s148 00:12:47.440 --> 00:12:50.320 They don't correlate with actual intelligence. s149 00:12:50.320 --> 00:12:53.360 So, you can reduce them or use them less. s150 00:12:53.360 --> 00:13:02.480 I'm not saying that you should remove uh like the the the rest of the data set, but you can use this for another application that we are going to talk s151 00:13:02.480 --> 00:13:09.160 that you can use a subset of items each time that you apply the benchmark, which is going to be extremely useful. s152 00:13:09.160 --> 00:13:14.720 But uh be careful because that uh this does not happen on every benchmark, okay? s153 00:13:14.720 --> 00:13:26.680 So, for example, GPQA, which is a extremely well-designed uh data set, the benchmark here, as you can see, even if you pick at random, you're going to get more or less the same result. s154 00:13:26.680 --> 00:13:33.440 And the reason for that is that every item here is extremely discrim- discriminative and also they don't overlap. s155 00:13:33.440 --> 00:13:40.440 So, they are all of them are useful at every time along the benchmark, okay? s156 00:13:40.800 --> 00:13:54.520 So, now given that we have for each item one function, we can also calculate the error, the unexpected behavior, the outliers for each question, which is going to be amazing for many applications. s157 00:13:54.520 --> 00:14:02.200 We can actually find out if we are leaking information, if we are overfitting with the benchmark, and other kinds of contaminations. s158 00:14:02.200 --> 00:14:06.269 So, let me go back to the matrix that we started with. s159 00:14:06.269 --> 00:14:06.760 [clears throat] s160 00:14:06.760 --> 00:14:12.760 And as you can see, every question is going to have the estimation and the actual answer. s161 00:14:12.760 --> 00:14:17.000 And that's good because we can observe a few outliers here. s162 00:14:17.000 --> 00:14:20.520 For example, the minute three pro should be able to answer that question. s163 00:14:20.520 --> 00:14:28.120 Actually, our model says that 80 size 86% of the times Gemini should be able to answer that. s164 00:14:28.120 --> 00:14:33.280 But even with that, uh that question is wrong. s165 00:14:33.280 --> 00:14:40.960 I cannot say why, but I can actually detect those outliers, those weird patterns, okay? s166 00:14:40.960 --> 00:14:51.640 And with that in mind, we can actually calculate the residuals, the error for that specific question, which is going to be a useful for different applications that I'm going to explain later. s167 00:14:51.640 --> 00:15:01.960 The other way around, we can actually see here that for example, deep seek R1 is having here a right answer even if it's not expected, okay? s168 00:15:01.960 --> 00:15:06.080 Again, that does not mean that uh we are overfitting or anything. s169 00:15:06.080 --> 00:15:14.839 Actually, we can sample the same question more than once, so we can average them together and maybe with that we can like resolve that outlier. s170 00:15:14.839 --> 00:15:20.600 But in any case, we have a new tool that we can use to analyze the data. s171 00:15:20.600 --> 00:15:29.840 On top of that, actually in psychometrics, we have a few techniques to see if the behavior is consistent, if that makes sense, okay? s172 00:15:29.840 --> 00:15:35.840 So, for example, here, I can observe that O4Minnie is the less consistent one. s173 00:15:35.840 --> 00:15:44.200 You can see that has a lot of red uh a lot of green questions here in the middle, which makes no sense. s174 00:15:44.200 --> 00:15:51.320 So, you should be able to use that that to, for example, detect if your inference platform is not working well. s175 00:15:51.320 --> 00:16:03.240 Because uh if your for whatever reason your inference platform is not actually running the models or the quantization is actually wrong, you're going to observe things like that, behaviors that are not expected, s176 00:16:03.240 --> 00:16:03.720 okay? s177 00:16:03.720 --> 00:16:04.760 Should be consistent. s178 00:16:04.760 --> 00:16:14.880 It's this matrix is not perfect, it's going to have noise, but we should be able to more or less estimate a level of intelligence given that we can predict more or less s179 00:16:14.880 --> 00:16:20.400 what questions are you going to get right at what question are you getting it are you going to get it wrong. s180 00:16:20.400 --> 00:16:21.360 Okay? s181 00:16:21.360 --> 00:16:25.560 Again, residuals are going to be extremely helpful for the next application. s182 00:16:25.560 --> 00:16:32.400 But only with this, I think you can actually look at the data in other [clears throat] way. s183 00:16:32.880 --> 00:16:33.440 Okay. s184 00:16:33.440 --> 00:16:39.040 So, let's say that you are building a benchmark, a very complicated one, very expensive one. s185 00:16:39.040 --> 00:16:43.040 So, you don't want that benchmark to be leaked on the internet. s186 00:16:43.040 --> 00:16:47.800 Neither you want other organizations to train their models with that benchmark, okay? s187 00:16:47.800 --> 00:16:54.320 Because it's extremely valuable and if you can't protect that benchmark over time, it's going to be more valuable. s188 00:16:54.320 --> 00:17:00.080 So, with that in mind, we can actually use what we call adaptive testing, okay? s189 00:17:00.080 --> 00:17:01.120 What is that? s190 00:17:01.120 --> 00:17:09.120 I'm going to pick a random set uh of items that are going to be representative of my benchmark, and I'm going to call that an anchor set. s191 00:17:09.120 --> 00:17:21.199 An anchor set should be representative of my entire benchmark, and I'm going to use those items with every applicant, with every model from any organization that I'm working with. s192 00:17:21.199 --> 00:17:26.959 But for every organization, I'm going to pick uh one individual set. s193 00:17:26.959 --> 00:17:33.680 I'm going to call that fingerprint set that I'm going to show only to that specific organization. s194 00:17:33.680 --> 00:17:37.120 And I can do also the same thing with another one. s195 00:17:37.120 --> 00:17:46.320 Pay attention to this because it's important that I pick extremely complicated items from my benchmark for those fingerprint sets. s196 00:17:46.320 --> 00:17:58.440 Over time, let's say a few months later, uh every organization is releasing their new models, and I'm going to observe I'm going to like run the benchmarks again, and I'm going to use residuals s197 00:17:58.440 --> 00:18:04.320 to uh see if those models are extremely good at those specific questions. s198 00:18:04.320 --> 00:18:06.920 So here, this is a synthetic data set. s199 00:18:06.920 --> 00:18:17.320 You can see that the average residual for that organization for that a specific fingerprint set is extremely unlikely. s200 00:18:17.320 --> 00:18:24.240 So this is not bulletproof, but this is a extremely good technique that you can use to protect your benchmarks. s201 00:18:24.240 --> 00:18:30.640 As you can see here, the average residuals for one organization is by far bigger than the other one. s202 00:18:31.720 --> 00:18:39.200 In psychology, it's really important to find out if one of our items are actually biased against one specific group. s203 00:18:39.200 --> 00:18:44.000 We can use the same techniques to research how models behave with items. s204 00:18:44.000 --> 00:18:46.560 So I'm going to show you that with one example. s205 00:18:46.560 --> 00:18:48.960 I'm going to split the data set in two groups. s206 00:18:48.960 --> 00:18:52.000 In this case, I'm going to use open weights and closed weights. s207 00:18:52.000 --> 00:18:59.440 You can use whatever variable that is interesting for you, and even you can create more than one group. s208 00:18:59.480 --> 00:19:06.920 After that, I'm going to for each item create two different curves for each group. s209 00:19:06.920 --> 00:19:14.720 And what we expect if the item is unbiased is that both lines are kind of overlapping. s210 00:19:14.720 --> 00:19:23.160 If not, we are going to calculate this difference, the gap between those ones, and that difference should be zero or close to zero. s211 00:19:23.160 --> 00:19:36.600 In this case, we can detect we can apply that technique to all items on my benchmark, and actually we can find out that there are few items that are better for closed weight of models and better for open weights models. s212 00:19:36.600 --> 00:19:47.240 I'm not going to show the items because I don't want to leak them on the internet, but if you apply this technique to a real benchmark, you can actually notice a few patterns there. s213 00:19:47.240 --> 00:19:53.120 There are a set of questions that have something in common that are better for open weight models. s214 00:19:53.120 --> 00:20:02.160 I cannot pinpoint the real reason for that, but I can say that this technique could be used to understand how they train their models. s215 00:20:02.760 --> 00:20:06.000 The last application that I'm going to show you is kind of interesting. s216 00:20:06.000 --> 00:20:10.280 It's the most complicated one, but in a way is the most interesting one. s217 00:20:10.280 --> 00:20:25.480 So, I'm going to use the residuals to actually have a DNA, a fingerprint for models, and the idea is that we can actually look at them and see if two different models are related somehow, okay? s218 00:20:25.480 --> 00:20:31.160 So, I'm going to make a correlation matrix here, and also I'm going to make a super simple projection. s219 00:20:31.160 --> 00:20:41.320 So, the first thing that you can notice is that there are a few models that are extremely close here, that they are from the same lab, and also they have the same history there. s220 00:20:41.320 --> 00:20:44.280 Like different versions for the same model. s221 00:20:44.280 --> 00:20:49.840 Also here, you can observe we have Deep Seek, different distillation of Deep Six. s222 00:20:49.840 --> 00:20:52.640 We have also Queen. s223 00:20:52.640 --> 00:20:55.720 So, we can observe some kind of patterns there. s224 00:20:55.720 --> 00:21:04.480 It makes sense because if a model has a similar history with other model, we can expect the same kind kind of errors, okay? s225 00:21:04.480 --> 00:21:06.240 And let Let you a few examples. s226 00:21:06.240 --> 00:21:08.840 The first one is going to be this one. s227 00:21:08.840 --> 00:21:14.680 Uh if we have two models that have the same base, we expect high correlations between them. s228 00:21:14.680 --> 00:21:19.720 In this case, we have .38, okay? s229 00:21:19.720 --> 00:21:31.840 Also, we can observe that between distillations and its base model, which could be extremely interesting if you want to detect distillation of your model uh without consent. s230 00:21:31.840 --> 00:21:39.000 Also, we can detect detect the same model with different effort levels, which makes sense because it's almost the same thing. s231 00:21:39.000 --> 00:21:46.760 And also, we can say uh we can actually detect if we have different versions or evolution of the same model. s232 00:21:46.760 --> 00:21:51.400 Here, we have Llama, but also work with Gemini here, okay? s233 00:21:51.400 --> 00:22:05.320 So, again, this is not as useful as others for day-to-day, but I think this opens an area of research to understand how models are related with each other and even detect distillations. s234 00:22:06.200 --> 00:22:09.640 I really hope that you find inspiring this talk. s235 00:22:09.640 --> 00:22:17.480 My goal here was to open the gates of uh psychometrical research for LLMs. s236 00:22:17.480 --> 00:22:24.080 I think we can improve a lot how we benchmark LLMs with very basic maths here. s237 00:22:24.080 --> 00:22:29.360 But there are a lot of uh ideas that you should explore because I didn't have time. s238 00:22:29.360 --> 00:22:32.840 The first one is multidimensionality and hierarchical models. s239 00:22:32.840 --> 00:22:40.600 I'm expecting if we apply them to LLMs to see different skill levels for different kind of task. s240 00:22:40.600 --> 00:22:45.120 Also, we can merge different benchmarks to improve the estimation of each one of them. s241 00:22:45.120 --> 00:22:47.880 Makes sense if you have IRT. s242 00:22:47.880 --> 00:22:55.000 And also, this has been done in a in a research paper called Meta-Benchmark that I highly recommend you to read. s243 00:22:55.000 --> 00:23:01.920 Also, we can add another signal that we think correlates with intelligence, for example, latency or tokens. s244 00:23:01.920 --> 00:23:12.120 And another very promising idea is to use psychometrical models to actually measure alignment and use that for also interpretability. s245 00:23:12.120 --> 00:23:18.120 I think mechanistic interpretability could help a lot psychometrics here. s246 00:23:18.120 --> 00:23:20.360 I'm working on all those areas. s247 00:23:20.360 --> 00:23:26.760 If you are working on them or you have any idea or you need help to apply them to your benchmark, let me know. s248 00:23:26.760 --> 00:23:35.440 Thank you for your time and here you have all the materials, skills, and benchmarks so you can play around with them.