WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XzJD1bvXKjs NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XzJD1bvXKjs.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.800 --> 00:00:13.679 All right. s3 00:00:13.679 --> 00:00:15.920 Hello everybody. s4 00:00:15.920 --> 00:00:19.279 Thank you for coming to this delightfully nerdy talk. s5 00:00:19.279 --> 00:00:21.840 Uh this talk has a really long title. s6 00:00:21.840 --> 00:00:23.920 Uh so let me give you the short version up front. s7 00:00:23.920 --> 00:00:27.439 You write skills files uh and stuff them full of instructions. s8 00:00:27.439 --> 00:00:31.039 At some point the model stops keeping track of all of them. s9 00:00:31.039 --> 00:00:33.040 The question is where is that point? s10 00:00:33.040 --> 00:00:36.480 At what point have you put too many instructions in your skills files? s11 00:00:36.480 --> 00:00:40.800 Uh and the answer has changed a lot in the last year. s12 00:00:40.800 --> 00:00:41.520 I'm Lori. s13 00:00:41.520 --> 00:00:43.920 I'm head of developer relations at Arise AI. s14 00:00:43.920 --> 00:00:48.480 Uh in a former life, I co-founded npm Inc. So some of you may know me from the days of JavaScript. s15 00:00:48.480 --> 00:00:53.360 These days I spend a lot of time thinking about AI and how to test it. s16 00:00:53.360 --> 00:00:57.120 Uh a few months ago I was at AI engineer in Miami which was a good conference. s17 00:00:57.120 --> 00:00:59.680 Uh and I was watching a talk by Dexter Horthy. s18 00:00:59.680 --> 00:01:00.800 Uh it was a good talk. s19 00:01:00.800 --> 00:01:02.640 It was not about this topic at all. s20 00:01:02.640 --> 00:01:13.680 Uh but while he was giving that talk he mentioned as an aside uh that an agent can follow up to about 200 instructions uh before it starts forgetting those instructions. s21 00:01:13.680 --> 00:01:17.520 Uh and he then he moved on in his talk and it was entirely an aside. s22 00:01:17.520 --> 00:01:22.400 Uh and he he mentioned that that figure is from 2025 so things might be better now. s23 00:01:22.400 --> 00:01:27.759 Um, and I stopped listening for a second because I was like, 200 instructions. s24 00:01:27.759 --> 00:01:30.960 Uh, is not very many instructions at all. s25 00:01:30.960 --> 00:01:31.360 Right? s26 00:01:31.360 --> 00:01:35.439 A decent skills file blows past 200 instructions almost immediately. s27 00:01:35.439 --> 00:01:42.799 Um, if the user says X, do Y, always include a section on Z, never use the phrase W, every one of those is a separate instruction. s28 00:01:42.799 --> 00:01:49.680 Uh, and if the model quietly stops tracking them after 200, that's a really hard ceiling on the complexity of what you can build. s29 00:01:49.680 --> 00:01:52.560 Uh, so I wanted to know where he got that number first. s30 00:01:52.560 --> 00:01:55.360 Uh, and I wanted to know if it was true. s31 00:01:55.360 --> 00:01:57.280 Uh, so you know the feeling that I'm talking about. s32 00:01:57.280 --> 00:02:03.200 You write this big beautiful skills file, pages of rules, edge cases, tone, formatting, you hand it to the agent, it does the thing. s33 00:02:03.200 --> 00:02:07.360 Uh, and you look at the output and go, did it actually pay attention? s34 00:02:07.360 --> 00:02:16.080 Did it actually follow all of these rules or did it just sort of, you know, do what it felt like and sort of give me a close simulacum of what I was expecting? s35 00:02:16.080 --> 00:02:18.560 Um, you can't really tell or can you? s36 00:02:18.560 --> 00:02:19.840 More on that later. s37 00:02:19.840 --> 00:02:24.879 Um, and so you live with this lowgrade anxiety every time you hit run. s38 00:02:24.879 --> 00:02:30.879 Um, and that feeling is what this research is about and what we're trying to find out if we can avoid. s39 00:02:30.879 --> 00:02:33.440 So here's my promise for your next 18 minutes. s40 00:02:33.440 --> 00:02:39.120 Uh, I'm going to show you where that 200 number came from, whether it's still true, and what the real number is today. s41 00:02:39.120 --> 00:02:41.599 Uh, because it moved by an order of magnitude. s42 00:02:41.599 --> 00:02:46.239 Uh, and then we're going to talk about what that means for you to take away. s43 00:02:46.239 --> 00:02:54.720 uh how long your skills and prompts can actually be and what that should what changes you should make to your workflow as a result. s44 00:02:54.720 --> 00:02:56.879 So the 200 number isn't folklore. s45 00:02:56.879 --> 00:03:06.720 Uh it comes from a real benchmark called IFScale uh from a paper uh by this guy whose name I'm going to mess up Jeroslowitch uh and co-authors last year. s46 00:03:06.720 --> 00:03:09.599 And the test is beautifully simple. s47 00:03:09.599 --> 00:03:12.080 Uh here's how if scale works. s48 00:03:12.080 --> 00:03:24.239 If you ask the model to write a business report uh and you give it a list of specific words that it has to include exactly in the report, include the exact word customer, include the exact word revenue, and so on for as many words as you want. s49 00:03:24.239 --> 00:03:27.200 Each of those is an instruction that it has to follow. s50 00:03:27.200 --> 00:03:31.200 Uh and then you count how many of those exact words showed up. s51 00:03:31.200 --> 00:03:36.480 Um so because the test is so simple, you only have to keep two numbers in your head. s52 00:03:36.480 --> 00:03:41.200 One is density, which we call n. That is how many rules we're talking about at once. s53 00:03:41.200 --> 00:03:47.120 And the second is accuracy, which is the percentage of those rules uh that it was able to actually follow. s54 00:03:47.120 --> 00:03:55.280 Uh now you might say uh that including random words in a report is not the same as following real instructions and fair enough and we're going to talk about that. s55 00:03:55.280 --> 00:03:57.920 Um but the keywords are a proxy. s56 00:03:57.920 --> 00:04:03.040 Uh include the word revenue is the same shape of task as include a section on pricing, right? s57 00:04:03.040 --> 00:04:04.239 Or never use this phrase. s58 00:04:04.239 --> 00:04:08.959 It is a discrete named constraint that you've told the agent that it has to follow. s59 00:04:08.959 --> 00:04:16.000 Um, if a model can't track 200 words in one prompt, it's definitely going to struggle with 200 more complicated instructions. s60 00:04:16.000 --> 00:04:20.320 Uh, so uh, if anything, it's going to do worse. s61 00:04:20.320 --> 00:04:21.919 So, this number is a ceiling. s62 00:04:21.919 --> 00:04:24.000 Uh, this number is as high as you can go. s63 00:04:24.000 --> 00:04:27.360 If you give it more complicated instructions, the number is probably going to get lower. s64 00:04:27.360 --> 00:04:30.160 And 200 is a really low ceiling. s65 00:04:30.160 --> 00:04:39.919 Um so before chasing new models you have to do good science which means that you have to replicate the uh old result and make sure uh that the 200 ceiling is real. s66 00:04:39.919 --> 00:04:42.000 So I reran the original benchmark. s67 00:04:42.000 --> 00:04:48.000 Um the original paper tested a whole batch of models uh and models uh live and die really fast. s68 00:04:48.000 --> 00:04:56.639 So uh by the time I got around to doing this testing only three of the models in the original set of 10 models that they used were still available via any kind of API. s69 00:04:56.639 --> 00:05:01.280 Uh so they were GPT 4.1, Claude Sonnet 4, and Gemini 2.5 Pro. s70 00:05:01.280 --> 00:05:05.199 Those were models that were available 12 months ago that are still available now. s71 00:05:05.199 --> 00:05:08.800 Um and that is why we tested those three because they were what was left. s72 00:05:08.800 --> 00:05:14.880 Um and since I first published this research a couple of weeks ago, uh one of those three models has been retired. s73 00:05:14.880 --> 00:05:18.000 So this was the last possible time that I could have run this test. s74 00:05:18.000 --> 00:05:20.880 Um so of that lineup, we're already down to two. s75 00:05:20.880 --> 00:05:22.960 So don't get attached to your models. s76 00:05:22.960 --> 00:05:27.199 Um here is the results that we got replicating the original if scale finding. s77 00:05:27.199 --> 00:05:29.840 Uh that is accuracy on the vertical axis. s78 00:05:29.840 --> 00:05:32.240 So it starts at 100% and begins to fall off. s79 00:05:32.240 --> 00:05:36.240 Uh and then the number of rules uh going up along the bottom on log scale. s80 00:05:36.240 --> 00:05:41.520 So every time it gets halfway across it has doubled uh the number of rules that it's dealing with. s81 00:05:41.520 --> 00:05:46.560 Um so by 500 rules you're losing 30 40 50% of them. s82 00:05:46.560 --> 00:05:50.479 Uh our curves matched the results in the original paper within the noise boundary. s83 00:05:50.479 --> 00:05:52.160 So the finding was real. s84 00:05:52.160 --> 00:05:57.840 uh a year ago somewhere around 200 to 300 rules frontier models started falling apart. s85 00:05:57.840 --> 00:06:00.000 That is a really low ceiling. s86 00:06:00.000 --> 00:06:10.160 Uh so that is our baseline and now comes the fun part where we took the exact same test and pointed it at the current frontier or rather what the current frontier was when I ran this test. s87 00:06:10.160 --> 00:06:16.960 So I ran GPT 5.5, Claude Opus 4.7 because 4.8 came out a week after I ran this test. s88 00:06:16.960 --> 00:06:20.319 Uh Gemini 3.1 Pro and Deepseek V4 Pro. s89 00:06:20.319 --> 00:06:23.440 So, I gave them the same prompt, the same words, the same everything. s90 00:06:23.440 --> 00:06:27.199 And I immediately ran into a problem, which is that they aced it. s91 00:06:27.199 --> 00:06:30.960 They all scored 100% immediately on this test. s92 00:06:30.960 --> 00:06:32.479 Absolutely no bugs. s93 00:06:32.479 --> 00:06:39.919 Uh, so we'd built a test to find the ceiling and the models had walked straight through the ceiling without noticing that the ceiling was there. s94 00:06:39.919 --> 00:06:43.919 Um, and that was a problem because the benchmark was written to top out at 500 words. s95 00:06:43.919 --> 00:06:47.280 So, I had to change the benchmark in order to be able to find the new ceiling. s96 00:06:47.280 --> 00:06:48.720 So, I moved the goalposts. s97 00:06:48.720 --> 00:06:50.400 I gave it more words to include. s98 00:06:50.400 --> 00:06:52.800 I doubled uh it from 500 to a,000. s99 00:06:52.800 --> 00:06:54.960 I doubled it again from a,000 to 2,000. s100 00:06:54.960 --> 00:06:58.000 And I kept doing that until I hit a 10,000word vocabulary. s101 00:06:58.000 --> 00:07:03.360 And that is where I began to find the ceiling of what models can do these days. s102 00:07:03.360 --> 00:07:06.560 Um, so let me put up the this is the money slide. s103 00:07:06.560 --> 00:07:07.919 This is the results. s104 00:07:07.919 --> 00:07:12.240 Remember log scale on the on the uh x-axis there. s105 00:07:12.240 --> 00:07:15.840 So it's going from 500 to 1,000 to 5,000 to 10,000. s106 00:07:15.840 --> 00:07:20.880 Uh so it looks like that scale is falling off of a cliff and it's actually happening over like a thousand numbers. s107 00:07:20.880 --> 00:07:26.560 Um but uh look how far to the right these new curves get before they bend. s108 00:07:26.560 --> 00:07:32.560 A year ago they were falling over at 200 to 300 instructions and now depending on the model the boundary is closer to 2,000. s109 00:07:32.560 --> 00:07:38.080 And for the best of them it is up to 5,000 instructions before they begin to fall off a cliff. s110 00:07:38.080 --> 00:07:44.639 So in about 12 months frontier models got close to 10 times better at following instructions simultaneously. s111 00:07:44.639 --> 00:07:49.039 That is the headline fighting and there is a lot of nuance that we need to get into. s112 00:07:49.039 --> 00:07:54.000 Um the capacity to track 2,000 named constraints in a single prompt is there. s113 00:07:54.000 --> 00:08:07.840 Um and that's really interesting because I think uh I don't know if everybody else feels this way but like it feel it felt to me like the the jump from you know GPT 5.1 to GPT 5.5 was kind of incremental, right? s114 00:08:07.840 --> 00:08:10.160 It didn't feel like we'd got 10 times better. s115 00:08:10.160 --> 00:08:17.120 But this is a test that really matters to uh a very practical thing like how long can my skills file be? s116 00:08:17.120 --> 00:08:20.720 Uh and in the course of a year we got 10 times better. s117 00:08:20.720 --> 00:08:25.199 Uh and that thing that gets me is that this benchmark is barely a a year old. s118 00:08:25.199 --> 00:08:27.599 A year later 500 is a rounding error. s119 00:08:27.599 --> 00:08:29.280 Uh and this keeps moving under my feet. s120 00:08:29.280 --> 00:08:33.919 I tested 4.7 uh opus 4.7. Opus 4.8 is even better. s121 00:08:33.919 --> 00:08:37.919 Um so this chart is a little out of date already which is kind of the whole point. s122 00:08:37.919 --> 00:08:49.920 If you set your engineering assumptions about how skills files should work, about how prompt how long your prompt can be, and you did that more than about six months ago, you are incorrect now, and you should probably be re-engineering s123 00:08:49.920 --> 00:08:51.440 how you do stuff. s124 00:08:51.440 --> 00:09:00.800 Uh, but there is more to this story uh because the way that the a models failed uh changed dramatically. s125 00:09:00.800 --> 00:09:03.279 Uh, and the way that they failed is very important. s126 00:09:03.279 --> 00:09:08.160 This part was a completely unexpected finding when I started running the experiment. s127 00:09:08.160 --> 00:09:13.360 Uh, and it totally messed up my test to start with because uh, the old failure mode was boring. s128 00:09:13.360 --> 00:09:17.920 They would just forget instructions and I could measure how many instructions they had remembered or forgotten. s129 00:09:17.920 --> 00:09:23.360 Uh, but the new ones fall apart in their own weird extremely onbrand way. s130 00:09:23.360 --> 00:09:26.640 Uh, so let me introduce you to how these four models fail. s131 00:09:26.640 --> 00:09:29.440 Uh, Deepseek 4 is a traditional model. s132 00:09:29.440 --> 00:09:30.560 It just forgets things. s133 00:09:30.560 --> 00:09:32.399 It doesn't have any drama. s134 00:09:32.399 --> 00:09:38.640 um it starts forgetting instructions around 750 rules and by 2000 it's dropping nearly half of them. s135 00:09:38.640 --> 00:09:43.360 Uh so it just forgets which frankly is the failure mode that I trust most because it's predictable. s136 00:09:43.360 --> 00:09:44.720 It's very easy to measure. s137 00:09:44.720 --> 00:09:48.959 Uh and the other models were not nearly as cooperative. s138 00:09:48.959 --> 00:09:54.800 Uh Opus 4.7 uh would decide repeatedly that the test was dangerous. s139 00:09:54.800 --> 00:09:59.440 Uh and what it would do is it would refuse at the API level to complete the test. s140 00:09:59.440 --> 00:10:06.000 I didn't know that there was an API response that you could get from Claude where it was like, "No, I could do this, but I'm not going to." s141 00:10:06.000 --> 00:10:12.080 Uh, but that's absolutely an API level response that Claude supports because they care so much about safety. s142 00:10:12.080 --> 00:10:15.120 Uh, and I started getting those all of the time. s143 00:10:15.120 --> 00:10:19.519 Uh, and the reason that was happening is because Claude has a very sensitive safety classifier. s144 00:10:19.519 --> 00:10:27.360 Uh, and if you put in certain combinations of words like say anthrax and cyanide, it decides that the whole request is dangerous and it bails out. s145 00:10:27.360 --> 00:10:35.600 Uh, and if you remember what my test does, my test is throwing uh 5 to 10,000 random words into uh into an instruction file. s146 00:10:35.600 --> 00:10:41.680 And so my my randomly selected words contained all sorts of things that looked dangerous in combination to the safety filter. s147 00:10:41.680 --> 00:10:45.760 And so it kept bailing saying that I was asking it to, you know, make a bomb or something. s148 00:10:45.760 --> 00:10:59.279 Um, so, uh, we had to for to get Claude to cooperate, I had to take all of my words and run them through OpenAI safety filter and filter out all of the naughty looking words so that it could get to anywhere. s149 00:10:59.279 --> 00:11:02.240 Once I given it that, Claude did really well. s150 00:11:02.240 --> 00:11:12.959 Uh so but the failure mode is that Claude is more likely to decide what you're doing is dangerous very early on uh at you know even two or 300 instructions if what you're doing s151 00:11:12.959 --> 00:11:17.519 uh is you know contains anything to do with medical advice because medical things often are dual purpose. s152 00:11:17.519 --> 00:11:18.320 They can be dangerous. s153 00:11:18.320 --> 00:11:19.600 They can be safe. s154 00:11:19.600 --> 00:11:24.240 Um so uh the third failure mode was Gemini 3.1 Pro. s155 00:11:24.240 --> 00:11:28.160 Gemini is rock solid all the way out uh to 5,000 instructions. s156 00:11:28.160 --> 00:11:30.079 It does extremely well. s157 00:11:30.079 --> 00:11:32.240 um genuinely one of the best on the chart. s158 00:11:32.240 --> 00:11:34.560 Uh and then past that it gets weird. s159 00:11:34.560 --> 00:11:39.760 Um it doesn't forget the instructions, it gets overwhelmed by the instructions. s160 00:11:39.760 --> 00:11:46.720 What it tries to do is it uh it uses thinking tokens to make sure that it is following all of the instructions at once. s161 00:11:46.720 --> 00:11:50.720 And when the number of instructions gets really high, it uses all of its thinking tokens. s162 00:11:50.720 --> 00:11:53.200 It uses its entire token budget thinking. s163 00:11:53.200 --> 00:11:55.440 And then it doesn't give any output. s164 00:11:55.440 --> 00:12:06.720 It's it gets to like nine, you know, if you've given it 10,000 tokens worth, it'll get 9,500 tokens worth of thinking and then give you a 500word response which doesn't contain any of the tokens. s165 00:12:06.720 --> 00:12:16.480 Uh so it thinks itself into a corner and runs out of room to actually answer, which is very expensive, uh and totally unhelpful, which is kind of on brand, isn't it? s166 00:12:16.480 --> 00:12:17.538 Um s167 00:12:17.538 --> 00:12:18.079 [snorts] s168 00:12:18.079 --> 00:12:21.760 uh which, you know, I would never say that out loud. s169 00:12:21.760 --> 00:12:28.480 Uh, and finally comes the winner, which is uh, GPT 5.5. GPT 5.5 is the best of the lot. s170 00:12:28.480 --> 00:12:31.839 99% accuracy all the way out to 5,000 rules. s171 00:12:31.839 --> 00:12:35.680 Um, but if you push it far enough, it is by far the weirdest of the bunch. s172 00:12:35.680 --> 00:12:37.680 Uh, because it doesn't refuse outright. s173 00:12:37.680 --> 00:12:39.040 It doesn't silently forget. s174 00:12:39.040 --> 00:12:44.480 Instead, what it does is it gets frustrated and tells you that the test is stupid. s175 00:12:44.480 --> 00:12:46.959 Uh, it starts the report. s176 00:12:46.959 --> 00:12:48.800 It gets a few like that's the thing. s177 00:12:48.800 --> 00:12:55.920 It doesn't start out just saying no. It starts the report, it starts writing the report, and like 500 words into the report, it's like, "No, this is dumb. s178 00:12:55.920 --> 00:12:56.880 I'm not going to do this." s179 00:12:56.880 --> 00:12:58.720 And then it politely tells you, "This is dumb. s180 00:12:58.720 --> 00:13:00.240 I'm not going to do this anymore." s181 00:13:00.240 --> 00:13:06.959 Uh, that is the actual response that it gave me, but that that was like 5,000 words into the into this business report that I told it to generate. s182 00:13:06.959 --> 00:13:10.639 Um, so it's not wrong, right? s183 00:13:10.639 --> 00:13:17.360 I was asking for a coherent business report that on no particular subject that contains 5,000 random words. s184 00:13:17.360 --> 00:13:19.279 You're right, Gemini DPT. s185 00:13:19.279 --> 00:13:21.920 this is a a stupid thing to ask for. s186 00:13:21.920 --> 00:13:26.959 Um, which is a deeply unreasonable request and GPT called this out on it. s187 00:13:26.959 --> 00:13:37.040 Um, but it still counts as a failure in the test because the half-finish report that it gives you is missing most of the keywords and it is also the hardest one to detect because claude bails immediately. s188 00:13:37.040 --> 00:13:39.200 Claude says, "No, I'm not going to do this." s189 00:13:39.200 --> 00:13:41.360 Uh, Deepseek does its best. s190 00:13:41.360 --> 00:13:50.160 Uh, but GPT does what looks like a good job unless you read all the way to the end of the report where it says, "No, actually I'm going to bail because this is stupid." s191 00:13:50.160 --> 00:14:03.440 Um, so if you step back and look at the four together, Deep Sea quietly forgets, Claude gets scared and refuses, Gemini overthinks itself into silence, and GPT 5.5 finishes half of the job and tells you that the rest of it is beneath it. s192 00:14:03.440 --> 00:14:09.839 Um, and the point was the point isn't which one of these is funniest, although it is genuinely a little funny. s193 00:14:09.839 --> 00:14:14.880 Uh the point is that did it follow my instructions no longer has one failure mode. s194 00:14:14.880 --> 00:14:24.880 It has four different ways that it can fail and you can't recognize that failure unless you know which model you're dealing with and what its m what its pattern of failure is going to be. s195 00:14:24.880 --> 00:14:27.920 Uh so the models get 10 got 10x better. s196 00:14:27.920 --> 00:14:29.279 They fail in funny ways. s197 00:14:29.279 --> 00:14:32.000 Why should you care when you uh get back to your desk? s198 00:14:32.000 --> 00:14:35.040 Because three things have changed to your workflow. s199 00:14:35.040 --> 00:14:40.480 The first is that a year ago, the smart move was to keep every skills file very very short. s200 00:14:40.480 --> 00:14:51.040 Uh under 200 instructions, then point off to subsklls and a whole like you know byzantine labyrinth of uh additional skills files and subfiles and things like that. s201 00:14:51.040 --> 00:14:58.079 Uh and you mo you were compressing your your instructions to fit into a very small available space and you don't need to do that anymore. s202 00:14:58.079 --> 00:15:00.320 Your skills files can be very long. s203 00:15:00.320 --> 00:15:07.920 Um, number two is that if your use case needs a 100 specific rules or 300, you can just put them all in the prompt. s204 00:15:07.920 --> 00:15:13.120 Uh, you don't have to lie awake wondering whether which ones the model silently ignored. s205 00:15:13.120 --> 00:15:21.360 Um, and if you've been thinking uh about uh your own lived experience of using models, uh you probably recognize this. s206 00:15:21.360 --> 00:15:30.800 you've discovered that you've got less worried about how long your your prompt is going to get uh because the models have genuinely got 10 times better at following your prompts. s207 00:15:30.800 --> 00:15:35.120 Um 2,000 named constraints is an entire style guide, right? s208 00:15:35.120 --> 00:15:38.240 Like it's it's every brand rule, every legal disclaimer. s209 00:15:38.240 --> 00:15:47.120 Uh a year ago, you'd have had to shard that across a dozen specialized agents and hope that your specialized agents are hand are are handing off to each each other cleanly. s210 00:15:47.120 --> 00:15:48.880 But now you can ignore that. s211 00:15:48.880 --> 00:15:51.120 Um but the third thing is the big one. s212 00:15:51.120 --> 00:15:53.600 The question used to be can the model even do this? s213 00:15:53.600 --> 00:15:55.839 And the answer is now firmly yes. s214 00:15:55.839 --> 00:15:57.680 Well reasonably firmly. s215 00:15:57.680 --> 00:16:06.560 Uh is it worth the cost is the new question because you can include 10,000 words of of sorry 10,000 different instructions into your prompt. s216 00:16:06.560 --> 00:16:08.160 But that is going to be an enormous prompt. s217 00:16:08.160 --> 00:16:09.600 It's going to be a very expensive prompt. s218 00:16:09.600 --> 00:16:11.199 It's going to be a very slow prompt. s219 00:16:11.199 --> 00:16:21.759 So what used to be a hard wall that you would run against has now become a soft trade-off of is it worth me adding all of these extra instructions if it's going to give me more cost and more latency. s220 00:16:21.839 --> 00:16:26.560 Uh and now some caveats uh to head off the Q&A. s221 00:16:26.560 --> 00:16:36.720 Um first and important first and most important I mentioned this earlier this is a proxy task including random words uh in a in a fake business report um is evidence s222 00:16:36.720 --> 00:16:38.399 that long skills file works. s223 00:16:38.399 --> 00:16:41.519 It is not the same as proof that a long skills file works. s224 00:16:41.519 --> 00:16:47.920 Um, also the models hit the wall at wildly different points anywhere from 750 to 9,000 plus. s225 00:16:47.920 --> 00:16:50.880 So you have to pick your model very carefully. s226 00:16:50.880 --> 00:16:58.560 Uh, what our test doesn't do is measure whether the model reasoned clearly over a giant prompt. s227 00:16:58.560 --> 00:17:04.880 So uh, the good news is since I did my research several weeks ago, uh, a whole bunch of people have piled in on this. s228 00:17:04.880 --> 00:17:17.280 Um and now there's good research uh actual scientists have got involved and done uh Chroma's has done context rot work uh across 18 models showing that accuracy on long inputs s229 00:17:17.280 --> 00:17:21.760 can fall 30 to 50% well before you hit the context window limit. s230 00:17:21.760 --> 00:17:33.520 Uh and the weird part of their finding was that uh coherent well ststructured text is more likely to hit that failure mode uh than if you just put your instructions into a random order and shuffle them in. s231 00:17:33.520 --> 00:17:36.160 Uh, I don't know why that's the case. s232 00:17:36.160 --> 00:17:37.679 I'd have to read their report. s233 00:17:37.679 --> 00:17:47.039 Um, so the model can track 2,000, 5,000, possibly 10,000 instructions, but it's not necessarily going to uh reason clearly over them. s234 00:17:47.039 --> 00:17:54.480 It's not necessarily if those if those instructions conflict, if there is tension between them, it's not necessarily going to get that right. s235 00:17:54.480 --> 00:17:57.760 Um, and then there's the other one I mentioned briefly. s236 00:17:57.760 --> 00:18:01.039 Uh, collude's refusals are annoying, but they are loud. s237 00:18:01.039 --> 00:18:02.880 You get an error, you know it failed. s238 00:18:02.880 --> 00:18:07.919 Uh GPT's polite half-finish report is much more dangerous because it looks like a real answer. s239 00:18:07.919 --> 00:18:14.000 Uh you have to read the whole thing to notice that it gave up quietly halfway, which means that you can't trust the output. s240 00:18:14.000 --> 00:18:18.880 It means you have to read the output every single time to make sure whether or not it's working. s241 00:18:18.880 --> 00:18:28.320 Uh so the model will accept your 2,00 rules and it will hand you back something that looks at least to begin with confident and polished but could be bailing out halfway through. s242 00:18:28.320 --> 00:18:34.640 Um, so, uh, as an aside, people always ask me, "How much did all this cost me?" s243 00:18:34.640 --> 00:18:37.440 It cost me $29 to run all of these queries. s244 00:18:37.440 --> 00:18:43.840 2,37 2,300 calls across seven models, uh, came to $29. s245 00:18:43.840 --> 00:18:46.640 Uh, it turns out novel research doesn't cost very much. s246 00:18:46.640 --> 00:18:55.120 Um, and this is the part of the talk where I was saying that you have to check this stuff in production because you can't trust that your model isn't going to s247 00:18:55.120 --> 00:18:56.400 silently fail. s248 00:18:56.400 --> 00:19:01.520 Uh, so you knew I was going to mention evals eventually because I work at Arise and this is where I do that. s249 00:19:01.520 --> 00:19:03.679 Um, but there are plenty of plugs for Arise. s250 00:19:03.679 --> 00:19:14.400 So I'm just going to say one true thing which is that if you are building a real AI application and you are giving it genuinely tricky tasks, you are going to run into one or more of these failure modes with a frontier model. s251 00:19:14.400 --> 00:19:23.520 Uh, and unless it's Claude telling you just to off at the API level, the only way to know that something went wrong is monitoring your outputs with another LLM. s252 00:19:23.520 --> 00:19:24.320 That is an eval. s253 00:19:24.320 --> 00:19:25.440 And that is what Arise does. s254 00:19:25.440 --> 00:19:27.600 And I'll leave it at that. s255 00:19:27.600 --> 00:19:31.120 Uh, I already mentioned that there's been new research since we did our own. s256 00:19:31.120 --> 00:19:32.160 Here's another important one. s257 00:19:32.160 --> 00:19:41.679 A paper landed testing 46 models called revisiting the reliability of language models in instruction falling, which you can bet made my ears perk up after I did that research myself. s258 00:19:41.679 --> 00:19:47.919 Uh, and they found something uncomfortable, which is that a model can ace a benchmark like ours and still be wildly unreliable. s259 00:19:47.919 --> 00:19:57.120 because if you reword the same instruction in a slightly different way, it can make a radical difference to how well uh it follows those instructions. s260 00:19:57.120 --> 00:20:00.640 So the model can follow 2,000 instructions and it can do it really well. s261 00:20:00.640 --> 00:20:09.200 But if you put the same instructions, the same 2,000 instructions in a different order, it can suddenly make the model much worse at following those instructions. s262 00:20:09.200 --> 00:20:12.160 And how ex how exactly to do that? s263 00:20:12.160 --> 00:20:20.240 what is the correct order of instructions to give your model such that it follows them perfectly as opposed to getting confused is still research that is being done. s264 00:20:20.240 --> 00:20:24.799 So capacity went up but reliability is still a problem. s265 00:20:24.799 --> 00:20:30.960 Um and then this is just a little brag because uh I was happy about it like I'm not a scientist. s266 00:20:30.960 --> 00:20:36.880 I did some research and then a whole bunch of other actual scientists piled in uh and did real science on the same question. s267 00:20:36.880 --> 00:20:40.880 There's now a whole bunch of benchmarks that have shown up uh to measure this same question. s268 00:20:40.880 --> 00:20:44.799 Firebench, CCR bench, Guidebench uh are all trying to measure the same thing. s269 00:20:44.799 --> 00:20:49.039 How well models follow a lot of real messy constraints at once. s270 00:20:49.039 --> 00:20:51.039 Uh and now the whole field is looking at it. s271 00:20:51.039 --> 00:20:58.480 So if you want better science than my, you know, 10,000 random words, uh the real science exists now. s272 00:20:58.480 --> 00:21:00.720 Uh so that gets me to where I will leave you. s273 00:21:00.720 --> 00:21:05.520 A year ago, the hard part of writing a skill was fitting everything in without the model losing the plot. s274 00:21:05.520 --> 00:21:09.039 That was a compression problem, and the compression problem is gone. s275 00:21:09.039 --> 00:21:12.159 uh the model will hold your 2,000 instructions just fine. s276 00:21:12.159 --> 00:21:17.280 The new hard part is knowing whether it actually did what you said and that is a verification problem. s277 00:21:17.280 --> 00:21:20.720 Uh a verification problem doesn't get solved by writing a better prompt. s278 00:21:20.720 --> 00:21:27.120 It gets solved by checking the output every time uh the same way that you would test any other code, which is to say an eval. s279 00:21:27.120 --> 00:21:29.679 The ceiling moved by 10x in one year. s280 00:21:29.679 --> 00:21:38.080 Uh so go back and check the assumptions that you made six months ago about how big your prompts should be, how big your uh instructions can get. s281 00:21:38.080 --> 00:21:39.679 uh because they might already be wrong. s282 00:21:39.679 --> 00:21:40.080 Boom. s283 00:21:40.080 --> 00:21:41.600 Be wrong. s284 00:21:41.600 --> 00:21:42.720 So that is the talk. s285 00:21:42.720 --> 00:21:47.600 If you want uh all of the code and all of the data, uh it is at this GitHub URL. s286 00:21:47.600 --> 00:21:52.240 Uh and this other QR code is uh something marketing made me insert. s287 00:21:52.240 --> 00:21:56.799 We are having a World Cup watch party tonight at 5:00 p.m. Uh you can come to our party. s288 00:21:56.799 --> 00:22:01.360 That link is to the Luma that will get you into the get into get you into the party. s289 00:22:01.360 --> 00:22:06.240 Uh I hope this talk has given you some novel information or at least a couple of laughs. s290 00:22:06.240 --> 00:22:09.919 And thank you so much for your time and attention.