WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=8g1N_a3h9H0 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/8g1N_a3h9H0.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.719 --> 00:00:05.520 Hey everyone, I'm Alvin, the head of agent intelligence at OutReval. s2 00:00:05.520 --> 00:00:08.320 By now, everyone's heard of token maxing. s3 00:00:08.320 --> 00:00:12.880 People have built leaderboards inside companies for who's using the most AI tokens. s4 00:00:12.880 --> 00:00:20.400 At Meta, they even had a title for this, tooken legend, a badge of honor for whoever's burning the most compute. s5 00:00:20.400 --> 00:00:26.960 It's a funny meme, but if you're actually shipping a product, token usage is a bizarre measurement of success. s6 00:00:26.960 --> 00:00:28.560 Tokens aren't the work. s7 00:00:28.560 --> 00:00:31.199 They're what you pay to get the work done. s8 00:00:31.199 --> 00:00:37.120 At OutRrival, our digital workers handle complex tasks over phone, text, and email. s9 00:00:37.120 --> 00:00:41.520 At the application layer, we're processing billions of tokens every single day. s10 00:00:41.520 --> 00:00:50.000 And when you operate at that scale, every redundant model call compounds into massive latency and millions of dollars in compute costs. s11 00:00:50.000 --> 00:00:51.440 Take a look at this prompt. s12 00:00:51.440 --> 00:00:56.800 I've built a synthetic proxy that mirrors what structural bloat looks like in a pipeline. s13 00:00:56.800 --> 00:01:01.760 Let's pretend we're classifying customer intents for a fictional company called HarborBite. s14 00:01:01.760 --> 00:01:03.520 We've all seen one of these. s15 00:01:03.520 --> 00:01:11.119 It's got 20 edge cases, overly pedantic definitions, and a paragraph practically begging the model not to hallucinate. s16 00:01:11.119 --> 00:01:17.520 When you're sending this much text on every single request, millions of times a day, it isn't just a prompt anymore. s17 00:01:17.520 --> 00:01:20.799 It's a massive self-inflicted compute tax. s18 00:01:20.799 --> 00:01:26.080 If you want to give your agents more capabilities, you need the architectural headroom to do it. s19 00:01:26.080 --> 00:01:29.040 So today, we're reclaiming token maxing. s20 00:01:29.040 --> 00:01:37.840 It shouldn't be about burning the most tokens, but about extracting the maximum surface area of intelligence from the absolute minimum token payload. s21 00:01:37.840 --> 00:01:41.040 So here's how to actually token max. s22 00:01:41.040 --> 00:01:43.439 Let's get into the playbook. s23 00:01:43.439 --> 00:01:45.119 First, observe. s24 00:01:45.119 --> 00:01:47.920 You can't fix what you can't measure. s25 00:01:47.920 --> 00:01:57.680 Instrument your shared model client so each attempt becomes part of a trace connecting the model calls, tools, and retries for a single task. s26 00:01:57.680 --> 00:02:04.560 Record the prompt version, model settings, token usage, latency, and cost. s27 00:02:04.560 --> 00:02:09.200 Open telemetry is a great framework for tracking this. s28 00:02:09.200 --> 00:02:15.120 Now that you have that telemetry, you can actually build a benchmark of how production is behaving. s29 00:02:15.120 --> 00:02:19.120 But here's where most optimization efforts hit a wall. s30 00:02:19.120 --> 00:02:21.360 Let me tell you how it usually goes. s31 00:02:21.360 --> 00:02:31.440 You take your massive prompt, you hand it to a smaller, cheaper model, and you run it against a few thousand of those historical production examples. s32 00:02:31.520 --> 00:02:32.879 We did this. s33 00:02:32.879 --> 00:02:40.080 Our cheaper candidate model disagreed with our expensive production baseline about 18% of the time. s34 00:02:40.080 --> 00:02:42.640 So naturally, we threw it out. s35 00:02:42.640 --> 00:02:44.400 It wasn't good enough. s36 00:02:44.400 --> 00:02:49.120 But what does that 18% actually represent? s37 00:02:49.120 --> 00:02:59.519 Because language models in production are dealing with inherently ambiguous real life inputs, that 18% is actually a composite metric. s38 00:02:59.519 --> 00:03:06.319 It introduces a critical hidden variable, the baseline variance of the model itself. s39 00:03:06.319 --> 00:03:14.640 When the input is messy, your production prompt will inevitably disagree with its own past decisions. s40 00:03:14.959 --> 00:03:23.840 So to isolate that hidden variable and understand what your rewrite actually changed, we asked a really dumb question. s41 00:03:23.840 --> 00:03:26.720 Does prod even agree with prod? s42 00:03:26.720 --> 00:03:28.959 We ran an AA test. s43 00:03:28.959 --> 00:03:31.280 We call this the ceiling test. s44 00:03:31.280 --> 00:03:40.560 We took our massive expensive production model and ran it three separate times on the same inputs and compared the answers. s45 00:03:40.560 --> 00:03:45.360 We discovered that it disagreed with itself roughly 7% of the time. s46 00:03:45.360 --> 00:03:50.080 Production's own agreement was only about 93%. s47 00:03:50.080 --> 00:03:58.959 So we were demanding a standard of perfection from the cheap candidate that our existing expensive system couldn't even hit. s48 00:03:58.959 --> 00:04:04.319 Once you establish that ceiling test benchmark, you finally have a measured baseline. s49 00:04:04.319 --> 00:04:07.360 Now you can start making the work cheaper. s50 00:04:07.360 --> 00:04:12.720 There are three technical levers I like to pull and they run in a sequence. s51 00:04:12.720 --> 00:04:15.760 The very first is caching. s52 00:04:15.760 --> 00:04:26.639 Specifically, structuring your prompts to maximize your prefix cache rate is a really easy and really effective way to bring down the cost to run given tasks. s53 00:04:27.440 --> 00:04:32.800 The second lever is to actually shrink the footprint of the prompt itself. s54 00:04:32.800 --> 00:04:37.199 So, let's bring that Harborbite prompt example back up. s55 00:04:37.199 --> 00:04:40.479 This prompt is almost 5,000 tokens long. s56 00:04:40.479 --> 00:04:41.759 And why? s57 00:04:41.759 --> 00:04:47.360 Because over time, developers kept bolting on paragraphs of rules. s58 00:04:47.360 --> 00:04:54.479 For example, do not confuse enthusiasm with authorization is one such example. s59 00:04:54.560 --> 00:05:01.199 Uh, plus massive hard-coded transcripts that were acting as worked examples. s60 00:05:01.199 --> 00:05:05.360 Nobody wants to be the one who deletes a line and breaks prod. s61 00:05:05.360 --> 00:05:07.039 So what do you do? s62 00:05:07.039 --> 00:05:08.400 You automate it. s63 00:05:08.400 --> 00:05:11.919 You turn prompting into a search problem. s64 00:05:11.919 --> 00:05:18.320 With a framework like DSPI, you can actually stop manually writing instructions. s65 00:05:18.320 --> 00:05:26.000 You define the exact interface contract, the inputs, the required output schema, and the scoring function. s66 00:05:26.000 --> 00:05:30.400 And then you hand it off to an optimizer like Jeepa. s67 00:05:30.400 --> 00:05:33.919 The optimizer treats your prompt as a search space. s68 00:05:33.919 --> 00:05:38.960 It runs against your benchmark attempting to match those historical labels. s69 00:05:38.960 --> 00:05:43.440 And it uses feedback on failures to revise the instructions. s70 00:05:43.440 --> 00:05:50.080 It stripped out almost all of our long worked examples and replaced them with compact rules. s71 00:05:50.080 --> 00:05:56.479 A 51% reduction in instruction tokens with the same application contract. s72 00:05:56.479 --> 00:05:59.759 Now we can move to test whether quality holds. s73 00:05:59.759 --> 00:06:01.919 That is instant margin. s74 00:06:01.919 --> 00:06:06.800 The third lever is to actually downsize the model. s75 00:06:06.800 --> 00:06:14.319 Now that you have a tight prompt and a measurable search loop, you don't always need a massive frontier model. s76 00:06:14.319 --> 00:06:17.520 You can start aggressively cutting compute. s77 00:06:17.520 --> 00:06:20.400 There are two interesting strategies here. s78 00:06:20.400 --> 00:06:25.199 First, you can break a massive prompt into smaller components. s79 00:06:25.199 --> 00:06:35.039 you use a cheaper model to handle something like initial routing or data extraction and only wake up your expensive frontier models for heavy reasoning. s80 00:06:35.039 --> 00:06:42.400 Second, you can use that exact same DSPI search loop we just built, but plug a smaller model into it. s81 00:06:42.400 --> 00:06:50.080 You let the smaller model search its own prompt space to match the benchmark answers generated by your large production model. s82 00:06:50.080 --> 00:06:58.880 You're essentially using Jeepa to teach a tiny model to match the task performance of a massive one for your specific data set. s83 00:06:58.880 --> 00:07:04.639 But the reality of optimization is not quite so easy when it comes to these small models. s84 00:07:04.639 --> 00:07:14.240 Um if you're trying to compress the behavior of a large model into a smaller one, you'll notice that you introduce new points of failure and overfitting. s85 00:07:14.240 --> 00:07:25.759 Um, and in many of our downsizing experiments, uh, we found that the prompts weren't good enough to release, which brings us to the final hurdle. s86 00:07:25.759 --> 00:07:32.560 We've cut Harbor's prompt in half, but now we have to prove that we actually didn't break the product. s87 00:07:32.560 --> 00:07:36.160 Remember, these models are probabilistic. s88 00:07:36.160 --> 00:07:38.880 Production doesn't always agree with itself. s89 00:07:38.880 --> 00:07:44.960 And likewise, our new cheaper candidate is also going to have its own internal variance. s90 00:07:44.960 --> 00:07:52.319 So you run a massive shadow deployment on the exact same inputs and measure the system from three angles. s91 00:07:52.319 --> 00:07:58.879 Production against production, candidate against candidate, and candidate against production. s92 00:07:58.879 --> 00:08:01.120 There's a useful bit of math here. s93 00:08:01.120 --> 00:08:08.000 You take the disagreement between the two versions and subtract the average disagreement inside each one. s94 00:08:08.000 --> 00:08:13.840 What's left gives you a measure of how different their decision probabilities are. s95 00:08:13.840 --> 00:08:16.479 This is where that ceiling test pays off. s96 00:08:16.479 --> 00:08:17.919 We aren't guessing. s97 00:08:17.919 --> 00:08:25.280 We have a mathematical baseline for how much these answers move around when we haven't changed a thing. s98 00:08:25.280 --> 00:08:27.360 And here's the secret about that drift. s99 00:08:27.360 --> 00:08:30.240 Different doesn't automatically mean wrong. s100 00:08:30.240 --> 00:08:38.000 If your optimizer did his job, the distributions should be different because the cheaper prompt is actually doing a better job. s101 00:08:38.000 --> 00:08:46.399 We aren't necessarily trying to build a perfect clone of a production system that may already have some flaws baked into it. s102 00:08:46.399 --> 00:08:52.640 So, you stop panicking about the raw differences, you pull the diffs, and you manually review some of them. s103 00:08:52.640 --> 00:08:54.160 Was the candidate better? s104 00:08:54.160 --> 00:08:55.440 Was production better? s105 00:08:55.440 --> 00:09:01.040 or were they just two different equally acceptable ways to handle an ambiguous input? s106 00:09:01.040 --> 00:09:11.279 For Harbor Bite, let's say the completed review confirmed 443 regressions, but it also found 2,463 outright improvements. s107 00:09:11.279 --> 00:09:18.560 That is almost six improvements for every one regression from a prompt that is half the size. s108 00:09:18.880 --> 00:09:21.600 But you do not ship on a point estimate. s109 00:09:21.600 --> 00:09:24.160 You have to account for sampling noise. s110 00:09:24.160 --> 00:09:33.839 We measured a massive gain on this specific batch of requests, but we need to know exactly how much statistical confidence we can put behind it. s111 00:09:33.839 --> 00:09:37.440 So, we calculate a confidence interval around that net gain. s112 00:09:37.440 --> 00:09:46.560 You keep the comparison strictly paired, both versions answering the exact same request, and you keep the repeated runs of the same conversations grouped together. s113 00:09:46.560 --> 00:09:51.519 To say the candidate is better, we want the lower confidence bound above zero. s114 00:09:51.519 --> 00:09:56.640 If the interval crosses zero, we need more evidence before making that claim. s115 00:09:56.640 --> 00:10:02.640 And one final sanity check is that you have to run this math on your critical categories separately. s116 00:10:02.640 --> 00:10:10.959 A great global average can actually uh silently hide that one vital customer intent is tanking. s117 00:10:10.959 --> 00:10:17.360 If your critical categories hold up and your lower confidence bound is above zero, you have your answer. s118 00:10:17.360 --> 00:10:19.680 Ship it and keep measuring. s119 00:10:19.680 --> 00:10:27.519 So this process is how you can replace the guesswork of prompt engineering with an explicit release rule. s120 00:10:27.519 --> 00:10:29.360 Cache the obvious stuff. s121 00:10:29.360 --> 00:10:32.160 Let an optimizer shrink your instructions. s122 00:10:32.160 --> 00:10:37.200 Rightize your model and use the ceiling test to understand the variation. s123 00:10:37.200 --> 00:10:40.240 Then test whether quality holds. s124 00:10:40.240 --> 00:10:42.399 So here's your homework for Monday morning. s125 00:10:42.399 --> 00:10:44.800 Go pick your most expensive task. s126 00:10:44.800 --> 00:10:46.560 Measure your baseline variance. s127 00:10:46.560 --> 00:10:51.680 put it through a search loop and find out exactly how much bloat you're paying for. s128 00:10:51.680 --> 00:10:54.800 We need to reclaim token maxing. s129 00:10:54.800 --> 00:10:58.800 Let the rest of the industry fight for the high score on the leaderboard. s130 00:10:58.800 --> 00:11:04.240 The real token legends are the ones making every single token count.