How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival
https://www.youtube.com/watch?v=8g1N_a3h9H0
AI Engineer · 11 min · 130 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:00] Hey everyone, I'm Alvin, the head of agent intelligence at OutReval.
[00:05] By now, everyone's heard of token maxing.
[00:08] People have built leaderboards inside companies for who's using the most AI tokens.
[00:12] At Meta, they even had a title for this, tooken legend, a badge of honor for whoever's burning the most compute.
[00:20] It's a funny meme, but if you're actually shipping a product, token usage is a bizarre measurement of success.
[00:26] Tokens aren't the work.
[00:28] They're what you pay to get the work done.
[00:31] At OutRrival, our digital workers handle complex tasks over phone, text, and email.
[00:37] At the application layer, we're processing billions of tokens every single day.
[00:41] And when you operate at that scale, every redundant model call compounds into massive latency and millions of dollars in compute costs.
[00:50] Take a look at this prompt.
[00:51] I've built a synthetic proxy that mirrors what structural bloat looks like in a pipeline.
[00:56] Let's pretend we're classifying customer intents for a fictional company called HarborBite.
[01:01] We've all seen one of these.
[01:03] It's got 20 edge cases, overly pedantic definitions, and a paragraph practically begging the model not to hallucinate.
[01:11] When you're sending this much text on every single request, millions of times a day, it isn't just a prompt anymore.
[01:17] It's a massive self-inflicted compute tax.
[01:20] If you want to give your agents more capabilities, you need the architectural headroom to do it.
[01:26] So today, we're reclaiming token maxing.
[01:29] It shouldn't be about burning the most tokens, but about extracting the maximum surface area of intelligence from the absolute minimum token payload.
[01:37] So here's how to actually token max.
[01:41] Let's get into the playbook.
[01:43] First, observe.
[01:45] You can't fix what you can't measure.
[01:47] Instrument your shared model client so each attempt becomes part of a trace connecting the model calls, tools, and retries for a single task.
[01:57] Record the prompt version, model settings, token usage, latency, and cost.
[02:04] Open telemetry is a great framework for tracking this.
[02:09] Now that you have that telemetry, you can actually build a benchmark of how production is behaving.
[02:15] But here's where most optimization efforts hit a wall.
[02:19] Let me tell you how it usually goes.
[02:21] You take your massive prompt, you hand it to a smaller, cheaper model, and you run it against a few thousand of those historical production examples.
[02:31] We did this.
[02:32] Our cheaper candidate model disagreed with our expensive production baseline about 18% of the time.
[02:40] So naturally, we threw it out.
[02:42] It wasn't good enough.
[02:44] But what does that 18% actually represent?
[02:49] Because language models in production are dealing with inherently ambiguous real life inputs, that 18% is actually a composite metric.
[02:59] It introduces a critical hidden variable, the baseline variance of the model itself.
[03:06] When the input is messy, your production prompt will inevitably disagree with its own past decisions.
[03:14] So to isolate that hidden variable and understand what your rewrite actually changed, we asked a really dumb question.
[03:23] Does prod even agree with prod?
[03:26] We ran an AA test.
[03:28] We call this the ceiling test.
[03:31] We took our massive expensive production model and ran it three separate times on the same inputs and compared the answers.
[03:40] We discovered that it disagreed with itself roughly 7% of the time.
[03:45] Production's own agreement was only about 93%.
[03:50] So we were demanding a standard of perfection from the cheap candidate that our existing expensive system couldn't even hit.
[03:58] Once you establish that ceiling test benchmark, you finally have a measured baseline.
[04:04] Now you can start making the work cheaper.
[04:07] There are three technical levers I like to pull and they run in a sequence.
[04:12] The very first is caching.
[04:15] Specifically, structuring your prompts to maximize your prefix cache rate is a really easy and really effective way to bring down the cost to run given tasks.
[04:27] The second lever is to actually shrink the footprint of the prompt itself.
[04:32] So, let's bring that Harborbite prompt example back up.
[04:37] This prompt is almost 5,000 tokens long.
[04:40] And why?
[04:41] Because over time, developers kept bolting on paragraphs of rules.
[04:47] For example, do not confuse enthusiasm with authorization is one such example.
[04:54] Uh, plus massive hard-coded transcripts that were acting as worked examples.
[05:01] Nobody wants to be the one who deletes a line and breaks prod.
[05:05] So what do you do?
[05:07] You automate it.
[05:08] You turn prompting into a search problem.
[05:11] With a framework like DSPI, you can actually stop manually writing instructions.
[05:18] You define the exact interface contract, the inputs, the required output schema, and the scoring function.
[05:26] And then you hand it off to an optimizer like Jeepa.
[05:30] The optimizer treats your prompt as a search space.
[05:33] It runs against your benchmark attempting to match those historical labels.
[05:38] And it uses feedback on failures to revise the instructions.
[05:43] It stripped out almost all of our long worked examples and replaced them with compact rules.
[05:50] A 51% reduction in instruction tokens with the same application contract.
[05:56] Now we can move to test whether quality holds.
[05:59] That is instant margin.
[06:01] The third lever is to actually downsize the model.
[06:06] Now that you have a tight prompt and a measurable search loop, you don't always need a massive frontier model.
[06:14] You can start aggressively cutting compute.
[06:17] There are two interesting strategies here.
[06:20] First, you can break a massive prompt into smaller components.
[06:25] you use a cheaper model to handle something like initial routing or data extraction and only wake up your expensive frontier models for heavy reasoning.
[06:35] Second, you can use that exact same DSPI search loop we just built, but plug a smaller model into it.
[06:42] You let the smaller model search its own prompt space to match the benchmark answers generated by your large production model.
[06:50] You're essentially using Jeepa to teach a tiny model to match the task performance of a massive one for your specific data set.
[06:58] But the reality of optimization is not quite so easy when it comes to these small models.
[07:04] Um if you're trying to compress the behavior of a large model into a smaller one, you'll notice that you introduce new points of failure and overfitting.
[07:14] Um, and in many of our downsizing experiments, uh, we found that the prompts weren't good enough to release, which brings us to the final hurdle.
[07:25] We've cut Harbor's prompt in half, but now we have to prove that we actually didn't break the product.
[07:32] Remember, these models are probabilistic.
[07:36] Production doesn't always agree with itself.
[07:38] And likewise, our new cheaper candidate is also going to have its own internal variance.
[07:44] So you run a massive shadow deployment on the exact same inputs and measure the system from three angles.
[07:52] Production against production, candidate against candidate, and candidate against production.
[07:58] There's a useful bit of math here.
[08:01] You take the disagreement between the two versions and subtract the average disagreement inside each one.
[08:08] What's left gives you a measure of how different their decision probabilities are.
[08:13] This is where that ceiling test pays off.
[08:16] We aren't guessing.
[08:17] We have a mathematical baseline for how much these answers move around when we haven't changed a thing.
[08:25] And here's the secret about that drift.
[08:27] Different doesn't automatically mean wrong.
[08:30] If your optimizer did his job, the distributions should be different because the cheaper prompt is actually doing a better job.
[08:38] We aren't necessarily trying to build a perfect clone of a production system that may already have some flaws baked into it.
[08:46] So, you stop panicking about the raw differences, you pull the diffs, and you manually review some of them.
[08:52] Was the candidate better?
[08:54] Was production better?
[08:55] or were they just two different equally acceptable ways to handle an ambiguous input?
[09:01] For Harbor Bite, let's say the completed review confirmed 443 regressions, but it also found 2,463 outright improvements.
[09:11] That is almost six improvements for every one regression from a prompt that is half the size.
[09:18] But you do not ship on a point estimate.
[09:21] You have to account for sampling noise.
[09:24] We measured a massive gain on this specific batch of requests, but we need to know exactly how much statistical confidence we can put behind it.
[09:33] So, we calculate a confidence interval around that net gain.
[09:37] You keep the comparison strictly paired, both versions answering the exact same request, and you keep the repeated runs of the same conversations grouped together.
[09:46] To say the candidate is better, we want the lower confidence bound above zero.
[09:51] If the interval crosses zero, we need more evidence before making that claim.
[09:56] And one final sanity check is that you have to run this math on your critical categories separately.
[10:02] A great global average can actually uh silently hide that one vital customer intent is tanking.
[10:10] If your critical categories hold up and your lower confidence bound is above zero, you have your answer.
[10:17] Ship it and keep measuring.
[10:19] So this process is how you can replace the guesswork of prompt engineering with an explicit release rule.
[10:27] Cache the obvious stuff.
[10:29] Let an optimizer shrink your instructions.
[10:32] Rightize your model and use the ceiling test to understand the variation.
[10:37] Then test whether quality holds.
[10:40] So here's your homework for Monday morning.
[10:42] Go pick your most expensive task.
[10:44] Measure your baseline variance.
[10:46] put it through a search loop and find out exactly how much bloat you're paying for.
[10:51] We need to reclaim token maxing.
[10:54] Let the rest of the industry fight for the high score on the leaderboard.
[10:58] The real token legends are the ones making every single token count.