How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival

AI Engineer · 11 min · 130 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:00Hey everyone, I'm Alvin, the head of agent intelligence at OutReval.
2. 00:05By now, everyone's heard of token maxing.
3. 00:08People have built leaderboards inside companies for who's using the most AI tokens.
4. 00:12At Meta, they even had a title for this, tooken legend, a badge of honor for whoever's burning the most compute.
5. 00:20It's a funny meme, but if you're actually shipping a product, token usage is a bizarre measurement of success.
6. 00:26Tokens aren't the work.
7. 00:28They're what you pay to get the work done.
8. 00:31At OutRrival, our digital workers handle complex tasks over phone, text, and email.
9. 00:37At the application layer, we're processing billions of tokens every single day.
10. 00:41And when you operate at that scale, every redundant model call compounds into massive latency and millions of dollars in compute costs.
11. 00:50Take a look at this prompt.
12. 00:51I've built a synthetic proxy that mirrors what structural bloat looks like in a pipeline.
13. 00:56Let's pretend we're classifying customer intents for a fictional company called HarborBite.
14. 01:01We've all seen one of these.
15. 01:03It's got 20 edge cases, overly pedantic definitions, and a paragraph practically begging the model not to hallucinate.
16. 01:11When you're sending this much text on every single request, millions of times a day, it isn't just a prompt anymore.
17. 01:17It's a massive self-inflicted compute tax.
18. 01:20If you want to give your agents more capabilities, you need the architectural headroom to do it.
19. 01:26So today, we're reclaiming token maxing.
20. 01:29It shouldn't be about burning the most tokens, but about extracting the maximum surface area of intelligence from the absolute minimum token payload.
21. 01:37So here's how to actually token max.
22. 01:41Let's get into the playbook.
23. 01:43First, observe.
24. 01:45You can't fix what you can't measure.
25. 01:47Instrument your shared model client so each attempt becomes part of a trace connecting the model calls, tools, and retries for a single task.
26. 01:57Record the prompt version, model settings, token usage, latency, and cost.
27. 02:04Open telemetry is a great framework for tracking this.
28. 02:09Now that you have that telemetry, you can actually build a benchmark of how production is behaving.
29. 02:15But here's where most optimization efforts hit a wall.
30. 02:19Let me tell you how it usually goes.
31. 02:21You take your massive prompt, you hand it to a smaller, cheaper model, and you run it against a few thousand of those historical production examples.
32. 02:31We did this.
33. 02:32Our cheaper candidate model disagreed with our expensive production baseline about 18% of the time.
34. 02:40So naturally, we threw it out.
35. 02:42It wasn't good enough.
36. 02:44But what does that 18% actually represent?
37. 02:49Because language models in production are dealing with inherently ambiguous real life inputs, that 18% is actually a composite metric.
38. 02:59It introduces a critical hidden variable, the baseline variance of the model itself.
39. 03:06When the input is messy, your production prompt will inevitably disagree with its own past decisions.
40. 03:14So to isolate that hidden variable and understand what your rewrite actually changed, we asked a really dumb question.
41. 03:23Does prod even agree with prod?
42. 03:26We ran an AA test.
43. 03:28We call this the ceiling test.
44. 03:31We took our massive expensive production model and ran it three separate times on the same inputs and compared the answers.
45. 03:40We discovered that it disagreed with itself roughly 7% of the time.
46. 03:45Production's own agreement was only about 93%.
47. 03:50So we were demanding a standard of perfection from the cheap candidate that our existing expensive system couldn't even hit.
48. 03:58Once you establish that ceiling test benchmark, you finally have a measured baseline.
49. 04:04Now you can start making the work cheaper.
50. 04:07There are three technical levers I like to pull and they run in a sequence.
51. 04:12The very first is caching.
52. 04:15Specifically, structuring your prompts to maximize your prefix cache rate is a really easy and really effective way to bring down the cost to run given tasks.
53. 04:27The second lever is to actually shrink the footprint of the prompt itself.
54. 04:32So, let's bring that Harborbite prompt example back up.
55. 04:37This prompt is almost 5,000 tokens long.
56. 04:40And why?
57. 04:41Because over time, developers kept bolting on paragraphs of rules.
58. 04:47For example, do not confuse enthusiasm with authorization is one such example.
59. 04:54Uh, plus massive hard-coded transcripts that were acting as worked examples.
60. 05:01Nobody wants to be the one who deletes a line and breaks prod.
61. 05:05So what do you do?
62. 05:07You automate it.
63. 05:08You turn prompting into a search problem.
64. 05:11With a framework like DSPI, you can actually stop manually writing instructions.
65. 05:18You define the exact interface contract, the inputs, the required output schema, and the scoring function.
66. 05:26And then you hand it off to an optimizer like Jeepa.
67. 05:30The optimizer treats your prompt as a search space.
68. 05:33It runs against your benchmark attempting to match those historical labels.
69. 05:38And it uses feedback on failures to revise the instructions.
70. 05:43It stripped out almost all of our long worked examples and replaced them with compact rules.
71. 05:50A 51% reduction in instruction tokens with the same application contract.
72. 05:56Now we can move to test whether quality holds.
73. 05:59That is instant margin.
74. 06:01The third lever is to actually downsize the model.
75. 06:06Now that you have a tight prompt and a measurable search loop, you don't always need a massive frontier model.
76. 06:14You can start aggressively cutting compute.
77. 06:17There are two interesting strategies here.
78. 06:20First, you can break a massive prompt into smaller components.
79. 06:25you use a cheaper model to handle something like initial routing or data extraction and only wake up your expensive frontier models for heavy reasoning.
80. 06:35Second, you can use that exact same DSPI search loop we just built, but plug a smaller model into it.
81. 06:42You let the smaller model search its own prompt space to match the benchmark answers generated by your large production model.
82. 06:50You're essentially using Jeepa to teach a tiny model to match the task performance of a massive one for your specific data set.
83. 06:58But the reality of optimization is not quite so easy when it comes to these small models.
84. 07:04Um if you're trying to compress the behavior of a large model into a smaller one, you'll notice that you introduce new points of failure and overfitting.
85. 07:14Um, and in many of our downsizing experiments, uh, we found that the prompts weren't good enough to release, which brings us to the final hurdle.
86. 07:25We've cut Harbor's prompt in half, but now we have to prove that we actually didn't break the product.
87. 07:32Remember, these models are probabilistic.
88. 07:36Production doesn't always agree with itself.
89. 07:38And likewise, our new cheaper candidate is also going to have its own internal variance.
90. 07:44So you run a massive shadow deployment on the exact same inputs and measure the system from three angles.
91. 07:52Production against production, candidate against candidate, and candidate against production.
92. 07:58There's a useful bit of math here.
93. 08:01You take the disagreement between the two versions and subtract the average disagreement inside each one.
94. 08:08What's left gives you a measure of how different their decision probabilities are.
95. 08:13This is where that ceiling test pays off.
96. 08:16We aren't guessing.
97. 08:17We have a mathematical baseline for how much these answers move around when we haven't changed a thing.
98. 08:25And here's the secret about that drift.
99. 08:27Different doesn't automatically mean wrong.
100. 08:30If your optimizer did his job, the distributions should be different because the cheaper prompt is actually doing a better job.
101. 08:38We aren't necessarily trying to build a perfect clone of a production system that may already have some flaws baked into it.
102. 08:46So, you stop panicking about the raw differences, you pull the diffs, and you manually review some of them.
103. 08:52Was the candidate better?
104. 08:54Was production better?
105. 08:55or were they just two different equally acceptable ways to handle an ambiguous input?
106. 09:01For Harbor Bite, let's say the completed review confirmed 443 regressions, but it also found 2,463 outright improvements.
107. 09:11That is almost six improvements for every one regression from a prompt that is half the size.
108. 09:18But you do not ship on a point estimate.
109. 09:21You have to account for sampling noise.
110. 09:24We measured a massive gain on this specific batch of requests, but we need to know exactly how much statistical confidence we can put behind it.
111. 09:33So, we calculate a confidence interval around that net gain.
112. 09:37You keep the comparison strictly paired, both versions answering the exact same request, and you keep the repeated runs of the same conversations grouped together.
113. 09:46To say the candidate is better, we want the lower confidence bound above zero.
114. 09:51If the interval crosses zero, we need more evidence before making that claim.
115. 09:56And one final sanity check is that you have to run this math on your critical categories separately.
116. 10:02A great global average can actually uh silently hide that one vital customer intent is tanking.
117. 10:10If your critical categories hold up and your lower confidence bound is above zero, you have your answer.
118. 10:17Ship it and keep measuring.
119. 10:19So this process is how you can replace the guesswork of prompt engineering with an explicit release rule.
120. 10:27Cache the obvious stuff.
121. 10:29Let an optimizer shrink your instructions.
122. 10:32Rightize your model and use the ceiling test to understand the variation.
123. 10:37Then test whether quality holds.
124. 10:40So here's your homework for Monday morning.
125. 10:42Go pick your most expensive task.
126. 10:44Measure your baseline variance.
127. 10:46put it through a search loop and find out exactly how much bloat you're paying for.
128. 10:51We need to reclaim token maxing.
129. 10:54Let the rest of the industry fight for the high score on the leaderboard.
130. 10:58The real token legends are the ones making every single token count.