Tokens Are the Cheap Part — Steve Jamison, Jellyfish
https://www.youtube.com/watch?v=U7ZYBCcdJPg
AI Engineer · 17 min · 198 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:02] Hi, I'm Steve Jameson, AI research and engineering leader at Jellyfish.
[00:07] I'm here to show you why tokens are the cheap part.
[00:11] The research that I'll present is drawn from the Jellyfish platform.
[00:14] The industry's largest set of engineering workflow data joined AI telemetry data.
[00:21] It's no secret that AI costs are climbing.
[00:24] They're up 60x from this time last year.
[00:27] The use cases are wide and varied.
[00:29] The models and harnesses are getting much better.
[00:32] The implications are exciting.
[00:34] But discussions around budget and spend are not as much fun.
[00:38] These conversations involve your peers and senior leadership and the board.
[00:42] Not always the easiest audiences, especially when you don't have the right data.
[00:47] And it's real money these days.
[00:51] Output is up, too.
[00:53] Merged PRs per engineer are up 43% over the same one-year horizon.
[00:58] This is phenomenal.
[00:59] We've not seen productivity inputs like this, especially on this time scale.
[01:06] There's plenty of research and anecdotal data to know that AI and AI-driven coding are driving these throughput gains.
[01:13] Heck, even our charts show the correlation.
[01:15] Notice where the inflection points are.
[01:18] AI spend increased in early 2026 when throughput gains start a steady increase of their own.
[01:25] There's a tension with these two numbers, though.
[01:28] How do you reconcile them?
[01:30] Spend is up 60x.
[01:32] Throughput is also up 43%.
[01:34] Is that good?
[01:36] Bad?
[01:37] 43% is a big number.
[01:39] But so is 60x.
[01:41] How do you know if you're spending enough?
[01:45] Is there more productivity gain locked behind a checkbook?
[01:49] Should we increase spend 50%?
[01:52] 2x more?
[01:54] Or how do you know if you spent too much?
[01:57] I'm sure many of you have felt the tension here.
[02:00] The CFO or board are asking for the receipts of the spend.
[02:04] Can you show them?
[02:05] Well, merge PRs are up 43% satisfy them?
[02:10] Also, what happens if you cap use and spend?
[02:14] Are you stopping from setting tokens on fire?
[02:18] Or are you stopping further efficiency gains?
[02:21] You can ask peers at other companies what they're paying.
[02:24] For many, it's still a moving target though.
[02:27] But, how do you equate that to your org?
[02:30] Surely, just comparing their bill to yours is inadequate.
[02:34] We've heard that reports of We've heard the reports of companies spending 100k, 500k, even millions on AI.
[02:43] Setting then resetting their AI budgets as reality sets in.
[02:48] Ultimately, these two numbers can't tell the story.
[02:51] And even together, they miss the biggest line on the cost side.
[02:57] These two numbers leave out the biggest cost of all, developer salary.
[03:03] AI is making developers more productive on a scale we haven't seen before.
[03:07] It isn't just another tool, it's intelligence.
[03:11] This intelligence dramatically changes how much each dev can produce.
[03:15] That's why we need to think about its cost differently.
[03:18] Not off to the side as overhead, but it's part of what it takes to produce each PR.
[03:24] We're going to call this intelligence costs of a PR.
[03:32] What you pay your engineers plus what you pay for AI.
[03:36] Put them together in one cost bucket.
[03:40] AI is about 2 and 1/2% of the cost to produce another unit of merged code.
[03:46] Salary is the other 97 and 1/2%.
[03:50] This is in line with other reported figures, and this data is from August 2026.
[03:57] Here, tokens are the cheap part.
[04:01] But, that's current relative costs, not dollars.
[04:05] Let's look at what happened to the cost of a merged PR over the last year.
[04:15] [clears throat]
[04:16] The dollar cost of a merged PR fell 26% over the last year.
[04:21] Not PRs created, but PRs merged.
[04:24] Review time is factored in, and some level of quality gates has been cleared.
[04:30] What That decline in cost per merged PR includes a 60x increase in AI costs over the same time frame.
[04:37] The change is nearly universal, too.
[04:39] 83% of the companies we studied showed a drop in cost per PR.
[04:44] What does this tell us?
[04:46] The additional throughput gains are being spread or are [clears throat] spreading salary thinner across PRs, and this is happening faster than the AI costs are increasing.
[05:00] What's the range in token spend?
[05:02] We ranked companies by token usage, then grouped them into quintiles or fifths.
[05:09] The range here is quite large.
[05:11] The lightest group, about 0.3% of the cost of a PR is is AI.
[05:20] This represents about 60 million tokens per dev per month.
[05:26] The top 20% of AI spend per dev, we're going to call the heaviest spend group.
[05:32] There, 8 and 1/2% of the PR costs is AI.
[05:36] This represents about 2.5 billion tokens per dev per month.
[05:41] About two orders of magnitude more than the lightest group.
[05:47] We'll use these groups on the next slides to see how token spend influences other metrics.
[05:53] First, PR throughput measured as merged PRs per dev.
[05:58] Split by the same AI spend quintiles on the previous slide.
[06:03] The bottom 4/5 of spenders saw PR throughput gains of around 40% give or take.
[06:12] But the heavy spend group improved more at a 67% increase.
[06:17] More AI spend led to more throughput gains.
[06:24] Now let's look at the cost per PR.
[06:27] Positive results are seen again across the board.
[06:30] Cost per PR is dropping for all AI usage groups.
[06:35] However, again, the heaviest fifth so the the companies that spent the most improved the most.
[06:44] They saw a cost per PR decrease of 32% over the last year.
[06:50] The other four groups saw about a 25% decrease in cost per PR.
[06:58] Investing in spending more on AI fueled more of an improvement in cost per PR compared to peers who leaned in less.
[07:07] So we saw that the pop- same same as we saw in the population average, cost per PR changes are driven largely by PR throughput, not by AI spend.
[07:20] The heavy spenders experienced PR throughput gains of 67% compared to about 40-ish percent for their peers.
[07:28] Spreading the same salary across more PRs in the month has more of an effect than the rising AI costs because tokens are the cheap part.
[07:42] This is not to say that org should light tokens on fire and spend with abandon.
[07:48] Here we're looking at tokens used on the horizontal axis and cost per PR on the vertical axis.
[07:54] There's a steep drop at the beginning where relative small AI spend results in large gains initially.
[08:01] As token usage increases, the cost per PR improvement decreases.
[08:07] This is diminishing returns.
[08:09] I want to stress though, diminishing returns is not the same as no returns.
[08:16] We're seeing that the top 10% are spending 8x the tokens compared to the median and achieving a 27 25% lower cost per PR.
[08:31] There will be a time when tokens aren't the cheap part, but by and large for most companies out in the wild we're not close to that threshold.
[08:42] More tokens leads to more PRs, which leads to a lower cost of delivery.
[08:55] So far we've talked about companies.
[08:58] Next I'd like to take a tangent and talk about a study we did with developers.
[09:04] Here we looked at developer weeks with heavy token use compared to the same developer weeks with lower token use.
[09:15] [cough]
[09:16] [clears throat]
[09:16] Here we saw the same phenomenon just at the individual level.
[09:21] Compare each developers heavy agent skill weeks with their own weeks without skills.
[09:29] They spend about twice as much on tokens, merged 27% more PRs, and reach PR and each PR costs about 18% less once salary and tokens are counted together.
[09:49] There are two fair curiosities I get when I talk about this work.
[09:53] First, does faster work result in worse work?
[10:00] And second, are the more PRs the throughput gains that we're seeing, are they being driven by smaller chunks of work?
[10:12] First curiosity is cheaper, faster work, worse work?
[10:17] If AI makes PRs faster and cheaper, are we just shipping more mistakes?
[10:22] One proxy for this is river reverts.
[10:25] Merged PRs that undo an earlier PR.
[10:29] The heaviest fifth started lowest before most of the token growth, and revert rates drop slightly for every group.
[10:38] Though these deltas are just slightly below the noise in the data.
[10:42] It's not evidence of a improvement here.
[10:45] However, more importantly, we're not seeing any worrying signs of rampant or severe quality issues with higher token spends and faster PR throughput.
[11:02] My second curiosity is throughput driven by smaller PRs?
[11:09] And this is demonstrably no. Not smaller PRs.
[11:13] In fact, AI is driving an increase in line level lines of code generated per PR.
[11:21] This is no surprise to the devs in the room.
[11:25] Raise your hand if you've rolled your eyes at verbose, unnecessary inline AI comments, or a file full of mostly irrelevant tests.
[11:34] Gen AI Gen AI likes to generate code.
[11:38] Also, the bar to create is low for devs, and there's a pull to do more per PR instead of spinning up another ticket.
[11:46] It's just one, maybe two prompts, and I'll handle this edge case, or I can combine this and that element.
[11:52] Fewer PRs to wrangle.
[11:54] This doesn't mesh well with the PR review stress folks are feeling these days, though.
[11:59] There's
[12:01] [clears throat]
[12:02] Also, AI is just becoming more capable, so we're getting more ambitious.
[12:08] It is beyond this talk to dig into what's actually driving those level of lines of code changes.
[12:16] However, the increase in PR throughput is not coming from smaller PRs.
[12:21] This is intuitive for most, but it's still good to verify.
[12:27] Back to the opening question.
[12:29] Is 60% more spend worth 43% more output?
[12:35] You can't tell in isolation.
[12:37] You should track both.
[12:39] They are important numbers, but they're supporting metrics.
[12:43] AI spend per PR is the intelligence that you're buying as a resource to help make your devs go faster.
[12:52] PRs per engineer is what you're getting with that spend.
[12:56] However, looking at those hides the trade-off.
[13:01] [clears throat]
[13:04] At but put salary into the equation and take the ratio of cost to throughput, and there you get your answer.
[13:12] Cost per PR is down 26% over that same time period.
[13:18] It's clear AI is driving cost efficiencies despite the increasing bills.
[13:27] And the AI the AI spend needed to get those efficiencies is small, relatively speaking.
[13:34] AI is still a sliver of what a PR costs.
[13:38] Two and a half percent on average.
[13:41] Under 10% for the heavy spender group.
[13:44] Tokens are the cheap part.
[13:47] [clears throat]
[13:47] In the range that we see, more spend kept paying.
[13:51] Heavier token users saw 67% PR throughput gain compared to 40% for their lower spend peers.
[14:00] This drove cost per PR down 32% compared to about 25% for the lower spend peers.
[14:09] Looking at the PR cost to token ratio curve, the top 10% of company months by token usage resulted in 10 8x the tokens, but 25% lower cost per PR
[14:25] compared to the median.
[14:28] AI is making developers more productive on a scale we haven't seen before.
[14:33] It isn't just another tool, it is intelligence, which changes how much each dev can produce.
[14:41] That's why we need to think about its cost differently, not off to the side as overhead, but part of what it takes to produce each PR.
[14:50] You need to add the AI spend and the human spend the human salary spend together to calculate your real costs per PR.
[15:02] This isn't to say spend frivolously.
[15:06] Sure, you should watch for runaway jobs.
[15:09] You should track your AI spend.
[15:12] I'm not saying don't look at how much AI is going how much AI spend is going in each PR.
[15:18] I'm saying don't let that be your driving metric.
[15:22] A new framing might steer how and when you think about other investments, like integrating model routers, post-training your own model, implementing token limits, etc. Will you get more efficiencies
[15:37] out of trying to control AI costs or by continuing to invest in things that increase throughput, like more AI enablement?
[15:46] Either way, track the total intelligence costs, human plus artificial, per PR every month.
[15:54] Next to that AI bill and next to the PR throughput.
[15:59] How are you trending?
[16:00] Is throughput continuing to improve?
[16:03] What about those cost efficiencies, the cost to deliver?
[16:07] Watch reverts and quality metrics beside all of this as a counterweight.
[16:12] We don't need AI slop mucking up the gears.
[16:16] So, heading into budget season, bring cost per PR to the table.
[16:21] Not just your token bill, not just the throughput bump.
[16:25] The bill tells you what you spent.
[16:28] Cost per PR tells you whether it worked.
[16:32] And right now, for most companies, it says keep investing.
[16:36] [clears throat] Tokens are the cheap part to get efficiency gains.
[16:43] Well done.
[16:45] You made it to the end.
[16:46] Here's how you can stay up to date with our latest research, which again draws from the largest engineering workflow and AI telemetry data set in the industry.
[16:55] You can follow me on LinkedIn, my colleagues, or Jellyfish.
[17:01] Subscribe to our Substack, or you can check out our Jellyfish blog on the website.
[17:08] If you want help tracking your org's engineering workflow and AI effectiveness, including cost per merge PR, go to our website and book a demo, or reach out to me and I'll get you connected to the right folks.
[17:22] Again, I'm Steve Jameson, AI research and engineering leader at Jellyfish.
[17:26] Thanks for watching.
[17:27] Cheers.