Tokens Are the Cheap Part — Steve Jamison, Jellyfish

AI Engineer · 17 min · 198 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:02Hi, I'm Steve Jameson, AI research and engineering leader at Jellyfish.
00:07I'm here to show you why tokens are the cheap part.
00:11The research that I'll present is drawn from the Jellyfish platform.
00:14The industry's largest set of engineering workflow data joined AI telemetry data.
00:21It's no secret that AI costs are climbing.
00:24They're up 60x from this time last year.
00:27The use cases are wide and varied.
00:29The models and harnesses are getting much better.
00:32The implications are exciting.
00:34But discussions around budget and spend are not as much fun.
00:38These conversations involve your peers and senior leadership and the board.
00:42Not always the easiest audiences, especially when you don't have the right data.
00:47And it's real money these days.
00:51Output is up, too.
00:53Merged PRs per engineer are up 43% over the same one-year horizon.
00:58This is phenomenal.
00:59We've not seen productivity inputs like this, especially on this time scale.
01:06There's plenty of research and anecdotal data to know that AI and AI-driven coding are driving these throughput gains.
01:13Heck, even our charts show the correlation.
01:15Notice where the inflection points are.
01:18AI spend increased in early 2026 when throughput gains start a steady increase of their own.
01:25There's a tension with these two numbers, though.
01:28How do you reconcile them?
01:30Spend is up 60x.
01:32Throughput is also up 43%.
01:34Is that good?
01:36Bad?
01:3743% is a big number.
01:39But so is 60x.
01:41How do you know if you're spending enough?
01:45Is there more productivity gain locked behind a checkbook?
01:49Should we increase spend 50%?
01:522x more?
01:54Or how do you know if you spent too much?
01:57I'm sure many of you have felt the tension here.
02:00The CFO or board are asking for the receipts of the spend.
02:04Can you show them?
02:05Well, merge PRs are up 43% satisfy them?
02:10Also, what happens if you cap use and spend?
02:14Are you stopping from setting tokens on fire?
02:18Or are you stopping further efficiency gains?
02:21You can ask peers at other companies what they're paying.
02:24For many, it's still a moving target though.
02:27But, how do you equate that to your org?
02:30Surely, just comparing their bill to yours is inadequate.
02:34We've heard that reports of We've heard the reports of companies spending 100k, 500k, even millions on AI.
02:43Setting then resetting their AI budgets as reality sets in.
02:48Ultimately, these two numbers can't tell the story.
02:51And even together, they miss the biggest line on the cost side.
02:57These two numbers leave out the biggest cost of all, developer salary.
03:03AI is making developers more productive on a scale we haven't seen before.
03:07It isn't just another tool, it's intelligence.
03:11This intelligence dramatically changes how much each dev can produce.
03:15That's why we need to think about its cost differently.
03:18Not off to the side as overhead, but it's part of what it takes to produce each PR.
03:24We're going to call this intelligence costs of a PR.
03:32What you pay your engineers plus what you pay for AI.
03:36Put them together in one cost bucket.
03:40AI is about 2 and 1/2% of the cost to produce another unit of merged code.
03:46Salary is the other 97 and 1/2%.
03:50This is in line with other reported figures, and this data is from August 2026.
03:57Here, tokens are the cheap part.
04:01But, that's current relative costs, not dollars.
04:05Let's look at what happened to the cost of a merged PR over the last year.
04:15[clears throat]
04:16The dollar cost of a merged PR fell 26% over the last year.
04:21Not PRs created, but PRs merged.
04:24Review time is factored in, and some level of quality gates has been cleared.
04:30What That decline in cost per merged PR includes a 60x increase in AI costs over the same time frame.
04:37The change is nearly universal, too.
04:3983% of the companies we studied showed a drop in cost per PR.
04:44What does this tell us?
04:46The additional throughput gains are being spread or are [clears throat] spreading salary thinner across PRs, and this is happening faster than the AI costs are increasing.
05:00What's the range in token spend?
05:02We ranked companies by token usage, then grouped them into quintiles or fifths.
05:09The range here is quite large.
05:11The lightest group, about 0.3% of the cost of a PR is is AI.
05:20This represents about 60 million tokens per dev per month.
05:26The top 20% of AI spend per dev, we're going to call the heaviest spend group.
05:32There, 8 and 1/2% of the PR costs is AI.
05:36This represents about 2.5 billion tokens per dev per month.
05:41About two orders of magnitude more than the lightest group.
05:47We'll use these groups on the next slides to see how token spend influences other metrics.
05:53First, PR throughput measured as merged PRs per dev.
05:58Split by the same AI spend quintiles on the previous slide.
06:03The bottom 4/5 of spenders saw PR throughput gains of around 40% give or take.
06:12But the heavy spend group improved more at a 67% increase.
06:17More AI spend led to more throughput gains.
06:24Now let's look at the cost per PR.
06:27Positive results are seen again across the board.
06:30Cost per PR is dropping for all AI usage groups.
06:35However, again, the heaviest fifth so the the companies that spent the most improved the most.
06:44They saw a cost per PR decrease of 32% over the last year.
06:50The other four groups saw about a 25% decrease in cost per PR.
06:58Investing in spending more on AI fueled more of an improvement in cost per PR compared to peers who leaned in less.
07:07So we saw that the pop- same same as we saw in the population average, cost per PR changes are driven largely by PR throughput, not by AI spend.
07:20The heavy spenders experienced PR throughput gains of 67% compared to about 40-ish percent for their peers.
07:28Spreading the same salary across more PRs in the month has more of an effect than the rising AI costs because tokens are the cheap part.
07:42This is not to say that org should light tokens on fire and spend with abandon.
07:48Here we're looking at tokens used on the horizontal axis and cost per PR on the vertical axis.
07:54There's a steep drop at the beginning where relative small AI spend results in large gains initially.
08:01As token usage increases, the cost per PR improvement decreases.
08:07This is diminishing returns.
08:09I want to stress though, diminishing returns is not the same as no returns.
08:16We're seeing that the top 10% are spending 8x the tokens compared to the median and achieving a 27 25% lower cost per PR.
08:31There will be a time when tokens aren't the cheap part, but by and large for most companies out in the wild we're not close to that threshold.
08:42More tokens leads to more PRs, which leads to a lower cost of delivery.
08:55So far we've talked about companies.
08:58Next I'd like to take a tangent and talk about a study we did with developers.
09:04Here we looked at developer weeks with heavy token use compared to the same developer weeks with lower token use.
09:15[cough]
09:16[clears throat]
09:16Here we saw the same phenomenon just at the individual level.
09:21Compare each developers heavy agent skill weeks with their own weeks without skills.
09:29They spend about twice as much on tokens, merged 27% more PRs, and reach PR and each PR costs about 18% less once salary and tokens are counted together.
09:49There are two fair curiosities I get when I talk about this work.
09:53First, does faster work result in worse work?
10:00And second, are the more PRs the throughput gains that we're seeing, are they being driven by smaller chunks of work?
10:12First curiosity is cheaper, faster work, worse work?
10:17If AI makes PRs faster and cheaper, are we just shipping more mistakes?
10:22One proxy for this is river reverts.
10:25Merged PRs that undo an earlier PR.
10:29The heaviest fifth started lowest before most of the token growth, and revert rates drop slightly for every group.
10:38Though these deltas are just slightly below the noise in the data.
10:42It's not evidence of a improvement here.
10:45However, more importantly, we're not seeing any worrying signs of rampant or severe quality issues with higher token spends and faster PR throughput.
11:02My second curiosity is throughput driven by smaller PRs?
11:09And this is demonstrably no. Not smaller PRs.
11:13In fact, AI is driving an increase in line level lines of code generated per PR.
11:21This is no surprise to the devs in the room.
11:25Raise your hand if you've rolled your eyes at verbose, unnecessary inline AI comments, or a file full of mostly irrelevant tests.
11:34Gen AI Gen AI likes to generate code.
11:38Also, the bar to create is low for devs, and there's a pull to do more per PR instead of spinning up another ticket.
11:46It's just one, maybe two prompts, and I'll handle this edge case, or I can combine this and that element.
11:52Fewer PRs to wrangle.
11:54This doesn't mesh well with the PR review stress folks are feeling these days, though.
11:59There's
12:01[clears throat]
12:02Also, AI is just becoming more capable, so we're getting more ambitious.
12:08It is beyond this talk to dig into what's actually driving those level of lines of code changes.
12:16However, the increase in PR throughput is not coming from smaller PRs.
12:21This is intuitive for most, but it's still good to verify.
12:27Back to the opening question.
12:29Is 60% more spend worth 43% more output?
12:35You can't tell in isolation.
12:37You should track both.
12:39They are important numbers, but they're supporting metrics.
12:43AI spend per PR is the intelligence that you're buying as a resource to help make your devs go faster.
12:52PRs per engineer is what you're getting with that spend.
12:56However, looking at those hides the trade-off.
13:01[clears throat]
13:04At but put salary into the equation and take the ratio of cost to throughput, and there you get your answer.
13:12Cost per PR is down 26% over that same time period.
13:18It's clear AI is driving cost efficiencies despite the increasing bills.
13:27And the AI the AI spend needed to get those efficiencies is small, relatively speaking.
13:34AI is still a sliver of what a PR costs.
13:38Two and a half percent on average.
13:41Under 10% for the heavy spender group.
13:44Tokens are the cheap part.
13:47[clears throat]
13:47In the range that we see, more spend kept paying.
13:51Heavier token users saw 67% PR throughput gain compared to 40% for their lower spend peers.
14:00This drove cost per PR down 32% compared to about 25% for the lower spend peers.
14:09Looking at the PR cost to token ratio curve, the top 10% of company months by token usage resulted in 10 8x the tokens, but 25% lower cost per PR
14:25compared to the median.
14:28AI is making developers more productive on a scale we haven't seen before.
14:33It isn't just another tool, it is intelligence, which changes how much each dev can produce.
14:41That's why we need to think about its cost differently, not off to the side as overhead, but part of what it takes to produce each PR.
14:50You need to add the AI spend and the human spend the human salary spend together to calculate your real costs per PR.
15:02This isn't to say spend frivolously.
15:06Sure, you should watch for runaway jobs.
15:09You should track your AI spend.
15:12I'm not saying don't look at how much AI is going how much AI spend is going in each PR.
15:18I'm saying don't let that be your driving metric.
15:22A new framing might steer how and when you think about other investments, like integrating model routers, post-training your own model, implementing token limits, etc. Will you get more efficiencies
15:37out of trying to control AI costs or by continuing to invest in things that increase throughput, like more AI enablement?
15:46Either way, track the total intelligence costs, human plus artificial, per PR every month.
15:54Next to that AI bill and next to the PR throughput.
15:59How are you trending?
16:00Is throughput continuing to improve?
16:03What about those cost efficiencies, the cost to deliver?
16:07Watch reverts and quality metrics beside all of this as a counterweight.
16:12We don't need AI slop mucking up the gears.
16:16So, heading into budget season, bring cost per PR to the table.
16:21Not just your token bill, not just the throughput bump.
16:25The bill tells you what you spent.
16:28Cost per PR tells you whether it worked.
16:32And right now, for most companies, it says keep investing.
16:36[clears throat] Tokens are the cheap part to get efficiency gains.
16:43Well done.
16:45You made it to the end.
16:46Here's how you can stay up to date with our latest research, which again draws from the largest engineering workflow and AI telemetry data set in the industry.
16:55You can follow me on LinkedIn, my colleagues, or Jellyfish.
17:01Subscribe to our Substack, or you can check out our Jellyfish blog on the website.
17:08If you want help tracking your org's engineering workflow and AI effectiveness, including cost per merge PR, go to our website and book a demo, or reach out to me and I'll get you connected to the right folks.
17:22Again, I'm Steve Jameson, AI research and engineering leader at Jellyfish.
17:26Thanks for watching.
17:27Cheers.