WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=U7ZYBCcdJPg NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/U7ZYBCcdJPg.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:02.840 --> 00:00:07.640 Hi, I'm Steve Jameson, AI research and engineering leader at Jellyfish. s2 00:00:07.640 --> 00:00:11.160 I'm here to show you why tokens are the cheap part. s3 00:00:11.160 --> 00:00:14.560 The research that I'll present is drawn from the Jellyfish platform. s4 00:00:14.560 --> 00:00:21.280 The industry's largest set of engineering workflow data joined AI telemetry data. s5 00:00:21.480 --> 00:00:24.400 It's no secret that AI costs are climbing. s6 00:00:24.400 --> 00:00:27.240 They're up 60x from this time last year. s7 00:00:27.240 --> 00:00:29.600 The use cases are wide and varied. s8 00:00:29.600 --> 00:00:32.320 The models and harnesses are getting much better. s9 00:00:32.320 --> 00:00:34.640 The implications are exciting. s10 00:00:34.640 --> 00:00:38.760 But discussions around budget and spend are not as much fun. s11 00:00:38.760 --> 00:00:42.960 These conversations involve your peers and senior leadership and the board. s12 00:00:42.960 --> 00:00:47.680 Not always the easiest audiences, especially when you don't have the right data. s13 00:00:47.680 --> 00:00:51.400 And it's real money these days. s14 00:00:51.600 --> 00:00:53.080 Output is up, too. s15 00:00:53.080 --> 00:00:58.480 Merged PRs per engineer are up 43% over the same one-year horizon. s16 00:00:58.480 --> 00:00:59.680 This is phenomenal. s17 00:00:59.680 --> 00:01:06.360 We've not seen productivity inputs like this, especially on this time scale. s18 00:01:06.600 --> 00:01:13.000 There's plenty of research and anecdotal data to know that AI and AI-driven coding are driving these throughput gains. s19 00:01:13.000 --> 00:01:15.560 Heck, even our charts show the correlation. s20 00:01:15.560 --> 00:01:18.200 Notice where the inflection points are. s21 00:01:18.200 --> 00:01:25.560 AI spend increased in early 2026 when throughput gains start a steady increase of their own. s22 00:01:25.760 --> 00:01:28.440 There's a tension with these two numbers, though. s23 00:01:28.440 --> 00:01:30.160 How do you reconcile them? s24 00:01:30.160 --> 00:01:32.200 Spend is up 60x. s25 00:01:32.200 --> 00:01:34.880 Throughput is also up 43%. s26 00:01:34.880 --> 00:01:36.000 Is that good? s27 00:01:36.000 --> 00:01:37.120 Bad? s28 00:01:37.120 --> 00:01:39.480 43% is a big number. s29 00:01:39.480 --> 00:01:41.880 But so is 60x. s30 00:01:41.880 --> 00:01:45.440 How do you know if you're spending enough? s31 00:01:45.440 --> 00:01:49.760 Is there more productivity gain locked behind a checkbook? s32 00:01:49.760 --> 00:01:52.440 Should we increase spend 50%? s33 00:01:52.440 --> 00:01:54.840 2x more? s34 00:01:54.840 --> 00:01:57.760 Or how do you know if you spent too much? s35 00:01:57.760 --> 00:02:00.240 I'm sure many of you have felt the tension here. s36 00:02:00.240 --> 00:02:04.080 The CFO or board are asking for the receipts of the spend. s37 00:02:04.080 --> 00:02:05.640 Can you show them? s38 00:02:05.640 --> 00:02:10.240 Well, merge PRs are up 43% satisfy them? s39 00:02:10.240 --> 00:02:14.760 Also, what happens if you cap use and spend? s40 00:02:14.760 --> 00:02:18.200 Are you stopping from setting tokens on fire? s41 00:02:18.200 --> 00:02:21.560 Or are you stopping further efficiency gains? s42 00:02:21.560 --> 00:02:24.360 You can ask peers at other companies what they're paying. s43 00:02:24.360 --> 00:02:27.320 For many, it's still a moving target though. s44 00:02:27.320 --> 00:02:30.480 But, how do you equate that to your org? s45 00:02:30.480 --> 00:02:34.360 Surely, just comparing their bill to yours is inadequate. s46 00:02:34.360 --> 00:02:43.160 We've heard that reports of We've heard the reports of companies spending 100k, 500k, even millions on AI. s47 00:02:43.160 --> 00:02:48.360 Setting then resetting their AI budgets as reality sets in. s48 00:02:48.360 --> 00:02:51.720 Ultimately, these two numbers can't tell the story. s49 00:02:51.720 --> 00:02:56.600 And even together, they miss the biggest line on the cost side. s50 00:02:57.959 --> 00:03:03.040 These two numbers leave out the biggest cost of all, developer salary. s51 00:03:03.040 --> 00:03:07.640 AI is making developers more productive on a scale we haven't seen before. s52 00:03:07.640 --> 00:03:11.239 It isn't just another tool, it's intelligence. s53 00:03:11.239 --> 00:03:15.880 This intelligence dramatically changes how much each dev can produce. s54 00:03:15.880 --> 00:03:18.920 That's why we need to think about its cost differently. s55 00:03:18.920 --> 00:03:24.840 Not off to the side as overhead, but it's part of what it takes to produce each PR. s56 00:03:24.840 --> 00:03:29.760 We're going to call this intelligence costs of a PR. s57 00:03:32.519 --> 00:03:36.800 What you pay your engineers plus what you pay for AI. s58 00:03:36.800 --> 00:03:40.200 Put them together in one cost bucket. s59 00:03:40.200 --> 00:03:46.560 AI is about 2 and 1/2% of the cost to produce another unit of merged code. s60 00:03:46.560 --> 00:03:50.840 Salary is the other 97 and 1/2%. s61 00:03:50.840 --> 00:03:56.880 This is in line with other reported figures, and this data is from August 2026. s62 00:03:57.960 --> 00:04:01.680 Here, tokens are the cheap part. s63 00:04:01.680 --> 00:04:05.120 But, that's current relative costs, not dollars. s64 00:04:05.120 --> 00:04:10.800 Let's look at what happened to the cost of a merged PR over the last year. s65 00:04:15.452 --> 00:04:16.359 [clears throat] s66 00:04:16.359 --> 00:04:21.160 The dollar cost of a merged PR fell 26% over the last year. s67 00:04:21.160 --> 00:04:24.520 Not PRs created, but PRs merged. s68 00:04:24.520 --> 00:04:30.200 Review time is factored in, and some level of quality gates has been cleared. s69 00:04:30.200 --> 00:04:37.080 What That decline in cost per merged PR includes a 60x increase in AI costs over the same time frame. s70 00:04:37.080 --> 00:04:39.440 The change is nearly universal, too. s71 00:04:39.440 --> 00:04:44.680 83% of the companies we studied showed a drop in cost per PR. s72 00:04:44.680 --> 00:04:46.320 What does this tell us? s73 00:04:46.320 --> 00:04:58.200 The additional throughput gains are being spread or are [clears throat] spreading salary thinner across PRs, and this is happening faster than the AI costs are increasing. s74 00:05:00.160 --> 00:05:02.400 What's the range in token spend? s75 00:05:02.400 --> 00:05:09.080 We ranked companies by token usage, then grouped them into quintiles or fifths. s76 00:05:09.080 --> 00:05:11.480 The range here is quite large. s77 00:05:11.480 --> 00:05:20.800 The lightest group, about 0.3% of the cost of a PR is is AI. s78 00:05:20.800 --> 00:05:26.040 This represents about 60 million tokens per dev per month. s79 00:05:26.040 --> 00:05:32.720 The top 20% of AI spend per dev, we're going to call the heaviest spend group. s80 00:05:32.720 --> 00:05:36.840 There, 8 and 1/2% of the PR costs is AI. s81 00:05:36.840 --> 00:05:41.720 This represents about 2.5 billion tokens per dev per month. s82 00:05:41.720 --> 00:05:46.880 About two orders of magnitude more than the lightest group. s83 00:05:47.000 --> 00:05:53.320 We'll use these groups on the next slides to see how token spend influences other metrics. s84 00:05:53.880 --> 00:05:58.320 First, PR throughput measured as merged PRs per dev. s85 00:05:58.320 --> 00:06:03.520 Split by the same AI spend quintiles on the previous slide. s86 00:06:03.520 --> 00:06:12.080 The bottom 4/5 of spenders saw PR throughput gains of around 40% give or take. s87 00:06:12.080 --> 00:06:17.680 But the heavy spend group improved more at a 67% increase. s88 00:06:17.680 --> 00:06:23.240 More AI spend led to more throughput gains. s89 00:06:24.680 --> 00:06:27.680 Now let's look at the cost per PR. s90 00:06:27.680 --> 00:06:30.960 Positive results are seen again across the board. s91 00:06:30.960 --> 00:06:35.160 Cost per PR is dropping for all AI usage groups. s92 00:06:35.160 --> 00:06:44.160 However, again, the heaviest fifth so the the companies that spent the most improved the most. s93 00:06:44.160 --> 00:06:50.880 They saw a cost per PR decrease of 32% over the last year. s94 00:06:50.880 --> 00:06:57.960 The other four groups saw about a 25% decrease in cost per PR. s95 00:06:58.080 --> 00:07:07.480 Investing in spending more on AI fueled more of an improvement in cost per PR compared to peers who leaned in less. s96 00:07:07.480 --> 00:07:20.200 So we saw that the pop- same same as we saw in the population average, cost per PR changes are driven largely by PR throughput, not by AI spend. s97 00:07:20.200 --> 00:07:28.800 The heavy spenders experienced PR throughput gains of 67% compared to about 40-ish percent for their peers. s98 00:07:28.800 --> 00:07:41.440 Spreading the same salary across more PRs in the month has more of an effect than the rising AI costs because tokens are the cheap part. s99 00:07:42.040 --> 00:07:48.520 This is not to say that org should light tokens on fire and spend with abandon. s100 00:07:48.520 --> 00:07:54.240 Here we're looking at tokens used on the horizontal axis and cost per PR on the vertical axis. s101 00:07:54.240 --> 00:08:01.960 There's a steep drop at the beginning where relative small AI spend results in large gains initially. s102 00:08:01.960 --> 00:08:07.240 As token usage increases, the cost per PR improvement decreases. s103 00:08:07.240 --> 00:08:09.360 This is diminishing returns. s104 00:08:09.360 --> 00:08:16.080 I want to stress though, diminishing returns is not the same as no returns. s105 00:08:16.400 --> 00:08:31.400 We're seeing that the top 10% are spending 8x the tokens compared to the median and achieving a 27 25% lower cost per PR. s106 00:08:31.400 --> 00:08:42.000 There will be a time when tokens aren't the cheap part, but by and large for most companies out in the wild we're not close to that threshold. s107 00:08:42.000 --> 00:08:48.720 More tokens leads to more PRs, which leads to a lower cost of delivery. s108 00:08:55.280 --> 00:08:58.000 So far we've talked about companies. s109 00:08:58.000 --> 00:09:04.200 Next I'd like to take a tangent and talk about a study we did with developers. s110 00:09:04.440 --> 00:09:15.446 Here we looked at developer weeks with heavy token use compared to the same developer weeks with lower token use. s111 00:09:15.446 --> 00:09:16.101 [cough] s112 00:09:16.101 --> 00:09:16.720 [clears throat] s113 00:09:16.720 --> 00:09:21.320 Here we saw the same phenomenon just at the individual level. s114 00:09:21.320 --> 00:09:29.200 Compare each developers heavy agent skill weeks with their own weeks without skills. s115 00:09:29.200 --> 00:09:45.720 They spend about twice as much on tokens, merged 27% more PRs, and reach PR and each PR costs about 18% less once salary and tokens are counted together. s116 00:09:49.320 --> 00:09:53.840 There are two fair curiosities I get when I talk about this work. s117 00:09:53.840 --> 00:10:00.280 First, does faster work result in worse work? s118 00:10:00.280 --> 00:10:11.640 And second, are the more PRs the throughput gains that we're seeing, are they being driven by smaller chunks of work? s119 00:10:12.720 --> 00:10:17.840 First curiosity is cheaper, faster work, worse work? s120 00:10:17.840 --> 00:10:22.800 If AI makes PRs faster and cheaper, are we just shipping more mistakes? s121 00:10:22.800 --> 00:10:25.880 One proxy for this is river reverts. s122 00:10:25.880 --> 00:10:29.240 Merged PRs that undo an earlier PR. s123 00:10:29.240 --> 00:10:38.640 The heaviest fifth started lowest before most of the token growth, and revert rates drop slightly for every group. s124 00:10:38.640 --> 00:10:42.520 Though these deltas are just slightly below the noise in the data. s125 00:10:42.520 --> 00:10:45.640 It's not evidence of a improvement here. s126 00:10:45.640 --> 00:11:01.080 However, more importantly, we're not seeing any worrying signs of rampant or severe quality issues with higher token spends and faster PR throughput. s127 00:11:02.240 --> 00:11:09.240 My second curiosity is throughput driven by smaller PRs? s128 00:11:09.320 --> 00:11:13.839 And this is demonstrably no. Not smaller PRs. s129 00:11:13.839 --> 00:11:21.640 In fact, AI is driving an increase in line level lines of code generated per PR. s130 00:11:21.640 --> 00:11:25.080 This is no surprise to the devs in the room. s131 00:11:25.080 --> 00:11:34.040 Raise your hand if you've rolled your eyes at verbose, unnecessary inline AI comments, or a file full of mostly irrelevant tests. s132 00:11:34.040 --> 00:11:38.840 Gen AI Gen AI likes to generate code. s133 00:11:38.840 --> 00:11:46.040 Also, the bar to create is low for devs, and there's a pull to do more per PR instead of spinning up another ticket. s134 00:11:46.040 --> 00:11:52.000 It's just one, maybe two prompts, and I'll handle this edge case, or I can combine this and that element. s135 00:11:52.000 --> 00:11:54.600 Fewer PRs to wrangle. s136 00:11:54.600 --> 00:11:59.960 This doesn't mesh well with the PR review stress folks are feeling these days, though. s137 00:11:59.960 --> 00:12:01.163 There's s138 00:12:01.163 --> 00:12:02.320 [clears throat] s139 00:12:02.320 --> 00:12:08.120 Also, AI is just becoming more capable, so we're getting more ambitious. s140 00:12:08.120 --> 00:12:16.200 It is beyond this talk to dig into what's actually driving those level of lines of code changes. s141 00:12:16.200 --> 00:12:21.840 However, the increase in PR throughput is not coming from smaller PRs. s142 00:12:21.840 --> 00:12:26.800 This is intuitive for most, but it's still good to verify. s143 00:12:27.680 --> 00:12:29.640 Back to the opening question. s144 00:12:29.640 --> 00:12:35.240 Is 60% more spend worth 43% more output? s145 00:12:35.240 --> 00:12:37.760 You can't tell in isolation. s146 00:12:37.760 --> 00:12:39.040 You should track both. s147 00:12:39.040 --> 00:12:43.000 They are important numbers, but they're supporting metrics. s148 00:12:43.000 --> 00:12:52.040 AI spend per PR is the intelligence that you're buying as a resource to help make your devs go faster. s149 00:12:52.040 --> 00:12:56.880 PRs per engineer is what you're getting with that spend. s150 00:12:56.880 --> 00:13:00.800 However, looking at those hides the trade-off. s151 00:13:01.423 --> 00:13:03.423 [clears throat] s152 00:13:04.560 --> 00:13:12.120 At but put salary into the equation and take the ratio of cost to throughput, and there you get your answer. s153 00:13:12.120 --> 00:13:18.120 Cost per PR is down 26% over that same time period. s154 00:13:18.120 --> 00:13:25.400 It's clear AI is driving cost efficiencies despite the increasing bills. s155 00:13:27.760 --> 00:13:34.960 And the AI the AI spend needed to get those efficiencies is small, relatively speaking. s156 00:13:34.960 --> 00:13:38.640 AI is still a sliver of what a PR costs. s157 00:13:38.640 --> 00:13:41.600 Two and a half percent on average. s158 00:13:41.600 --> 00:13:44.880 Under 10% for the heavy spender group. s159 00:13:44.880 --> 00:13:47.274 Tokens are the cheap part. s160 00:13:47.274 --> 00:13:47.880 [clears throat] s161 00:13:47.880 --> 00:13:51.840 In the range that we see, more spend kept paying. s162 00:13:51.840 --> 00:14:00.920 Heavier token users saw 67% PR throughput gain compared to 40% for their lower spend peers. s163 00:14:00.920 --> 00:14:09.400 This drove cost per PR down 32% compared to about 25% for the lower spend peers. s164 00:14:09.400 --> 00:14:25.800 Looking at the PR cost to token ratio curve, the top 10% of company months by token usage resulted in 10 8x the tokens, but 25% lower cost per PR s165 00:14:25.800 --> 00:14:28.360 compared to the median. s166 00:14:28.600 --> 00:14:33.520 AI is making developers more productive on a scale we haven't seen before. s167 00:14:33.520 --> 00:14:41.400 It isn't just another tool, it is intelligence, which changes how much each dev can produce. s168 00:14:41.400 --> 00:14:50.720 That's why we need to think about its cost differently, not off to the side as overhead, but part of what it takes to produce each PR. s169 00:14:50.760 --> 00:15:02.560 You need to add the AI spend and the human spend the human salary spend together to calculate your real costs per PR. s170 00:15:02.560 --> 00:15:06.160 This isn't to say spend frivolously. s171 00:15:06.160 --> 00:15:09.880 Sure, you should watch for runaway jobs. s172 00:15:09.880 --> 00:15:12.280 You should track your AI spend. s173 00:15:12.280 --> 00:15:18.680 I'm not saying don't look at how much AI is going how much AI spend is going in each PR. s174 00:15:18.680 --> 00:15:22.600 I'm saying don't let that be your driving metric. s175 00:15:22.600 --> 00:15:37.920 A new framing might steer how and when you think about other investments, like integrating model routers, post-training your own model, implementing token limits, etc. Will you get more efficiencies s176 00:15:37.920 --> 00:15:46.840 out of trying to control AI costs or by continuing to invest in things that increase throughput, like more AI enablement? s177 00:15:46.840 --> 00:15:54.560 Either way, track the total intelligence costs, human plus artificial, per PR every month. s178 00:15:54.560 --> 00:15:59.520 Next to that AI bill and next to the PR throughput. s179 00:15:59.520 --> 00:16:00.920 How are you trending? s180 00:16:00.920 --> 00:16:03.680 Is throughput continuing to improve? s181 00:16:03.680 --> 00:16:07.440 What about those cost efficiencies, the cost to deliver? s182 00:16:07.440 --> 00:16:12.720 Watch reverts and quality metrics beside all of this as a counterweight. s183 00:16:12.720 --> 00:16:16.360 We don't need AI slop mucking up the gears. s184 00:16:16.360 --> 00:16:21.720 So, heading into budget season, bring cost per PR to the table. s185 00:16:21.720 --> 00:16:25.880 Not just your token bill, not just the throughput bump. s186 00:16:25.880 --> 00:16:28.080 The bill tells you what you spent. s187 00:16:28.080 --> 00:16:32.520 Cost per PR tells you whether it worked. s188 00:16:32.520 --> 00:16:36.266 And right now, for most companies, it says keep investing. s189 00:16:36.266 --> 00:16:42.680 [clears throat] Tokens are the cheap part to get efficiency gains. s190 00:16:43.839 --> 00:16:45.000 Well done. s191 00:16:45.000 --> 00:16:46.960 You made it to the end. s192 00:16:46.960 --> 00:16:55.839 Here's how you can stay up to date with our latest research, which again draws from the largest engineering workflow and AI telemetry data set in the industry. s193 00:16:55.839 --> 00:17:01.520 You can follow me on LinkedIn, my colleagues, or Jellyfish. s194 00:17:01.520 --> 00:17:08.199 Subscribe to our Substack, or you can check out our Jellyfish blog on the website. s195 00:17:08.199 --> 00:17:22.160 If you want help tracking your org's engineering workflow and AI effectiveness, including cost per merge PR, go to our website and book a demo, or reach out to me and I'll get you connected to the right folks. s196 00:17:22.160 --> 00:17:26.240 Again, I'm Steve Jameson, AI research and engineering leader at Jellyfish. s197 00:17:26.240 --> 00:17:27.680 Thanks for watching. s198 00:17:27.680 --> 00:17:29.680 Cheers.