Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish
https://www.youtube.com/watch?v=Ve2bAarPA2I
AI Engineer · 16 min · 228 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] I'm I'm Nicholas Arcolano.
[00:14] I'm the head of AI and research at Jellyfish.
[00:16] It's great to be back here at AI Engineer.
[00:19] And today I want to talk about why if you're not getting the outcomes that you want with AI, uh chances are it's actually not the model, the the problem is you.
[00:29] Um since we're here, uh you know, we're doing this this finance theme thing in the leadership track, we're going to start with a golf analogy.
[00:37] Seems appropriate.
[00:38] Uh is anybody here a golfer?
[00:40] Any golfers?
[00:41] Anybody?
[00:42] If you, okay.
[00:43] Um so yeah, I if you don't golf, bear with me.
[00:47] I apologize.
[00:48] If if you do golf, I also apologize.
[00:50] It's a very painful sport.
[00:52] Um I love golf.
[00:54] Uh I'm a bad golfer.
[00:56] Um it's part of the culture to say you're a bad golfer, but I I genuinely am a bad golfer.
[01:01] And when you're out there, you just want to make the pain go away.
[01:04] And one of the things you think of is like, oh, good should I just buy better clubs, right?
[01:08] That would make this easier.
[01:10] Uh and then you get an instructor and he's like, no, like you're the problem.
[01:13] You don't need better clubs, right?
[01:15] Uh it's about the fundamentals.
[01:16] But, you know, I'm a I'm a data guy, so I'm like, I don't I don't know.
[01:20] Is that really true?
[01:22] Uh if you look at the data, like clubs matter.
[01:24] Clubs have gotten better.
[01:26] So, what you're looking at here, this is the average driving distance for the PGA tour uh from 1980 to the present.
[01:34] In 1980, I was shocked.
[01:35] I actually didn't know this.
[01:35] The average was 257 yd.
[01:37] Like I've hit the ball that far.
[01:39] And as I mentioned, I'm a terrible golfer.
[01:42] Uh so it's gone up about 20% since 1980.
[01:46] And you know, one bit of evidence that the clubs matter is uh this solid core ball era thing in the middle by uh 2000.
[01:55] A bunch of techno- technological changes happened, and there was like a 15-yard bump right at that time.
[02:01] So, people didn't magically get better over that 3-year span, like that was all technology-driven.
[02:07] And this uh chart reminded me a lot of what we're seeing with the model frontier, right?
[02:12] We're seeing the steady advance of models getting better and better.
[02:16] This is a chart from artificial analysis.
[02:18] I assume everyone has seen one or many of these and checks them every time a new model comes out, right?
[02:23] So, the models have gotten better and better.
[02:26] Um I think that's uncontroversial.
[02:28] We can do things that we just could not do a year ago, right?
[02:33] So, the models matter a lot.
[02:35] Right?
[02:36] Yes?
[02:37] Well, I So, I don't know.
[02:38] The So, this is interesting, right?
[02:40] So, I work um I work at this company called Jellyfish.
[02:45] Uh we uh join AI coding tool and agent telemetry with engineering workflow data, engineering outcomes.
[02:53] Uh we have the largest data set of its kind, hundreds of thousands of developers, about 1,300 companies.
[02:59] And we're tracking sort of all the things they're doing with AI uh in the development space.
[03:03] And one thing that I noticed in our data is um this is data from around the time when uh SpaceX announced its acquisition of Cursor, and we noticed uh a huge bump in Grok usage
[03:17] uh and the biggest model change that we've ever seen, far faster than just normal engineers sort of changing their preferences and trying to seek out the best model for the job.
[03:28] And when we dug into it, what happened was Cursor was doing a lot of things with the UI, sort of promoting Grok up in the uh up in the UI, changing how auto mode worked.
[03:38] And so, um you know, it it shot up very fast, and what was exciting about this as being a a data scientist is this is what we call uh kind of a natural experiment.
[03:49] There was this external shock, and a natural question to ask would be, well, what what happened?
[03:54] Right?
[03:55] Did output change, for instance?
[03:57] So, what you're looking at here on this chart, this is change in merged PRs per engineer.
[04:05] And so, over that time span I showed you, every dot here is an engineer.
[04:09] And the x-axis is how much of their model share went from not using Grok to using Grok on a scale from 0 to 100%.
[04:18] The y-axis is the relative change in their PR throughput relative to their colleagues at the same company.
[04:25] And the story here is a non-story.
[04:27] There's the the cloud sort of center around zero.
[04:29] There's no statistical correlation.
[04:31] Output didn't change.
[04:32] Sort of shunting everyone to Grok didn't really matter.
[04:37] So, I left this experiment confused and wanting to know how much does the model matter.
[04:44] And if it doesn't matter much, what are the other things that do actually matter?
[04:49] So, that's what we're going to spend the rest of the talk talking about.
[04:53] Uh So, here's what we're going to do.
[04:56] We're going to continue to focus on this output metric of merged PRs per engineer per week.
[05:02] We're going to use a curated data set here.
[05:04] This is a subset of our data where we've got all the pieces we need to do this experiment.
[05:08] About 50,000 engineers at 447 companies.
[05:12] Data from this year for Cursor and Claude code users.
[05:15] And some of the stuff that matters.
[05:17] So, you've got the tool.
[05:18] We're going to put the harness and the model together because they're very intertwined.
[05:22] This is like your clubs, okay?
[05:24] There's how well you use it.
[05:26] By that, I mean things like your skill use, cursor rules, the the harness customization, the tricks of the trade that we're all learning as AI engineers to make the way that we use these tools better.
[05:37] Golf equivalent would be like this is knowing how to get out of a bunker, which is not like a natural thing.
[05:42] It's a thing you have to learn.
[05:44] There's how much you use it.
[05:45] So, separate from these things, just raw amount of token spend.
[05:49] So, golf time on the course, right?
[05:52] Uh and then there's factors like where you work, your environment.
[05:56] So, your company and your team, this is the course itself, something you don't have control over as a golfer.
[06:02] And then there's you, the individual, all the things that you carry with yourself as an engineer that, holding all these other factors constant, that make you still distinct.
[06:12] [snorts]
[06:13] So, this is the map for the rest of our talk.
[06:15] You put all this stuff together, you do a bunch of big statistical analyses, the details of which I would love to get into here, but would bore everyone except for
[06:23] statisticians like me.
[06:25] But, this is the map.
[06:26] You've got about uh less than 1% the in terms of how your how much your PR throughput changes is controlled, explained by the tool.
[06:39] How well you use it, the skills part, that's about 3%.
[06:43] Uh how much you use it, that's like 10x uh the tool, that's 9%.
[06:48] Where you work is 12%.
[06:50] The biggest piece is you, the individual.
[06:52] We'll go through each of these and explain a little bit about why this looks like this.
[07:10] Mhm.
[07:10] No, there's no quality metric here, but quality's a thing we look at a lot.
[07:14] I'm happy to talk more about that later, but good question.
[07:17] [snorts]
[07:19] Uh so, let's do the tool first.
[07:23] So, we'll kind of speed through this.
[07:25] This is the least interesting, as you might guess uh um from the title of my talk.
[07:30] But, what happens when engineers switch tools?
[07:33] We saw what happened when they were kind of encouraged, let's say, to switch tools, uh but engineers uh switch the models they use all the time, they switch harnesses.
[07:43] So, you can do things like look at change in the weekly merge PRs.
[07:47] You can compare an engineer to themselves and look at what happens when they go all in on Opus versus Sonnet, switch to Chat GPT.
[07:55] Um what happens if they switch from Cursor to Cloud Code, which we have a lot of data around people who've made that switch.
[08:00] What happens when uh you switch from working totally in an autocomplete mode to an agentic mode.
[08:06] The story's kind of the same.
[08:07] Statistically, it's kind of a nothingburger.
[08:10] There's not a lot here in terms of explaining any meaningful change in um the amount of PR throughput that you get.
[08:20] What about how well you use it?
[08:22] This is 3%, so much bigger piece of the pie.
[08:25] Let's talk about skill use.
[08:27] This one's interesting because skills This should be bigger.
[08:31] Skills actually we find matter a lot.
[08:33] One of the reasons it doesn't matter more is and I don't This might surprise some folks.
[08:40] People who use skills that much like in the real world outside of like this this wonderful land of AI engineering conferences that that we're inhabiting.
[08:49] Uh this chart shows um you know, skill use engineer weeks over time.
[08:55] So, proportion of engineering weeks that have, you know, low zero skill use, a low level, a medium level, and a high level of skill use.
[09:02] And as of September, you know, almost half of the weeks here shown had essentially no skill use.
[09:11] Uh and only 12% had are using skills at a level that I think we would consider as like what you should be doing in terms of modern AI engineering.
[09:20] And the reason that this is so important is because skill use when it's prevalent moves output way more than model choice.
[09:29] So, I'm showing here again those results I flashed from earlier about changing model family as as kind of a reference.
[09:35] But on the bottom here is level of skill use and the amount of effect it has in terms of changing the level of PR throughput holding these other factors constant.
[09:46] And you can see you get um you know, when you get up to 30 plus runs uh of using skills uh per engineer per week, that's a 30% bump just from that factor alone holding other things constant.
[10:00] So, it's a big effect.
[10:01] [snorts]
[10:04] All right, let's talk about how much you use it.
[10:07] This is consumption.
[10:08] This is tokens.
[10:09] This is active use days.
[10:12] Uh this one I could do a whole talk about this.
[10:15] In fact, I did a whole talk about this at the AI Engineer World's Fair.
[10:18] So, I encourage you if you find this interesting to uh to take a look at that.
[10:23] Um but I'll give you the one slide version.
[10:26] So, the data you're looking at here is from that talk.
[10:29] So, this is um about 500,000 developer weeks of behavior.
[10:34] Each dot is a two percentile bin of uh folks that use the same level of consumption in that week, and it's their average token span and their merged PRs per developer per week.
[10:47] And the basic story is more tokens, more throughput.
[10:51] And there's diminishing returns.
[10:53] So, there's sort of a saturating curve.
[10:54] I likened it to the rocket equation in that talk.
[10:56] It's like rocket fuel.
[10:58] And what you get is if you look at the top 10% versus the median, uh you can basically double your output between the median and the top 10%.
[11:07] It costs you about 10x the tokens.
[11:09] But more tokens mean more output.
[11:14] Now we get to the bigger pieces of the pie.
[11:17] These are interesting.
[11:18] So, there's the factor of you know, where you work, your company, your team.
[11:23] Actually, if you tease out the factors, about a third of it's the team, two-thirds is the company.
[11:27] You know, your your code base, your review process, um your culture.
[11:33] So, once you sort of fix the amount of consumption, you know, do these other things still matter?
[11:42] One way to look at this is, you know, if you fix these factors and just look at the variability, you look at the spread of companies relative to the average company holding these other things fixed.
[11:52] Um, this is what it looks like.
[11:54] And one way to interpret this is, I just showed you that token curve.
[11:58] So, you could spend 10x the tokens to move from the median up to the top 10 percentile and get double the output.
[12:04] There's also a 2x spread between the top 10% company and the bottom 10% company in this distribution holding your consumption fixed.
[12:13] So, in other words, your culture, your team, your company, there's a lot of variation just in those things even for fixed AI spend.
[12:20] This matters a lot and it's interesting for this group in the in the leadership track here because this is what leaders can control.
[12:28] Like, this is our our job is to control these things even if maybe our CFO tells us we can't have any more tokens than we have.
[12:36] Um, which hopefully they don't.
[12:39] So, what about the biggest piece?
[12:42] You, the individual, right?
[12:44] This is this three quarters of this statistical model.
[12:48] So, it matters a lot.
[12:49] [snorts]
[12:50] Um, one way of thinking about this is this question that a lot of people have asked is, you know, does AI mean that you need different engineers?
[12:58] Do we need different people, right?
[13:00] Is it resorting the rankings?
[13:01] Is it changing what it means to be a good engineer?
[13:05] So, uh, you saw that distribution I just showed you with companies.
[13:11] If you do the same sort of distribution for engineers, it's even wider.
[13:14] So, there's even more variability and that's why it's such a big factor in this this statistical map we've we've laid out for ourselves to explain the different levers available to us.
[13:25] Uh, it's even wider.
[13:26] It's, you know, 3.5x spread between the bottom 10% and the top 10% uh instead of 2x as for companies.
[13:34] And there's a lot to unpack here.
[13:35] This will be a whole 'nother talk to unpack all the things that make different uh AI engineers uh better than others.
[13:42] One thing that we can answer a little bit in the few minutes I've got left is is AI changing who the most uh productive engineers are.
[13:52] So, one way to look at this, here's a small study ranking 10,000 engineers.
[13:57] We have long uh you know, longitudinal data, long timeline.
[14:00] Jellyfish has been around 10 years.
[14:01] So, we have, you know, tons of history about how engineers behave.
[14:04] And if you look at, you know, how um top engineers rank uh you know, quarter over quarter, 6 months over 6 months, uh we can look at, you know, how much do your top engineers cycle in terms of productivity here.
[14:18] Um PR throughput we're still talking about.
[14:21] How much uh they cycle out of the top quartile.
[14:24] And if you compare that uh so, you know, about 44% cycle out of the top quartile uh every half year.
[14:32] Uh if you look at before AI versus after, it actually tightens a little.
[14:35] People get a little less mobile.
[14:37] It's 41%.
[14:38] So, and we've done other studies like this.
[14:40] We're not seeing um you know, a big re-sorting of the world order.
[14:46] Your your best engineers before AI by and large are still your best engineers.
[14:50] So, the the sort of narrative that AI's leveling the the playing field uh there's craft still matters for a lot, and that's why part of why this factor is as big as it is.
[15:02] So, you know, going back to this this picture that I've I've been showing you.
[15:05] When you add it all up, you know, what I want you to take away is is these things on the left, you know, the tool especially, are like really exciting to talk about.
[15:14] It's the cool new tech.
[15:15] It takes up kind of all the air in the room, all the air on on Twitter.
[15:18] And like that's exciting, and I love that stuff.
[15:21] Um but we don't have a lot of control over that.
[15:24] I don't work at a foundation lab.
[15:25] I can't make the tools better.
[15:27] Uh, and as cost tighten, we have um, less and less freedom maybe to buy the tools that we want.
[15:34] But as leaders, the things we can control are on the other side of the uh, the equation.
[15:40] It's it's the people, it's the organization.
[15:43] And I think what's, you know, going back to our golf analogy, what's really exciting about that is unlike golf, you can change the course.
[15:50] You can change your environment.
[15:51] So, you can like fill in water hazards or do things that they yell at you if you try to do on a golf course.
[15:58] But um, those are things as leaders that are available to us.
[16:02] So, in closing, um, you know, if you're not getting the outcomes that you want, I would encourage you to think, you know, maybe it's not the model, you know, maybe it's you.
[16:16] So.
[16:17] [applause]
[16:18] Thank you.
[16:19] Appreciate that.
[16:20] That's all I've got.
[16:21] If you're interested, we publish monthly benchmarks with all of this data and more.
[16:25] Go to Jellyfish AI engineering trends.
[16:27] Uh, a lot of our research is published on our Substack.
[16:31] Uh, thank you.