Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

AI Engineer · 16 min · 228 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12I'm I'm Nicholas Arcolano.
00:14I'm the head of AI and research at Jellyfish.
00:16It's great to be back here at AI Engineer.
00:19And today I want to talk about why if you're not getting the outcomes that you want with AI, uh chances are it's actually not the model, the the problem is you.
00:29Um since we're here, uh you know, we're doing this this finance theme thing in the leadership track, we're going to start with a golf analogy.
00:37Seems appropriate.
00:38Uh is anybody here a golfer?
00:40Any golfers?
00:41Anybody?
00:42If you, okay.
00:43Um so yeah, I if you don't golf, bear with me.
00:47I apologize.
00:48If if you do golf, I also apologize.
00:50It's a very painful sport.
00:52Um I love golf.
00:54Uh I'm a bad golfer.
00:56Um it's part of the culture to say you're a bad golfer, but I I genuinely am a bad golfer.
01:01And when you're out there, you just want to make the pain go away.
01:04And one of the things you think of is like, oh, good should I just buy better clubs, right?
01:08That would make this easier.
01:10Uh and then you get an instructor and he's like, no, like you're the problem.
01:13You don't need better clubs, right?
01:15Uh it's about the fundamentals.
01:16But, you know, I'm a I'm a data guy, so I'm like, I don't I don't know.
01:20Is that really true?
01:22Uh if you look at the data, like clubs matter.
01:24Clubs have gotten better.
01:26So, what you're looking at here, this is the average driving distance for the PGA tour uh from 1980 to the present.
01:34In 1980, I was shocked.
01:35I actually didn't know this.
01:35The average was 257 yd.
01:37Like I've hit the ball that far.
01:39And as I mentioned, I'm a terrible golfer.
01:42Uh so it's gone up about 20% since 1980.
01:46And you know, one bit of evidence that the clubs matter is uh this solid core ball era thing in the middle by uh 2000.
01:55A bunch of techno- technological changes happened, and there was like a 15-yard bump right at that time.
02:01So, people didn't magically get better over that 3-year span, like that was all technology-driven.
02:07And this uh chart reminded me a lot of what we're seeing with the model frontier, right?
02:12We're seeing the steady advance of models getting better and better.
02:16This is a chart from artificial analysis.
02:18I assume everyone has seen one or many of these and checks them every time a new model comes out, right?
02:23So, the models have gotten better and better.
02:26Um I think that's uncontroversial.
02:28We can do things that we just could not do a year ago, right?
02:33So, the models matter a lot.
02:35Right?
02:36Yes?
02:37Well, I So, I don't know.
02:38The So, this is interesting, right?
02:40So, I work um I work at this company called Jellyfish.
02:45Uh we uh join AI coding tool and agent telemetry with engineering workflow data, engineering outcomes.
02:53Uh we have the largest data set of its kind, hundreds of thousands of developers, about 1,300 companies.
02:59And we're tracking sort of all the things they're doing with AI uh in the development space.
03:03And one thing that I noticed in our data is um this is data from around the time when uh SpaceX announced its acquisition of Cursor, and we noticed uh a huge bump in Grok usage
03:17uh and the biggest model change that we've ever seen, far faster than just normal engineers sort of changing their preferences and trying to seek out the best model for the job.
03:28And when we dug into it, what happened was Cursor was doing a lot of things with the UI, sort of promoting Grok up in the uh up in the UI, changing how auto mode worked.
03:38And so, um you know, it it shot up very fast, and what was exciting about this as being a a data scientist is this is what we call uh kind of a natural experiment.
03:49There was this external shock, and a natural question to ask would be, well, what what happened?
03:54Right?
03:55Did output change, for instance?
03:57So, what you're looking at here on this chart, this is change in merged PRs per engineer.
04:05And so, over that time span I showed you, every dot here is an engineer.
04:09And the x-axis is how much of their model share went from not using Grok to using Grok on a scale from 0 to 100%.
04:18The y-axis is the relative change in their PR throughput relative to their colleagues at the same company.
04:25And the story here is a non-story.
04:27There's the the cloud sort of center around zero.
04:29There's no statistical correlation.
04:31Output didn't change.
04:32Sort of shunting everyone to Grok didn't really matter.
04:37So, I left this experiment confused and wanting to know how much does the model matter.
04:44And if it doesn't matter much, what are the other things that do actually matter?
04:49So, that's what we're going to spend the rest of the talk talking about.
04:53Uh So, here's what we're going to do.
04:56We're going to continue to focus on this output metric of merged PRs per engineer per week.
05:02We're going to use a curated data set here.
05:04This is a subset of our data where we've got all the pieces we need to do this experiment.
05:08About 50,000 engineers at 447 companies.
05:12Data from this year for Cursor and Claude code users.
05:15And some of the stuff that matters.
05:17So, you've got the tool.
05:18We're going to put the harness and the model together because they're very intertwined.
05:22This is like your clubs, okay?
05:24There's how well you use it.
05:26By that, I mean things like your skill use, cursor rules, the the harness customization, the tricks of the trade that we're all learning as AI engineers to make the way that we use these tools better.
05:37Golf equivalent would be like this is knowing how to get out of a bunker, which is not like a natural thing.
05:42It's a thing you have to learn.
05:44There's how much you use it.
05:45So, separate from these things, just raw amount of token spend.
05:49So, golf time on the course, right?
05:52Uh and then there's factors like where you work, your environment.
05:56So, your company and your team, this is the course itself, something you don't have control over as a golfer.
06:02And then there's you, the individual, all the things that you carry with yourself as an engineer that, holding all these other factors constant, that make you still distinct.
06:12[snorts]
06:13So, this is the map for the rest of our talk.
06:15You put all this stuff together, you do a bunch of big statistical analyses, the details of which I would love to get into here, but would bore everyone except for
06:23statisticians like me.
06:25But, this is the map.
06:26You've got about uh less than 1% the in terms of how your how much your PR throughput changes is controlled, explained by the tool.
06:39How well you use it, the skills part, that's about 3%.
06:43Uh how much you use it, that's like 10x uh the tool, that's 9%.
06:48Where you work is 12%.
06:50The biggest piece is you, the individual.
06:52We'll go through each of these and explain a little bit about why this looks like this.
07:10Mhm.
07:10No, there's no quality metric here, but quality's a thing we look at a lot.
07:14I'm happy to talk more about that later, but good question.
07:17[snorts]
07:19Uh so, let's do the tool first.
07:23So, we'll kind of speed through this.
07:25This is the least interesting, as you might guess uh um from the title of my talk.
07:30But, what happens when engineers switch tools?
07:33We saw what happened when they were kind of encouraged, let's say, to switch tools, uh but engineers uh switch the models they use all the time, they switch harnesses.
07:43So, you can do things like look at change in the weekly merge PRs.
07:47You can compare an engineer to themselves and look at what happens when they go all in on Opus versus Sonnet, switch to Chat GPT.
07:55Um what happens if they switch from Cursor to Cloud Code, which we have a lot of data around people who've made that switch.
08:00What happens when uh you switch from working totally in an autocomplete mode to an agentic mode.
08:06The story's kind of the same.
08:07Statistically, it's kind of a nothingburger.
08:10There's not a lot here in terms of explaining any meaningful change in um the amount of PR throughput that you get.
08:20What about how well you use it?
08:22This is 3%, so much bigger piece of the pie.
08:25Let's talk about skill use.
08:27This one's interesting because skills This should be bigger.
08:31Skills actually we find matter a lot.
08:33One of the reasons it doesn't matter more is and I don't This might surprise some folks.
08:40People who use skills that much like in the real world outside of like this this wonderful land of AI engineering conferences that that we're inhabiting.
08:49Uh this chart shows um you know, skill use engineer weeks over time.
08:55So, proportion of engineering weeks that have, you know, low zero skill use, a low level, a medium level, and a high level of skill use.
09:02And as of September, you know, almost half of the weeks here shown had essentially no skill use.
09:11Uh and only 12% had are using skills at a level that I think we would consider as like what you should be doing in terms of modern AI engineering.
09:20And the reason that this is so important is because skill use when it's prevalent moves output way more than model choice.
09:29So, I'm showing here again those results I flashed from earlier about changing model family as as kind of a reference.
09:35But on the bottom here is level of skill use and the amount of effect it has in terms of changing the level of PR throughput holding these other factors constant.
09:46And you can see you get um you know, when you get up to 30 plus runs uh of using skills uh per engineer per week, that's a 30% bump just from that factor alone holding other things constant.
10:00So, it's a big effect.
10:01[snorts]
10:04All right, let's talk about how much you use it.
10:07This is consumption.
10:08This is tokens.
10:09This is active use days.
10:12Uh this one I could do a whole talk about this.
10:15In fact, I did a whole talk about this at the AI Engineer World's Fair.
10:18So, I encourage you if you find this interesting to uh to take a look at that.
10:23Um but I'll give you the one slide version.
10:26So, the data you're looking at here is from that talk.
10:29So, this is um about 500,000 developer weeks of behavior.
10:34Each dot is a two percentile bin of uh folks that use the same level of consumption in that week, and it's their average token span and their merged PRs per developer per week.
10:47And the basic story is more tokens, more throughput.
10:51And there's diminishing returns.
10:53So, there's sort of a saturating curve.
10:54I likened it to the rocket equation in that talk.
10:56It's like rocket fuel.
10:58And what you get is if you look at the top 10% versus the median, uh you can basically double your output between the median and the top 10%.
11:07It costs you about 10x the tokens.
11:09But more tokens mean more output.
11:14Now we get to the bigger pieces of the pie.
11:17These are interesting.
11:18So, there's the factor of you know, where you work, your company, your team.
11:23Actually, if you tease out the factors, about a third of it's the team, two-thirds is the company.
11:27You know, your your code base, your review process, um your culture.
11:33So, once you sort of fix the amount of consumption, you know, do these other things still matter?
11:42One way to look at this is, you know, if you fix these factors and just look at the variability, you look at the spread of companies relative to the average company holding these other things fixed.
11:52Um, this is what it looks like.
11:54And one way to interpret this is, I just showed you that token curve.
11:58So, you could spend 10x the tokens to move from the median up to the top 10 percentile and get double the output.
12:04There's also a 2x spread between the top 10% company and the bottom 10% company in this distribution holding your consumption fixed.
12:13So, in other words, your culture, your team, your company, there's a lot of variation just in those things even for fixed AI spend.
12:20This matters a lot and it's interesting for this group in the in the leadership track here because this is what leaders can control.
12:28Like, this is our our job is to control these things even if maybe our CFO tells us we can't have any more tokens than we have.
12:36Um, which hopefully they don't.
12:39So, what about the biggest piece?
12:42You, the individual, right?
12:44This is this three quarters of this statistical model.
12:48So, it matters a lot.
12:49[snorts]
12:50Um, one way of thinking about this is this question that a lot of people have asked is, you know, does AI mean that you need different engineers?
12:58Do we need different people, right?
13:00Is it resorting the rankings?
13:01Is it changing what it means to be a good engineer?
13:05So, uh, you saw that distribution I just showed you with companies.
13:11If you do the same sort of distribution for engineers, it's even wider.
13:14So, there's even more variability and that's why it's such a big factor in this this statistical map we've we've laid out for ourselves to explain the different levers available to us.
13:25Uh, it's even wider.
13:26It's, you know, 3.5x spread between the bottom 10% and the top 10% uh instead of 2x as for companies.
13:34And there's a lot to unpack here.
13:35This will be a whole 'nother talk to unpack all the things that make different uh AI engineers uh better than others.
13:42One thing that we can answer a little bit in the few minutes I've got left is is AI changing who the most uh productive engineers are.
13:52So, one way to look at this, here's a small study ranking 10,000 engineers.
13:57We have long uh you know, longitudinal data, long timeline.
14:00Jellyfish has been around 10 years.
14:01So, we have, you know, tons of history about how engineers behave.
14:04And if you look at, you know, how um top engineers rank uh you know, quarter over quarter, 6 months over 6 months, uh we can look at, you know, how much do your top engineers cycle in terms of productivity here.
14:18Um PR throughput we're still talking about.
14:21How much uh they cycle out of the top quartile.
14:24And if you compare that uh so, you know, about 44% cycle out of the top quartile uh every half year.
14:32Uh if you look at before AI versus after, it actually tightens a little.
14:35People get a little less mobile.
14:37It's 41%.
14:38So, and we've done other studies like this.
14:40We're not seeing um you know, a big re-sorting of the world order.
14:46Your your best engineers before AI by and large are still your best engineers.
14:50So, the the sort of narrative that AI's leveling the the playing field uh there's craft still matters for a lot, and that's why part of why this factor is as big as it is.
15:02So, you know, going back to this this picture that I've I've been showing you.
15:05When you add it all up, you know, what I want you to take away is is these things on the left, you know, the tool especially, are like really exciting to talk about.
15:14It's the cool new tech.
15:15It takes up kind of all the air in the room, all the air on on Twitter.
15:18And like that's exciting, and I love that stuff.
15:21Um but we don't have a lot of control over that.
15:24I don't work at a foundation lab.
15:25I can't make the tools better.
15:27Uh, and as cost tighten, we have um, less and less freedom maybe to buy the tools that we want.
15:34But as leaders, the things we can control are on the other side of the uh, the equation.
15:40It's it's the people, it's the organization.
15:43And I think what's, you know, going back to our golf analogy, what's really exciting about that is unlike golf, you can change the course.
15:50You can change your environment.
15:51So, you can like fill in water hazards or do things that they yell at you if you try to do on a golf course.
15:58But um, those are things as leaders that are available to us.
16:02So, in closing, um, you know, if you're not getting the outcomes that you want, I would encourage you to think, you know, maybe it's not the model, you know, maybe it's you.
16:16So.
16:17[applause]
16:18Thank you.
16:19Appreciate that.
16:20That's all I've got.
16:21If you're interested, we publish monthly benchmarks with all of this data and more.
16:25Go to Jellyfish AI engineering trends.
16:27Uh, a lot of our research is published on our Substack.
16:31Uh, thank you.