WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Ve2bAarPA2I

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Ve2bAarPA2I.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.840 --> 00:00:14.600
I'm I'm Nicholas Arcolano.

s3
00:00:14.600 --> 00:00:16.800
I'm the head of AI and research at Jellyfish.

s4
00:00:16.800 --> 00:00:19.520
It's great to be back here at AI Engineer.

s5
00:00:19.520 --> 00:00:29.720
And today I want to talk about why if you're not getting the outcomes that you want with AI, uh chances are it's actually not the model, the the problem is you.

s6
00:00:29.720 --> 00:00:37.120
Um since we're here, uh you know, we're doing this this finance theme thing in the leadership track, we're going to start with a golf analogy.

s7
00:00:37.120 --> 00:00:38.720
Seems appropriate.

s8
00:00:38.720 --> 00:00:40.520
Uh is anybody here a golfer?

s9
00:00:40.520 --> 00:00:41.160
Any golfers?

s10
00:00:41.160 --> 00:00:42.280
Anybody?

s11
00:00:42.280 --> 00:00:43.680
If you, okay.

s12
00:00:43.680 --> 00:00:47.600
Um so yeah, I if you don't golf, bear with me.

s13
00:00:47.600 --> 00:00:48.280
I apologize.

s14
00:00:48.280 --> 00:00:50.640
If if you do golf, I also apologize.

s15
00:00:50.640 --> 00:00:52.360
It's a very painful sport.

s16
00:00:52.360 --> 00:00:54.280
Um I love golf.

s17
00:00:54.280 --> 00:00:56.360
Uh I'm a bad golfer.

s18
00:00:56.360 --> 00:01:01.560
Um it's part of the culture to say you're a bad golfer, but I I genuinely am a bad golfer.

s19
00:01:01.560 --> 00:01:04.280
And when you're out there, you just want to make the pain go away.

s20
00:01:04.280 --> 00:01:08.040
And one of the things you think of is like, oh, good should I just buy better clubs, right?

s21
00:01:08.040 --> 00:01:10.080
That would make this easier.

s22
00:01:10.080 --> 00:01:13.120
Uh and then you get an instructor and he's like, no, like you're the problem.

s23
00:01:13.120 --> 00:01:15.560
You don't need better clubs, right?

s24
00:01:15.560 --> 00:01:16.920
Uh it's about the fundamentals.

s25
00:01:16.920 --> 00:01:20.600
But, you know, I'm a I'm a data guy, so I'm like, I don't I don't know.

s26
00:01:20.600 --> 00:01:22.120
Is that really true?

s27
00:01:22.120 --> 00:01:24.800
Uh if you look at the data, like clubs matter.

s28
00:01:24.800 --> 00:01:26.480
Clubs have gotten better.

s29
00:01:26.480 --> 00:01:34.160
So, what you're looking at here, this is the average driving distance for the PGA tour uh from 1980 to the present.

s30
00:01:34.160 --> 00:01:35.160
In 1980, I was shocked.

s31
00:01:35.160 --> 00:01:35.960
I actually didn't know this.

s32
00:01:35.960 --> 00:01:37.840
The average was 257 yd.

s33
00:01:37.840 --> 00:01:39.720
Like I've hit the ball that far.

s34
00:01:39.720 --> 00:01:42.480
And as I mentioned, I'm a terrible golfer.

s35
00:01:42.480 --> 00:01:46.400
Uh so it's gone up about 20% since 1980.

s36
00:01:46.400 --> 00:01:55.600
And you know, one bit of evidence that the clubs matter is uh this solid core ball era thing in the middle by uh 2000.

s37
00:01:55.600 --> 00:02:01.280
A bunch of techno- technological changes happened, and there was like a 15-yard bump right at that time.

s38
00:02:01.280 --> 00:02:07.040
So, people didn't magically get better over that 3-year span, like that was all technology-driven.

s39
00:02:07.040 --> 00:02:12.640
And this uh chart reminded me a lot of what we're seeing with the model frontier, right?

s40
00:02:12.640 --> 00:02:16.520
We're seeing the steady advance of models getting better and better.

s41
00:02:16.520 --> 00:02:18.280
This is a chart from artificial analysis.

s42
00:02:18.280 --> 00:02:23.240
I assume everyone has seen one or many of these and checks them every time a new model comes out, right?

s43
00:02:23.240 --> 00:02:26.280
So, the models have gotten better and better.

s44
00:02:26.280 --> 00:02:28.280
Um I think that's uncontroversial.

s45
00:02:28.280 --> 00:02:33.000
We can do things that we just could not do a year ago, right?

s46
00:02:33.000 --> 00:02:35.200
So, the models matter a lot.

s47
00:02:35.200 --> 00:02:36.000
Right?

s48
00:02:36.000 --> 00:02:37.120
Yes?

s49
00:02:37.120 --> 00:02:38.959
Well, I So, I don't know.

s50
00:02:38.959 --> 00:02:40.840
The So, this is interesting, right?

s51
00:02:40.840 --> 00:02:45.040
So, I work um I work at this company called Jellyfish.

s52
00:02:45.040 --> 00:02:53.120
Uh we uh join AI coding tool and agent telemetry with engineering workflow data, engineering outcomes.

s53
00:02:53.120 --> 00:02:59.280
Uh we have the largest data set of its kind, hundreds of thousands of developers, about 1,300 companies.

s54
00:02:59.280 --> 00:03:03.600
And we're tracking sort of all the things they're doing with AI uh in the development space.

s55
00:03:03.600 --> 00:03:17.600
And one thing that I noticed in our data is um this is data from around the time when uh SpaceX announced its acquisition of Cursor, and we noticed uh a huge bump in Grok usage

s56
00:03:17.600 --> 00:03:28.040
uh and the biggest model change that we've ever seen, far faster than just normal engineers sort of changing their preferences and trying to seek out the best model for the job.

s57
00:03:28.040 --> 00:03:38.440
And when we dug into it, what happened was Cursor was doing a lot of things with the UI, sort of promoting Grok up in the uh up in the UI, changing how auto mode worked.

s58
00:03:38.440 --> 00:03:49.360
And so, um you know, it it shot up very fast, and what was exciting about this as being a a data scientist is this is what we call uh kind of a natural experiment.

s59
00:03:49.360 --> 00:03:54.840
There was this external shock, and a natural question to ask would be, well, what what happened?

s60
00:03:54.840 --> 00:03:55.000
Right?

s61
00:03:55.000 --> 00:03:57.520
Did output change, for instance?

s62
00:03:57.520 --> 00:04:05.000
So, what you're looking at here on this chart, this is change in merged PRs per engineer.

s63
00:04:05.000 --> 00:04:09.440
And so, over that time span I showed you, every dot here is an engineer.

s64
00:04:09.440 --> 00:04:18.239
And the x-axis is how much of their model share went from not using Grok to using Grok on a scale from 0 to 100%.

s65
00:04:18.239 --> 00:04:25.480
The y-axis is the relative change in their PR throughput relative to their colleagues at the same company.

s66
00:04:25.480 --> 00:04:27.800
And the story here is a non-story.

s67
00:04:27.800 --> 00:04:29.919
There's the the cloud sort of center around zero.

s68
00:04:29.919 --> 00:04:31.400
There's no statistical correlation.

s69
00:04:31.400 --> 00:04:32.840
Output didn't change.

s70
00:04:32.840 --> 00:04:37.400
Sort of shunting everyone to Grok didn't really matter.

s71
00:04:37.400 --> 00:04:44.320
So, I left this experiment confused and wanting to know how much does the model matter.

s72
00:04:44.320 --> 00:04:49.600
And if it doesn't matter much, what are the other things that do actually matter?

s73
00:04:49.600 --> 00:04:53.520
So, that's what we're going to spend the rest of the talk talking about.

s74
00:04:53.520 --> 00:04:56.320
Uh So, here's what we're going to do.

s75
00:04:56.320 --> 00:05:02.240
We're going to continue to focus on this output metric of merged PRs per engineer per week.

s76
00:05:02.240 --> 00:05:04.880
We're going to use a curated data set here.

s77
00:05:04.880 --> 00:05:08.840
This is a subset of our data where we've got all the pieces we need to do this experiment.

s78
00:05:08.840 --> 00:05:12.320
About 50,000 engineers at 447 companies.

s79
00:05:12.320 --> 00:05:15.760
Data from this year for Cursor and Claude code users.

s80
00:05:15.760 --> 00:05:17.960
And some of the stuff that matters.

s81
00:05:17.960 --> 00:05:18.840
So, you've got the tool.

s82
00:05:18.840 --> 00:05:22.520
We're going to put the harness and the model together because they're very intertwined.

s83
00:05:22.520 --> 00:05:24.720
This is like your clubs, okay?

s84
00:05:24.720 --> 00:05:26.240
There's how well you use it.

s85
00:05:26.240 --> 00:05:37.560
By that, I mean things like your skill use, cursor rules, the the harness customization, the tricks of the trade that we're all learning as AI engineers to make the way that we use these tools better.

s86
00:05:37.560 --> 00:05:42.200
Golf equivalent would be like this is knowing how to get out of a bunker, which is not like a natural thing.

s87
00:05:42.200 --> 00:05:44.080
It's a thing you have to learn.

s88
00:05:44.080 --> 00:05:45.240
There's how much you use it.

s89
00:05:45.240 --> 00:05:49.360
So, separate from these things, just raw amount of token spend.

s90
00:05:49.360 --> 00:05:52.480
So, golf time on the course, right?

s91
00:05:52.480 --> 00:05:56.600
Uh and then there's factors like where you work, your environment.

s92
00:05:56.600 --> 00:06:02.400
So, your company and your team, this is the course itself, something you don't have control over as a golfer.

s93
00:06:02.400 --> 00:06:12.699
And then there's you, the individual, all the things that you carry with yourself as an engineer that, holding all these other factors constant, that make you still distinct.

s94
00:06:12.699 --> 00:06:13.960
[snorts]

s95
00:06:13.960 --> 00:06:15.720
So, this is the map for the rest of our talk.

s96
00:06:15.720 --> 00:06:23.680
You put all this stuff together, you do a bunch of big statistical analyses, the details of which I would love to get into here, but would bore everyone except for

s97
00:06:23.680 --> 00:06:25.280
statisticians like me.

s98
00:06:25.280 --> 00:06:26.440
But, this is the map.

s99
00:06:26.440 --> 00:06:39.320
You've got about uh less than 1% the in terms of how your how much your PR throughput changes is controlled, explained by the tool.

s100
00:06:39.320 --> 00:06:43.080
How well you use it, the skills part, that's about 3%.

s101
00:06:43.080 --> 00:06:48.800
Uh how much you use it, that's like 10x uh the tool, that's 9%.

s102
00:06:48.800 --> 00:06:50.040
Where you work is 12%.

s103
00:06:50.040 --> 00:06:52.919
The biggest piece is you, the individual.

s104
00:06:52.919 --> 00:06:58.160
We'll go through each of these and explain a little bit about why this looks like this.

s105
00:07:10.200 --> 00:07:10.760
Mhm.

s106
00:07:10.760 --> 00:07:14.640
No, there's no quality metric here, but quality's a thing we look at a lot.

s107
00:07:14.640 --> 00:07:17.544
I'm happy to talk more about that later, but good question.

s108
00:07:17.544 --> 00:07:19.280
[snorts]

s109
00:07:19.280 --> 00:07:23.040
Uh so, let's do the tool first.

s110
00:07:23.480 --> 00:07:25.600
So, we'll kind of speed through this.

s111
00:07:25.600 --> 00:07:30.280
This is the least interesting, as you might guess uh um from the title of my talk.

s112
00:07:30.280 --> 00:07:33.160
But, what happens when engineers switch tools?

s113
00:07:33.160 --> 00:07:43.240
We saw what happened when they were kind of encouraged, let's say, to switch tools, uh but engineers uh switch the models they use all the time, they switch harnesses.

s114
00:07:43.240 --> 00:07:47.080
So, you can do things like look at change in the weekly merge PRs.

s115
00:07:47.080 --> 00:07:55.160
You can compare an engineer to themselves and look at what happens when they go all in on Opus versus Sonnet, switch to Chat GPT.

s116
00:07:55.160 --> 00:08:00.120
Um what happens if they switch from Cursor to Cloud Code, which we have a lot of data around people who've made that switch.

s117
00:08:00.120 --> 00:08:06.360
What happens when uh you switch from working totally in an autocomplete mode to an agentic mode.

s118
00:08:06.360 --> 00:08:07.520
The story's kind of the same.

s119
00:08:07.520 --> 00:08:10.040
Statistically, it's kind of a nothingburger.

s120
00:08:10.040 --> 00:08:19.520
There's not a lot here in terms of explaining any meaningful change in um the amount of PR throughput that you get.

s121
00:08:20.480 --> 00:08:22.480
What about how well you use it?

s122
00:08:22.480 --> 00:08:25.400
This is 3%, so much bigger piece of the pie.

s123
00:08:25.400 --> 00:08:27.320
Let's talk about skill use.

s124
00:08:27.320 --> 00:08:31.040
This one's interesting because skills This should be bigger.

s125
00:08:31.040 --> 00:08:33.640
Skills actually we find matter a lot.

s126
00:08:33.640 --> 00:08:40.039
One of the reasons it doesn't matter more is and I don't This might surprise some folks.

s127
00:08:40.039 --> 00:08:49.480
People who use skills that much like in the real world outside of like this this wonderful land of AI engineering conferences that that we're inhabiting.

s128
00:08:49.480 --> 00:08:55.200
Uh this chart shows um you know, skill use engineer weeks over time.

s129
00:08:55.200 --> 00:09:02.920
So, proportion of engineering weeks that have, you know, low zero skill use, a low level, a medium level, and a high level of skill use.

s130
00:09:02.920 --> 00:09:11.880
And as of September, you know, almost half of the weeks here shown had essentially no skill use.

s131
00:09:11.880 --> 00:09:20.320
Uh and only 12% had are using skills at a level that I think we would consider as like what you should be doing in terms of modern AI engineering.

s132
00:09:20.320 --> 00:09:29.240
And the reason that this is so important is because skill use when it's prevalent moves output way more than model choice.

s133
00:09:29.240 --> 00:09:35.920
So, I'm showing here again those results I flashed from earlier about changing model family as as kind of a reference.

s134
00:09:35.920 --> 00:09:46.560
But on the bottom here is level of skill use and the amount of effect it has in terms of changing the level of PR throughput holding these other factors constant.

s135
00:09:46.560 --> 00:10:00.200
And you can see you get um you know, when you get up to 30 plus runs uh of using skills uh per engineer per week, that's a 30% bump just from that factor alone holding other things constant.

s136
00:10:00.200 --> 00:10:01.951
So, it's a big effect.

s137
00:10:01.951 --> 00:10:03.951
[snorts]

s138
00:10:04.080 --> 00:10:07.000
All right, let's talk about how much you use it.

s139
00:10:07.000 --> 00:10:08.240
This is consumption.

s140
00:10:08.240 --> 00:10:09.280
This is tokens.

s141
00:10:09.280 --> 00:10:12.400
This is active use days.

s142
00:10:12.400 --> 00:10:15.440
Uh this one I could do a whole talk about this.

s143
00:10:15.440 --> 00:10:18.920
In fact, I did a whole talk about this at the AI Engineer World's Fair.

s144
00:10:18.920 --> 00:10:23.880
So, I encourage you if you find this interesting to uh to take a look at that.

s145
00:10:23.880 --> 00:10:26.280
Um but I'll give you the one slide version.

s146
00:10:26.280 --> 00:10:29.280
So, the data you're looking at here is from that talk.

s147
00:10:29.280 --> 00:10:34.600
So, this is um about 500,000 developer weeks of behavior.

s148
00:10:34.600 --> 00:10:47.400
Each dot is a two percentile bin of uh folks that use the same level of consumption in that week, and it's their average token span and their merged PRs per developer per week.

s149
00:10:47.400 --> 00:10:51.520
And the basic story is more tokens, more throughput.

s150
00:10:51.520 --> 00:10:53.000
And there's diminishing returns.

s151
00:10:53.000 --> 00:10:54.720
So, there's sort of a saturating curve.

s152
00:10:54.720 --> 00:10:56.880
I likened it to the rocket equation in that talk.

s153
00:10:56.880 --> 00:10:58.360
It's like rocket fuel.

s154
00:10:58.360 --> 00:11:07.480
And what you get is if you look at the top 10% versus the median, uh you can basically double your output between the median and the top 10%.

s155
00:11:07.480 --> 00:11:09.160
It costs you about 10x the tokens.

s156
00:11:09.160 --> 00:11:13.080
But more tokens mean more output.

s157
00:11:14.280 --> 00:11:17.520
Now we get to the bigger pieces of the pie.

s158
00:11:17.520 --> 00:11:18.560
These are interesting.

s159
00:11:18.560 --> 00:11:23.480
So, there's the factor of you know, where you work, your company, your team.

s160
00:11:23.480 --> 00:11:27.839
Actually, if you tease out the factors, about a third of it's the team, two-thirds is the company.

s161
00:11:27.839 --> 00:11:33.200
You know, your your code base, your review process, um your culture.

s162
00:11:33.200 --> 00:11:41.600
So, once you sort of fix the amount of consumption, you know, do these other things still matter?

s163
00:11:42.120 --> 00:11:52.680
One way to look at this is, you know, if you fix these factors and just look at the variability, you look at the spread of companies relative to the average company holding these other things fixed.

s164
00:11:52.680 --> 00:11:54.840
Um, this is what it looks like.

s165
00:11:54.840 --> 00:11:58.000
And one way to interpret this is, I just showed you that token curve.

s166
00:11:58.000 --> 00:12:04.960
So, you could spend 10x the tokens to move from the median up to the top 10 percentile and get double the output.

s167
00:12:04.960 --> 00:12:13.520
There's also a 2x spread between the top 10% company and the bottom 10% company in this distribution holding your consumption fixed.

s168
00:12:13.520 --> 00:12:20.440
So, in other words, your culture, your team, your company, there's a lot of variation just in those things even for fixed AI spend.

s169
00:12:20.440 --> 00:12:28.080
This matters a lot and it's interesting for this group in the in the leadership track here because this is what leaders can control.

s170
00:12:28.080 --> 00:12:36.320
Like, this is our our job is to control these things even if maybe our CFO tells us we can't have any more tokens than we have.

s171
00:12:36.320 --> 00:12:39.120
Um, which hopefully they don't.

s172
00:12:39.120 --> 00:12:42.040
So, what about the biggest piece?

s173
00:12:42.040 --> 00:12:44.440
You, the individual, right?

s174
00:12:44.440 --> 00:12:48.120
This is this three quarters of this statistical model.

s175
00:12:48.120 --> 00:12:49.633
So, it matters a lot.

s176
00:12:49.633 --> 00:12:50.000
[snorts]

s177
00:12:50.000 --> 00:12:58.720
Um, one way of thinking about this is this question that a lot of people have asked is, you know, does AI mean that you need different engineers?

s178
00:12:58.720 --> 00:13:00.360
Do we need different people, right?

s179
00:13:00.360 --> 00:13:01.839
Is it resorting the rankings?

s180
00:13:01.839 --> 00:13:05.480
Is it changing what it means to be a good engineer?

s181
00:13:05.800 --> 00:13:11.320
So, uh, you saw that distribution I just showed you with companies.

s182
00:13:11.320 --> 00:13:14.839
If you do the same sort of distribution for engineers, it's even wider.

s183
00:13:14.839 --> 00:13:25.360
So, there's even more variability and that's why it's such a big factor in this this statistical map we've we've laid out for ourselves to explain the different levers available to us.

s184
00:13:25.360 --> 00:13:26.200
Uh, it's even wider.

s185
00:13:26.200 --> 00:13:34.080
It's, you know, 3.5x spread between the bottom 10% and the top 10% uh instead of 2x as for companies.

s186
00:13:34.080 --> 00:13:35.760
And there's a lot to unpack here.

s187
00:13:35.760 --> 00:13:42.760
This will be a whole 'nother talk to unpack all the things that make different uh AI engineers uh better than others.

s188
00:13:42.760 --> 00:13:52.280
One thing that we can answer a little bit in the few minutes I've got left is is AI changing who the most uh productive engineers are.

s189
00:13:52.320 --> 00:13:57.160
So, one way to look at this, here's a small study ranking 10,000 engineers.

s190
00:13:57.160 --> 00:14:00.080
We have long uh you know, longitudinal data, long timeline.

s191
00:14:00.080 --> 00:14:01.400
Jellyfish has been around 10 years.

s192
00:14:01.400 --> 00:14:04.800
So, we have, you know, tons of history about how engineers behave.

s193
00:14:04.800 --> 00:14:18.520
And if you look at, you know, how um top engineers rank uh you know, quarter over quarter, 6 months over 6 months, uh we can look at, you know, how much do your top engineers cycle in terms of productivity here.

s194
00:14:18.520 --> 00:14:21.000
Um PR throughput we're still talking about.

s195
00:14:21.000 --> 00:14:24.360
How much uh they cycle out of the top quartile.

s196
00:14:24.360 --> 00:14:32.320
And if you compare that uh so, you know, about 44% cycle out of the top quartile uh every half year.

s197
00:14:32.320 --> 00:14:35.280
Uh if you look at before AI versus after, it actually tightens a little.

s198
00:14:35.280 --> 00:14:37.200
People get a little less mobile.

s199
00:14:37.200 --> 00:14:38.720
It's 41%.

s200
00:14:38.720 --> 00:14:40.839
So, and we've done other studies like this.

s201
00:14:40.839 --> 00:14:46.360
We're not seeing um you know, a big re-sorting of the world order.

s202
00:14:46.360 --> 00:14:50.839
Your your best engineers before AI by and large are still your best engineers.

s203
00:14:50.839 --> 00:15:01.920
So, the the sort of narrative that AI's leveling the the playing field uh there's craft still matters for a lot, and that's why part of why this factor is as big as it is.

s204
00:15:02.200 --> 00:15:05.880
So, you know, going back to this this picture that I've I've been showing you.

s205
00:15:05.880 --> 00:15:14.360
When you add it all up, you know, what I want you to take away is is these things on the left, you know, the tool especially, are like really exciting to talk about.

s206
00:15:14.360 --> 00:15:15.320
It's the cool new tech.

s207
00:15:15.320 --> 00:15:18.960
It takes up kind of all the air in the room, all the air on on Twitter.

s208
00:15:18.960 --> 00:15:21.560
And like that's exciting, and I love that stuff.

s209
00:15:21.560 --> 00:15:24.080
Um but we don't have a lot of control over that.

s210
00:15:24.080 --> 00:15:25.680
I don't work at a foundation lab.

s211
00:15:25.680 --> 00:15:27.960
I can't make the tools better.

s212
00:15:27.960 --> 00:15:34.080
Uh, and as cost tighten, we have um, less and less freedom maybe to buy the tools that we want.

s213
00:15:34.080 --> 00:15:40.520
But as leaders, the things we can control are on the other side of the uh, the equation.

s214
00:15:40.520 --> 00:15:43.240
It's it's the people, it's the organization.

s215
00:15:43.240 --> 00:15:50.440
And I think what's, you know, going back to our golf analogy, what's really exciting about that is unlike golf, you can change the course.

s216
00:15:50.440 --> 00:15:51.880
You can change your environment.

s217
00:15:51.880 --> 00:15:58.000
So, you can like fill in water hazards or do things that they yell at you if you try to do on a golf course.

s218
00:15:58.000 --> 00:16:02.560
But um, those are things as leaders that are available to us.

s219
00:16:02.560 --> 00:16:15.240
So, in closing, um, you know, if you're not getting the outcomes that you want, I would encourage you to think, you know, maybe it's not the model, you know, maybe it's you.

s220
00:16:16.000 --> 00:16:17.271
So.

s221
00:16:17.271 --> 00:16:18.920
[applause]

s222
00:16:18.920 --> 00:16:19.480
Thank you.

s223
00:16:19.480 --> 00:16:20.320
Appreciate that.

s224
00:16:20.320 --> 00:16:21.120
That's all I've got.

s225
00:16:21.120 --> 00:16:25.040
If you're interested, we publish monthly benchmarks with all of this data and more.

s226
00:16:25.040 --> 00:16:27.720
Go to Jellyfish AI engineering trends.

s227
00:16:27.720 --> 00:16:31.160
Uh, a lot of our research is published on our Substack.

s228
00:16:31.160 --> 00:16:33.839
Uh, thank you.