WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=kspZP5BgkEU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/kspZP5BgkEU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.520 --> 00:00:13.280
My name is Laurie.

s3
00:00:13.280 --> 00:00:16.040
I'm head of developer relations at Rise AI.

s4
00:00:16.040 --> 00:00:21.520
Uh in a former life I co-founded NPM Inc. So some of may you some of you may remember me from JavaScript days.

s5
00:00:21.520 --> 00:00:28.480
Uh but these days I think about AI agents and uh in this case what they are doing to our jobs.

s6
00:00:28.480 --> 00:00:41.880
Uh so a little bit over 3 years ago um uh Swix wrote an essay called the rise of the AI engineer and his argument was that when it comes to shipping AI products you want engineers not researchers.

s7
00:00:41.880 --> 00:00:47.080
I think that is a message that resonated with everybody especially who the people who come to this conference.

s8
00:00:47.080 --> 00:00:53.520
Uh and so he created the job title of the AI engineer and companies since then have been hiring for it.

s9
00:00:53.520 --> 00:01:01.520
Um and part of what I'm here to do today to do is to convince you uh that he did not in fact go far enough.

s10
00:01:01.520 --> 00:01:05.239
Uh there's not just an AI engineer revolution happening.

s11
00:01:05.239 --> 00:01:09.040
There is a revolution happening to every kind of engineer.

s12
00:01:09.040 --> 00:01:17.840
Um all engineering jobs are being replaced by a new type of job and not everybody is going to be happy to get that news.

s13
00:01:17.840 --> 00:01:25.840
Um the underlying change that kicked all of this off is that agents have made writing code basically free.

s14
00:01:25.840 --> 00:01:31.680
Like you pay for the tokens but relative to the cost of the time it used to take you uh it is basically free.

s15
00:01:31.680 --> 00:01:36.280
Um everything that comes after writing the code did not become free at the same time.

s16
00:01:36.280 --> 00:01:46.480
So the the bottleneck moved from writing code is really slow and all of this other stuff happens at roughly the same speed to writing code is super fast uh and now we get stuck behind new things.

s17
00:01:46.480 --> 00:01:51.680
So the first the thing it's stuck behind right this second like this month is code review.

s18
00:01:51.680 --> 00:01:57.280
Um uh but my thesis is that it's not going to stop there.

s19
00:01:57.280 --> 00:01:59.960
We are going to figure out how to automate code review.

s20
00:01:59.960 --> 00:02:10.360
We are going to make uh code review move at the speed of agents, and then we are going to run into another bottleneck, and another bottleneck, and another bottleneck, and at some point we're going to stop.

s21
00:02:10.360 --> 00:02:12.800
We're going to hit one that we can't get past.

s22
00:02:12.800 --> 00:02:17.480
Uh and that is what is going to be left for the humans to do.

s23
00:02:17.480 --> 00:02:20.000
Whatever the bottleneck is that we can't get agents past.

s24
00:02:20.000 --> 00:02:27.120
So, I'm making a bet that I can think of a thing that agents really are not going to be able to do.

s25
00:02:27.120 --> 00:02:39.160
Uh and then once we've come to that bottleneck, we're going to have to figure out how do we train the next generation of engineers to be able to do that job because we're not training them to do that job right now.

s26
00:02:39.160 --> 00:02:40.959
So, a quick warning {slash} apology.

s27
00:02:40.959 --> 00:02:44.680
I have 18 minutes because that is how long Swiss gives for talks.

s28
00:02:44.680 --> 00:02:47.200
Uh and I have a really big pile of data.

s29
00:02:47.200 --> 00:02:49.000
I did a ton of research for this.

s30
00:02:49.000 --> 00:02:50.840
Uh and I don't have time to give it all to you.

s31
00:02:50.840 --> 00:02:53.360
I don't have time to like quote my sources and do all that stuff.

s32
00:02:53.360 --> 00:03:01.080
So, I'm just going to throw numbers at you with a bunch of citations on the slides, and I'm just going to have to sort of ask you to trust me, bro.

s33
00:03:01.080 --> 00:03:02.092
Uh

s34
00:03:02.092 --> 00:03:03.040
[laughter]

s35
00:03:03.040 --> 00:03:11.200
Uh so, developers who switch on autonomous coding agents write 741% more code, eight times as much.

s36
00:03:11.200 --> 00:03:14.000
Uh and but they ship only 30% faster.

s37
00:03:14.000 --> 00:03:15.760
Like, 30% faster is not nothing.

s38
00:03:15.760 --> 00:03:17.200
30% faster is amazing.

s39
00:03:17.200 --> 00:03:26.200
We are shipping a lot more code than we used to, but given that we're writing eight times more code, only a third more actual shipping is kind of bad, right?

s40
00:03:26.200 --> 00:03:28.519
It means that we're stuck somewhere.

s41
00:03:28.519 --> 00:03:30.519
Uh and where we're stuck is code review.

s42
00:03:30.519 --> 00:03:35.920
Um and the reason that we're stuck is because humans can't just review faster.

s43
00:03:35.920 --> 00:03:45.480
Uh there is an enormous study done by Cisco 20 years ago about how fast you can do code review, and how accurately you can do code review.

s44
00:03:45.480 --> 00:03:56.720
Um and it turns out that humans' ability to like parse what's going on in code falls off radically after about 450 lines of code per hour.

s45
00:03:56.720 --> 00:04:04.880
So, if you've produced a 10,000 line diff with your agent, it would take you like a week to reasonably review that code.

s46
00:04:04.880 --> 00:04:08.240
And the answer is that nobody's actually doing that, right?

s47
00:04:08.240 --> 00:04:12.360
Cuz we're all throwing 10,000 diffs around willy-nilly these days.

s48
00:04:12.360 --> 00:04:18.120
Um so, I did a whole talk about this phenomenon at a previous AIE, and that talk is called the death of the code review.

s49
00:04:18.120 --> 00:04:22.440
And if you want all of the information about the death of the code review, you can watch that talk.

s50
00:04:22.440 --> 00:04:31.600
Um but the problem facing us right now, uh which is not the same as the problem I'm trying to solve with this overall with this talk, is that the cost of producing

s51
00:04:31.600 --> 00:04:38.800
plausible code has collapsed while the cost of knowing whether to trust that code has stayed exactly the same.

s52
00:04:38.800 --> 00:04:43.919
Um so, the obvious review move right now is to make the machines do the review, like I said.

s53
00:04:43.919 --> 00:04:45.440
And that is already happening.

s54
00:04:45.440 --> 00:04:52.120
Um GitHub's Copilot uh does more than one in five of the code reviews on GitHub as of last month.

s55
00:04:52.120 --> 00:05:00.560
Uh you can, if you want to, opt in to let uh Copilot be the final reviewer before something goes to uh production.

s56
00:05:00.560 --> 00:05:05.000
You can make it have the approval power, if you want to, which is crazy.

s57
00:05:05.000 --> 00:05:08.760
I don't know who's turning on that option, but you can do it if you want to.

s58
00:05:08.760 --> 00:05:15.360
Um OpenAI says that at at OpenAI, humans may review their pull requests, but aren't required to.

s59
00:05:15.360 --> 00:05:19.600
Um but the problem is that the machine reviewers aren't very good yet.

s60
00:05:19.600 --> 00:05:30.400
Uh the data says that there's uh the data that I found put four AI reviewers through their paces, and they all found a bunch of bugs, but none of the same bugs.

s61
00:05:30.400 --> 00:05:33.560
They did not agree on any individual bug.

s62
00:05:33.560 --> 00:05:36.120
Uh they all found actual issues in the code.

s63
00:05:36.120 --> 00:05:38.600
They all found different issues in the code.

s64
00:05:38.600 --> 00:05:43.919
Um so, they're worth running, but if you are serious about this, you're never going to run just one of them.

s65
00:05:43.919 --> 00:05:47.120
Uh because they are not reliable enough yet.

s66
00:05:47.120 --> 00:05:51.480
Um And there are some people who went all the way.

s67
00:05:51.480 --> 00:05:53.800
They were like, "I'm just not going to review the code anymore.

s68
00:05:53.800 --> 00:05:57.000
I'm just going to assume my agent has one shot at this perfectly."

s69
00:05:57.000 --> 00:06:01.800
Um Dexter Horty, who is here, I don't know if he's in the audience, probably not.

s70
00:06:01.800 --> 00:06:09.160
Um uh he spent 6 months telling people not to read the code, and then after that 6 months, he got on stage and said, "Nope, I was wrong.

s71
00:06:09.160 --> 00:06:12.600
You can't actually uh skip code review entirely.

s72
00:06:12.600 --> 00:06:13.360
We tried that.

s73
00:06:13.360 --> 00:06:14.240
It was a disaster.

s74
00:06:14.240 --> 00:06:16.640
We had to rip replace whole systems."

s75
00:06:16.640 --> 00:06:19.000
Uh that's not an option.

s76
00:06:19.000 --> 00:06:20.080
The agents are not good enough.

s77
00:06:20.080 --> 00:06:22.240
They're not one shotting the code.

s78
00:06:22.240 --> 00:06:33.640
Um Amazon had a run of outages uh that they did postmortems on and decided that it was because their coding agent had produced a bunch of slop and they just released it into the wild without thinking.

s79
00:06:33.640 --> 00:06:36.080
Uh and so now they don't let them do that anymore.

s80
00:06:36.080 --> 00:06:39.960
Uh senior engineers must sign off on code reviews at Amazon.

s81
00:06:39.960 --> 00:06:46.480
Um and Anthropic published a postmortem uh on bugs that had got past every reviewer that they had, both human and machine.

s82
00:06:46.480 --> 00:06:53.160
So, the problem is that the automations that we've built so far to check whether our code is good enough are not themselves good enough.

s83
00:06:53.160 --> 00:06:59.880
Um Claude Fable 5 passes the tests uh on 80% of a standard coding benchmark.

s84
00:06:59.880 --> 00:07:08.320
Um and co- So, Cognition, the makers of Devin, um they built a benchmark that asks a different question, which is not does it pass the tests?

s85
00:07:08.320 --> 00:07:14.520
The question is would you as the reviewer, as the maintainer of this repo, merge this PR?

s86
00:07:14.520 --> 00:07:21.560
And if you ask that question instead, the pass rate falls from 87% uh to 54%.

s87
00:07:21.560 --> 00:07:28.240
Um because there's a lot of extra information that goes into deciding whether you would merge it.

s88
00:07:28.240 --> 00:07:35.160
There's code style, there's side effects, there's external systems, there's history that is not encoded into the code base.

s89
00:07:35.160 --> 00:07:38.120
It is encoded into the brain of the maintainer.

s90
00:07:38.120 --> 00:07:47.920
Um So, what the maintainer is adding, what maintainer is demonstrably adding, is something that we haven't given the agents yet.

s91
00:07:47.920 --> 00:07:51.680
There is information they don't have, there is a skill they don't have.

s92
00:07:51.680 --> 00:07:55.320
Um and it is the sense of what good means.

s93
00:07:55.320 --> 00:07:58.440
That is what we are That is what the agents don't have yet.

s94
00:07:58.440 --> 00:08:01.600
Uh and that is a critical insight that I'm going to come back to.

s95
00:08:01.600 --> 00:08:11.560
Um Eliyahu Goldratt, if I'm pronouncing that correctly, was a physicist who became a management guru and in 1984 he wrote a a novel about running a factory.

s96
00:08:11.560 --> 00:08:14.200
Does that sound like an interesting novel to you?

s97
00:08:14.200 --> 00:08:16.160
Um it is called The Goal.

s98
00:08:16.160 --> 00:08:19.400
Um and his big idea was the theory of constraints.

s99
00:08:19.400 --> 00:08:26.400
Uh every system has one bottleneck and speeding things up does not work unless the thing that you were speeding up is the bottleneck.

s100
00:08:26.400 --> 00:08:30.120
Uh that is our problem with code review.

s101
00:08:30.120 --> 00:08:33.360
Uh we've sped up the code and we have not sped up the review.

s102
00:08:33.360 --> 00:08:42.440
Um his second idea is the one that matters today, which is that when you fix the bottleneck, it doesn't go away, it just moves and finds another bottleneck to pile up against.

s103
00:08:42.440 --> 00:08:53.520
Um so let's assume, despite everything that I've said about how bad the code reviewers are right now, that we figured that out, that we get automated code review to a higher standard

s104
00:08:53.520 --> 00:08:55.320
and we move that bottleneck.

s105
00:08:55.320 --> 00:08:59.720
What is the next bottleneck uh that it is going to run into?

s106
00:08:59.720 --> 00:09:04.200
Um if we figure out review, the next bottleneck is probably going to be operations.

s107
00:09:04.200 --> 00:09:05.600
How do I deploy this thing?

s108
00:09:05.600 --> 00:09:07.400
How do I scale it up?

s109
00:09:07.400 --> 00:09:10.280
Uh and then you're probably going to run into security.

s110
00:09:10.280 --> 00:09:13.360
How do I make this thing safe at scale?

s111
00:09:13.360 --> 00:09:15.320
Uh how do I make sure that this thing is private?

s112
00:09:15.320 --> 00:09:18.280
How do I make sure that it's not radically going off the rails?

s113
00:09:18.280 --> 00:09:23.800
Uh but I'm going to make and this is the big leap that I'm making in this talk.

s114
00:09:23.800 --> 00:09:32.160
I'm going to assume we figured those things out as well because those things are very heavily solved by humans.

s115
00:09:32.160 --> 00:09:34.960
There's an enormous amount of data out there about them.

s116
00:09:34.960 --> 00:09:46.360
If we can figure out how to do operations, how to do scaling, how to do security using progressively more advanced agents, what is it that we are left to do?

s117
00:09:46.360 --> 00:09:49.640
Like, what is the role of a human in that situation?

s118
00:09:49.640 --> 00:09:54.400
Um it's also it's a big question [clears throat] and it's also a long-term question.

s119
00:09:54.400 --> 00:09:56.760
I'm not telling you what is going to happen in the next 12 months.

s120
00:09:56.760 --> 00:10:00.760
In the next 12 months, I think we'll probably make some real progress on code reviews.

s121
00:10:00.760 --> 00:10:05.040
I do not think 12 months from now we'll have sorted out operations and scaling and security.

s122
00:10:05.040 --> 00:10:09.080
I think that is going to take somewhere between 5 and 10 years to get done.

s123
00:10:09.080 --> 00:10:12.680
Uh despite people who are like, "AGI 2027."

s124
00:10:12.680 --> 00:10:16.080
No, we're going to slowly iterate on this stuff.

s125
00:10:16.080 --> 00:10:26.880
Um But the other assumption that you need to make for my for the my thesis to be correct is that the amount of software that we need is effectively infinite.

s126
00:10:26.880 --> 00:10:29.160
And I feel that one is a much easier case to make.

s127
00:10:29.160 --> 00:10:36.320
Think of every website you've ever run into uh in the world in production, and it's probably pretty bad.

s128
00:10:36.320 --> 00:10:41.000
Uh and the reason it's bad is not because we don't know how to make really great websites.

s129
00:10:41.000 --> 00:10:45.080
The problem is that there's not enough people to make all of the websites good.

s130
00:10:45.080 --> 00:10:53.839
Every time you run into a terrible website at your hairdresser or the government or like, you know, your bank, like it's not because there's not somebody who knows how to make that website good.

s131
00:10:53.839 --> 00:11:00.880
It's because A, it's very expensive to make it good, and B, the people who are doing that are busy doing other things.

s132
00:11:00.880 --> 00:11:03.280
Uh you know, probably shipping AI agents.

s133
00:11:03.280 --> 00:11:13.560
Um so assume that we can make as much software as we want basically for free, and that the appetite for making software is essentially infinite.

s134
00:11:13.560 --> 00:11:15.560
That is a wonderful situation in which to be in.

s135
00:11:15.560 --> 00:11:19.120
We've got infinite demand and also suddenly infinite supply.

s136
00:11:19.120 --> 00:11:21.320
Uh where does the bottleneck end up?

s137
00:11:21.320 --> 00:11:23.960
What do we think the agents are not going to eat?

s138
00:11:23.960 --> 00:11:31.480
Um to try and figure that out, I split up the job of software development into as many pieces as I could think of, and I sorted them by how soon the agents

s139
00:11:31.480 --> 00:11:32.960
uh get to them.

s140
00:11:32.960 --> 00:11:37.080
Uh writing code is already collapsed, reviewing and maintaining code is going soon.

s141
00:11:37.080 --> 00:11:42.960
Um shipping to production and scaling are next on the chopping block, although there isn't a lot of data there yet.

s142
00:11:42.960 --> 00:11:54.440
Uh and then there's the stuff that I think is probably safe, which is deciding what to build, uh defining what good means, and finally making it delightful.

s143
00:11:54.440 --> 00:11:57.560
What's left is finding out what people actually want.

s144
00:11:57.560 --> 00:12:03.840
Every piece of software is a formalization of something that a human wanted to get done.

s145
00:12:03.840 --> 00:12:13.600
Uh So, if a customer tells you, "I need to to keep track of my orders," there are 10,000 pieces of software that keep track of your orders.

s146
00:12:13.600 --> 00:12:15.240
That's what they do.

s147
00:12:15.240 --> 00:12:21.280
Uh but only one of those, if you're lucky, is good if the thing that you're doing is you're running a bakery.

s148
00:12:21.280 --> 00:12:24.720
If you're running like an auto parts store, that's going to be different.

s149
00:12:24.720 --> 00:12:30.880
If you are running uh you know, a giant manufacturing concern, it's going to be completely different.

s150
00:12:30.880 --> 00:12:33.240
The That requirement is not sufficient.

s151
00:12:33.240 --> 00:12:39.880
The whole job is figuring out what exactly do you mean when you say that you want to keep track of your orders.

s152
00:12:39.880 --> 00:12:41.280
What does that mean for your business?

s153
00:12:41.280 --> 00:12:42.400
What does it have to do?

s154
00:12:42.400 --> 00:12:44.080
How does that work?

s155
00:12:44.080 --> 00:12:46.440
Uh and that knowledge isn't in the training data.

s156
00:12:46.440 --> 00:12:47.360
It isn't in the weights.

s157
00:12:47.360 --> 00:12:50.480
It is in the mind of the person who runs that business.

s158
00:12:50.480 --> 00:12:56.560
And the only way to get it out is by flapping your meat parts at them and having them flap their meat parts back.

s159
00:12:56.560 --> 00:12:58.080
Like, you have to talk to them.

s160
00:12:58.080 --> 00:13:00.800
That is the only way to get the information out.

s161
00:13:00.800 --> 00:13:06.960
Um And that work has a very important property, which is that it doesn't transfer.

s162
00:13:06.960 --> 00:13:13.680
Uh if you think about what good means for like a calendar app and what good means for a scheduling app, those are solving almost the same problem, right?

s163
00:13:13.680 --> 00:13:17.680
A calendar app and a scheduling [clears throat] app, but they're completely different pieces of software.

s164
00:13:17.680 --> 00:13:26.160
They have totally different requirements and a totally different definition of good, even though [clears throat] those two use cases are so close together, and we have a zillion use cases.

s165
00:13:26.160 --> 00:13:32.209
Um Anytime that you've used software that has a zillion configuration options and it still doesn't

s166
00:13:32.209 --> 00:13:32.440
[snorts]

s167
00:13:32.440 --> 00:13:33.400
do what you need.

s168
00:13:33.400 --> 00:13:35.080
I'm looking at you, Jira.

s169
00:13:35.080 --> 00:13:43.720
Uh like that is the problem you're running into of software that is trying to do everything and still failing at it.

s170
00:13:43.720 --> 00:13:56.600
Um so the bottleneck moves to describing the problem and defining what it means to have solved the problem and my big bet for this talk and for the next 10 years is that is where it stays.

s171
00:13:56.600 --> 00:14:04.720
Uh Fred Brooks, who ran IBM System 360 project and wrote uh The Mythical Man-Month, uh he wrote about this in 1986.

s172
00:14:04.720 --> 00:14:06.920
He split the difficulty of software into two kinds.

s173
00:14:06.920 --> 00:14:16.760
Accidental complexity is the difficulty of the tools or the languages, the typing, the syntax, and then there's the essential complexity, which is the prob- difficulty of the problem itself.

s174
00:14:16.760 --> 00:14:23.960
Uh and he predicted that no single tool would make us 10 times more productive because the essential part is the hard part.

s175
00:14:23.960 --> 00:14:34.520
Uh and Brooks's prediction is looking pretty solid because we made code eight times faster and we still didn't get very much faster at shipping because the hard part is the essential part.

s176
00:14:34.520 --> 00:14:38.520
Um Shawn Grove from OpenAI made the same point uh 2 years ago.

s177
00:14:38.520 --> 00:14:40.320
Uh no, sorry, a year ago.

s178
00:14:40.320 --> 00:14:41.600
Uh what year is it?

s179
00:14:41.600 --> 00:14:50.080
Um he said that code is a lossy projection of the specification and whoever writes the spec, if it's a PM or a lawmaker or an engineer or a marketer,

s180
00:14:50.080 --> 00:14:52.120
they become the programmer.

s181
00:14:52.120 --> 00:15:00.600
Uh and so writing [clears throat] down what good means is where the value is now.

s182
00:15:00.600 --> 00:15:06.240
Uh I'm beginning to run out of time because I haven't timed this well.

s183
00:15:06.240 --> 00:15:14.080
Um anything that you can verify cheaply, you can train against until you beat it if you are model maker.

s184
00:15:14.080 --> 00:15:23.840
Uh so uh Cognition tried to write down because of their benchmark about what what it counts to merge something, does it, you know, is is this code mergeable?

s185
00:15:23.840 --> 00:15:28.840
They were like, let's see if we can figure out a benchmark that defines what mergeable really means.

s186
00:15:28.840 --> 00:15:35.040
Let's see if we can get all of that other information into the benchmark, and then we can train a model on it.

s187
00:15:35.040 --> 00:15:45.160
Um uh Cursor's reviewer BugBot turns every thumbs down that you that your team gives it into a rule, and they have like 44,000 rules now.

s188
00:15:45.160 --> 00:15:53.035
Uh so the moment the maintainer takes knowledge out of their brain and puts it somewhere written down that a model can train on it,

s189
00:15:53.035 --> 00:15:53.600
[laughter]

s190
00:15:53.600 --> 00:15:55.720
uh the bottleneck moves again.

s191
00:15:55.720 --> 00:15:57.480
That is why I think security can go.

s192
00:15:57.480 --> 00:15:59.200
That is why I think operations can go.

s193
00:15:59.200 --> 00:16:03.520
I think we can write those things down eventually, even though we haven't done it yet.

s194
00:16:03.520 --> 00:16:19.520
Uh Andrew Ng, the co-founder of Google Brain and Coursera Coursera, says that one of his team proposed having twice as many product managers as engineers, uh and he said that that was a first in his life, uh but his answer is since then hasn't been hire more PMs.

s195
00:16:19.520 --> 00:16:21.640
It's to blur the jobs together.

s196
00:16:21.640 --> 00:16:33.920
In his words, when an engineer understands users and can make decisions on what to build and build it directly, uh they can execute incredibly quickly, and that is where I think we are beginning to hit on the answer.

s197
00:16:33.920 --> 00:16:38.280
Uh in the 1960s, there there was a job called the systems analyst.

s198
00:16:38.280 --> 00:16:40.040
Their job was not to write code.

s199
00:16:40.040 --> 00:16:44.120
Their job was to talk to the business and find out what the problem was.

s200
00:16:44.120 --> 00:16:52.800
Uh then that analyst turned into something that we call a product manager today, and we've had two professions uh where there used to be one and a half.

s201
00:16:52.800 --> 00:16:56.520
I think what is happening is they are collapsing back into one.

s202
00:16:56.520 --> 00:16:58.440
We are going back to the '60s.

s203
00:16:58.440 --> 00:17:01.520
So that is not an AI engineer.

s204
00:17:01.520 --> 00:17:02.720
They're not just building AI.

s205
00:17:02.720 --> 00:17:04.640
They're not just building with AI.

s206
00:17:04.640 --> 00:17:06.120
They are building everything.

s207
00:17:06.120 --> 00:17:08.319
They are building all software.

s208
00:17:08.319 --> 00:17:11.959
Uh and I've decided to call this job the product engineer.

s209
00:17:11.959 --> 00:17:28.800
This is somebody who finds out what people need, defines what good looks like, and ships it uh with agents doing the typing um Gergo Orosz whose name I have probably butchered he writes the pragmatic engineer newsletter he described

s210
00:17:28.800 --> 00:17:36.560
the product-minded engineer back in 2019 as a nice trait for an engineer to have and what I'm saying now is it has gone from nice to essential.

s211
00:17:36.560 --> 00:17:47.680
It has gone from the from a skill that senior engineers used to have to a skill that every engineer has to have if they're going to be an engineer.

s212
00:17:47.680 --> 00:17:52.040
um And you can see the market already paying for this.

s213
00:17:52.040 --> 00:18:01.520
Forward deployed engineers who look very much like product engineers job postings for them rose 800% in 9 months of 2025.

s214
00:18:01.520 --> 00:18:03.840
They are everywhere everybody is trying to hire them.

s215
00:18:03.840 --> 00:18:08.520
Who in this room is trying to hire a forward deployed engineer show of hands.

s216
00:18:08.520 --> 00:18:11.600
Okay, well more I was expecting more hands oh well.

s217
00:18:11.600 --> 00:18:23.960
uh So uh The average comp for forward deployed forward deployed engineer by the way is $240,000 which is to say that it pays like a senior engineer because it is a senior engineer.

s218
00:18:23.960 --> 00:18:26.320
It's a very useful kind of engineer.

s219
00:18:26.320 --> 00:18:29.000
um But nobody agrees on what to call it yet.

s220
00:18:29.000 --> 00:18:36.440
There are like a thousand different job titles for this thing that I am describing and have decided arbitrarily to call the product engineer.

s221
00:18:36.440 --> 00:18:43.560
um And probably at this point some of you are thinking that isn't programming and you are correct.

s222
00:18:43.560 --> 00:18:44.600
It is not programming.

s223
00:18:44.600 --> 00:18:46.360
The programming is being done by an agent.

s224
00:18:46.360 --> 00:19:00.120
It is however recognizably still software development and we are going to have to wrap our minds around the idea that being a software developer and being a programmer have become divorced

s225
00:19:00.120 --> 00:19:02.000
in 2026.

s226
00:19:02.000 --> 00:19:20.040
um And when I say stuff like well I think the hard part is deciding what good is I realize that I'm falling into a trap that lots of people in AI land fall into, which is all of the parts everything about everything can be automated by AI except the thing that I'm good at because that's too hard.

s227
00:19:20.040 --> 00:19:29.440
I heard a VC say that the other day and I was like, "Buddy, uh like, no. No, someone is going to automate your job any day now.

s228
00:19:29.440 --> 00:19:34.080
Um uh so you have to take this with a grain of salt.

s229
00:19:34.080 --> 00:19:41.600
Am I Am I just thinking that the thing that I enjoy about software development, which is figuring out what the problem is, is the hard part of software?

s230
00:19:41.600 --> 00:19:43.560
You can tell me if I'm wrong.

s231
00:19:43.560 --> 00:19:47.320
Um but say you believe me so far.

s232
00:19:47.320 --> 00:19:51.080
Programming goes to zero, product engineering goes to infinity.

s233
00:19:51.080 --> 00:20:01.280
That brings us to a new and important problem, which is junior developers because the way we used to train a junior developer is we used to give them very well-specified

s234
00:20:01.280 --> 00:20:09.880
tickets that said, "Write this software this way in particular and then give it to me so that I can tell you all the ways that you were wrong."

s235
00:20:09.880 --> 00:20:16.320
Uh and that is gone because that's what we do with the That's what we do with the coding agents now.

s236
00:20:16.320 --> 00:20:19.120
We say, "Write this thing and then I'll tell you how you're wrong."

s237
00:20:19.120 --> 00:20:21.160
And then we have a conversation with them.

s238
00:20:21.160 --> 00:20:25.560
Uh and there's That is not a job that we need a human to do anymore.

s239
00:20:25.560 --> 00:20:34.000
But we still need an enormous number of forward-deployed engineers or uh product engineers and we have to get them from somewhere.

s240
00:20:34.000 --> 00:20:39.400
We can't just abandon the current generation of new developers who've showed up and said, "How do I do this stuff?"

s241
00:20:39.400 --> 00:20:44.880
We're like, "I don't know because I don't know what your job is anymore."

s242
00:20:44.880 --> 00:20:45.824
Uh

s243
00:20:45.824 --> 00:20:45.840
[snorts]

s244
00:20:45.840 --> 00:20:47.440
this is not a theoretical future.

s245
00:20:47.440 --> 00:20:51.400
This is an actual graph of labor statistics since January 2023.

s246
00:20:51.400 --> 00:20:52.640
It is sorted by age.

s247
00:20:52.640 --> 00:20:56.560
The red line is engineers aged 22 to 25.

s248
00:20:56.560 --> 00:20:58.360
Their employment is collapsing.

s249
00:20:58.360 --> 00:21:00.160
We are not hiring junior engineers.

s250
00:21:00.160 --> 00:21:02.120
This is not a thing that I think is going to happen.

s251
00:21:02.120 --> 00:21:04.320
This is a thing that's been happening for 2 years.

s252
00:21:04.320 --> 00:21:06.160
Job market sucks for them.

s253
00:21:06.160 --> 00:21:20.520
Meanwhile, the job for older engineers 41 to 49 has gone way up because senior engineers are still super valuable, more valuable than before, but junior engineers are being screwed and we need to fix that as a profession.

s254
00:21:20.520 --> 00:21:29.000
Uh so if we're going to reap the benefits of a world where we 10x code output and figured out the rest of the software supply chain and figured out what humans need to do,

s255
00:21:29.000 --> 00:21:31.360
uh then we are going to need a lot of people to do that.

s256
00:21:31.360 --> 00:21:34.000
I said that the demand for the software is going to infinity.

s257
00:21:34.000 --> 00:21:35.440
We need infinity people.

s258
00:21:35.440 --> 00:21:37.160
Where are we going to get them?

s259
00:21:37.160 --> 00:21:42.960
Uh we've been creating them accidentally so far and we need to figure out a way to do them on purpose.

s260
00:21:42.960 --> 00:21:51.200
Google has a product uh has a program called the associate product manager program which turns out people that look very much like product engineers.

s261
00:21:51.200 --> 00:21:54.040
They turn out 50 of them per year.

s262
00:21:54.040 --> 00:21:57.400
We need 10,000 of them per year.

s263
00:21:57.400 --> 00:22:04.520
Even if you look at all of the other pro all of the other companies that have something a program like Google's APM, it's being extensively copied.

s264
00:22:04.520 --> 00:22:11.280
They're only turning out about a thousand people a year and we need 10 times that number at minimum uh to keep up with demand.

s265
00:22:11.280 --> 00:22:12.285
Uh

s266
00:22:12.285 --> 00:22:12.800
[snorts]

s267
00:22:12.800 --> 00:22:21.760
universities teach data structures, boot camps teach React, nobody teaches go sit with a baker for a week and come back with a spec and that is the person that we need.

s268
00:22:21.760 --> 00:22:26.600
Um so if you lead an engineering team, you're going to have to grow them on purpose.

s269
00:22:26.600 --> 00:22:32.680
Uh IBM is already redesigning its entry-level role around talking to customers and writing specs instead of programming.

s270
00:22:32.680 --> 00:22:35.640
Um you're going to have to figure out how to do that for yourself.

s271
00:22:35.640 --> 00:22:38.800
You're going to have to figure out how do you train product people?

s272
00:22:38.800 --> 00:22:53.200
Uh and you're going to have to do it very quickly because the time we should have started was two years ago and there's a whole cohort of very sad unemployed juniors right now who have this problem and we are not hiring them and we need to do something about that or we are going to be screwed and

s273
00:22:53.200 --> 00:22:54.760
not just them.

s274
00:22:54.760 --> 00:23:05.800
Uh so if all of that happens, there's one big thing that I don't want to just gloss over or hand wave away, which is a lot of people got into programming

s275
00:23:05.800 --> 00:23:07.920
because they love the craft.

s276
00:23:07.920 --> 00:23:15.680
They love a clean abstraction, the feeling of a hard bug finally giving up, making a machine do exactly what you told it to do.

s277
00:23:15.680 --> 00:23:19.680
They did not sign up to interview bakers about what they want in their lives.

s278
00:23:19.680 --> 00:23:22.280
Like that is not the job that they signed up for.

s279
00:23:22.280 --> 00:23:25.960
Um and they are going to hate this.

s280
00:23:25.960 --> 00:23:27.080
They already hate this.

s281
00:23:27.080 --> 00:23:29.400
They are complaining about it very loudly.

s282
00:23:29.400 --> 00:23:32.000
I have no consolation prize for them.

s283
00:23:32.000 --> 00:23:44.920
Um but uh I do have an observation, which is that for a lot of people who think they love the typing, maybe what you loved was the moment just before you started typing, when you figured out the problem.

s284
00:23:44.920 --> 00:23:46.920
That That part doesn't go away.

s285
00:23:46.920 --> 00:23:48.360
That part is still there.

s286
00:23:48.360 --> 00:23:54.120
That creative spark when you're like, I know how to make this into a repeatable system that glows in my brain.

s287
00:23:54.120 --> 00:23:55.880
That's not going away.

s288
00:23:55.880 --> 00:24:03.320
Uh and the craft of programming will survive uh in the same way that like woodworking survived the furniture factory.

s289
00:24:03.320 --> 00:24:06.680
Like programmers program the way that singers sing, right?

s290
00:24:06.680 --> 00:24:12.160
They do it in an empty room with nobody watching just because they love it and that won't go away.

s291
00:24:12.160 --> 00:24:13.800
You can still do that.

s292
00:24:13.800 --> 00:24:19.960
Um it just won't be the thing that people pay you hundreds of thousands of dollars every year to do.

s293
00:24:19.960 --> 00:24:21.440
Um so here is my prediction.

s294
00:24:21.440 --> 00:24:23.560
The next 10 years are going to be ugly.

s295
00:24:23.560 --> 00:24:25.560
The last 2 years have been ugly.

s296
00:24:25.560 --> 00:24:29.280
Um the load of this stuff is arriving before the tools are ready.

s297
00:24:29.280 --> 00:24:39.920
The junior ladder is gone before we have figured out how to train more juniors uh and a lot of people who love the craft are going to have to decide if they love the job that is replacing it.

s298
00:24:39.920 --> 00:24:41.800
Um but this has happened before.

s299
00:24:41.800 --> 00:24:52.480
Compilers and then frameworks and then open source each made a generations worth of typing before them uh unnecessary and each time we came out of it with more software developers.

s300
00:24:52.480 --> 00:24:55.560
We just changed what software development meant.

s301
00:24:55.560 --> 00:24:58.200
Uh and I think that's going to happen again.

s302
00:24:58.200 --> 00:25:04.960
Uh, in 10 years time we will get a profession that is bigger than today's, pays about as well, and is mostly shaped like product engineering.

s303
00:25:04.960 --> 00:25:08.560
That is the case that I'm making for you today.

s304
00:25:08.560 --> 00:25:11.080
But that is a whole lot of high-level predictions, and I hate those.

s305
00:25:11.080 --> 00:25:12.200
What does this mean for real?

s306
00:25:12.200 --> 00:25:14.840
What should you actually do with this today?

s307
00:25:14.840 --> 00:25:19.960
Uh, first, be prepared for a world in which humans are barely involved in the production of code.

s308
00:25:19.960 --> 00:25:30.200
So, write down your high blast radius code, and make that be the only thing that you are reviewing anymore, because the rest of it your automatic reviewers are going to get done.

s309
00:25:30.200 --> 00:25:34.080
Uh, then delegate that everything that's cheap to verify.

s310
00:25:34.080 --> 00:25:45.800
Spend a lot of time thinking about how would I build an automatic way of telling whether this thing, whatever that thing is, is good or not, and then feed that to your agent instead of trying to figure out the review process.

s311
00:25:45.800 --> 00:25:50.920
Uh, and finally, put your best people's judgment into specs and rubrics and evals.

s312
00:25:50.920 --> 00:25:54.120
You know, I the arise guy was going to say evals at some point.

s313
00:25:54.120 --> 00:25:58.760
Uh, get stuff that exists only in people's heads into code.

s314
00:25:58.760 --> 00:26:01.360
That is your job as a product product engineer.

s315
00:26:01.360 --> 00:26:08.760
It is turning the sense of the problem into a spec of the problem, so that the agents can build it for us.

s316
00:26:08.760 --> 00:26:10.400
Uh, and measure the right thing.

s317
00:26:10.400 --> 00:26:13.000
741% more code was never the point.

s318
00:26:13.000 --> 00:26:15.840
Measure how much stuff you are actually shipping.

s319
00:26:15.840 --> 00:26:18.680
Uh, measure how often your reviewers comments get acted on.

s320
00:26:18.680 --> 00:26:22.400
Measure the gap between the tests [clears throat] passed and a maintainer would merge this.

s321
00:26:22.400 --> 00:26:25.800
Uh, and remember that the last reviewer is production.

s322
00:26:25.800 --> 00:26:29.520
Watching what it does there with another model grading it, that is an eval.

s323
00:26:29.520 --> 00:26:34.560
I'm not going to pitch arise, that's just what we do, and that's why we're so popular right now.

s324
00:26:34.560 --> 00:26:36.760
Uh, and I'll leave it at that.

s325
00:26:36.760 --> 00:26:43.120
Um, 3 years ago, Swyx wrote about the rise of the AI engineer, which is engineers learning to build with models.

s326
00:26:43.120 --> 00:26:48.240
And the rise of the product engineer is about what's left when the models do the building.

s327
00:26:48.240 --> 00:26:54.280
Uh, finding out what people need, writing down what good means, and making it something people like, that is the job now.

s328
00:26:54.280 --> 00:26:58.520
It is going to be the job for everyone in this room, whether we like it or not.

s329
00:26:58.520 --> 00:27:01.000
We are all product engineers now.

s330
00:27:01.000 --> 00:27:03.108
Thank you so much for your time and attention.

s331
00:27:03.108 --> 00:27:05.108
[applause]