WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=geMAe-PLJF4

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/geMAe-PLJF4.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:13.200
Welcome.

s3
00:00:13.200 --> 00:00:15.440
Thank you for coming to my talk.

s4
00:00:15.440 --> 00:00:23.480
My name is Nick Nisi and I am a software engineer and engineering manager at WorkOS where I lead the developer and agent experiences team.

s5
00:00:23.480 --> 00:00:27.720
Uh and it's a lot of fun and I do lots of really fun stuff there.

s6
00:00:27.720 --> 00:00:30.200
Uh and at WorkOS we do have a booth here.

s7
00:00:30.200 --> 00:00:37.040
So if you want to chat with me about this talk or about uh WorkOS itself, come down and see us uh downstairs.

s8
00:00:37.280 --> 00:00:42.760
Uh but I want to talk to you about what I see as the big bottleneck in software development right now.

s9
00:00:42.760 --> 00:00:45.480
Uh code is being written faster than ever.

s10
00:00:45.480 --> 00:00:48.960
Uh and we are putting up more PRs than ever before.

s11
00:00:48.960 --> 00:00:51.640
But there's a bottleneck there.

s12
00:00:51.640 --> 00:00:55.020
We've got good CI, we've got linting, we've got agents running our

s13
00:00:55.020 --> 00:00:55.400
[snorts]

s14
00:00:55.400 --> 00:00:57.920
running like finding things on there.

s15
00:00:57.920 --> 00:00:59.760
But it's just stuck waiting for me.

s16
00:00:59.760 --> 00:01:04.720
So I I've become the the uh I've become the bottleneck.

s17
00:01:04.720 --> 00:01:09.960
Uh and that is uh something that hopefully we can solve and I think that we really can.

s18
00:01:09.960 --> 00:01:11.960
Um that's just the biggest piece of it.

s19
00:01:11.960 --> 00:01:15.880
If we're not writing the code anymore, why do we need a human to review the code?

s20
00:01:15.880 --> 00:01:19.960
Uh it doesn't quite make sense when the functionality is what matters.

s21
00:01:19.960 --> 00:01:24.320
So let's uh dive into that a little bit.

s22
00:01:24.400 --> 00:01:24.495
Um

s23
00:01:24.495 --> 00:01:24.760
[snorts]

s24
00:01:24.760 --> 00:01:27.520
like I said, I do think that this is a bottleneck that's going to be solved.

s25
00:01:27.520 --> 00:01:33.640
Uh code Claude co- Claude Codex um etc. has solved the coding problem.

s26
00:01:33.640 --> 00:01:36.360
Uh and we're still there, but we're managing it.

s27
00:01:36.360 --> 00:01:39.000
We've become managers, project managers.

s28
00:01:39.000 --> 00:01:41.040
Uh we've got agents and we've got CI.

s29
00:01:41.040 --> 00:01:43.800
We've always had CI kind of doing checks on our pull requests.

s30
00:01:43.800 --> 00:01:45.960
Uh and now we've got agents running on them as well.

s31
00:01:45.960 --> 00:01:47.880
Graphite and Code Rabbit and other ones.

s32
00:01:47.880 --> 00:01:54.720
If Graphite gives me a five out of five, I am excited and uh I'm pretty confident that I'm not going to break production.

s33
00:01:54.720 --> 00:01:58.880
So um it doesn't matter as much to look at the code.

s34
00:01:58.880 --> 00:02:04.732
What I really want to know is does it actually do what I expect it to do?

s35
00:02:04.732 --> 00:02:10.652
Uh and if code review does stick around, I think that it's just going to change like the reasoning why it's sticking around.

s36
00:02:10.652 --> 00:02:16.172
I don't think that it's going to be for me to give a thumbs up, yes I approve this even though that is a part of it.

s37
00:02:16.172 --> 00:02:27.252
It's more going to be to bring the context back to the human to say here's the change that is being proposed and why it matters to you and that's going to be the most important piece of it, but it's not going to be please go

s38
00:02:27.252 --> 00:02:35.412
make sure that Reptile didn't find uh didn't miss something that a human could find cuz only a human could find it cuz I don't believe that that's the case anymore.

s39
00:02:35.412 --> 00:02:39.972
Uh but before we dive into that more, let's take a step back and uh talk about bugs.

s40
00:02:39.972 --> 00:02:51.812
Uh so the term bug in software came from a literal moth that was found in a relay uh that was disrupting a program that was running uh way back in on my birthday in uh 1947

s41
00:02:51.812 --> 00:02:54.692
although despite the greys I wasn't around then.

s42
00:02:54.692 --> 00:02:58.172
Um and that's where the term came from.

s43
00:02:58.172 --> 00:03:04.732
Now skip ahead to 2026 and I'm going to use a bug to find bugs in my software.

s44
00:03:04.852 --> 00:03:06.492
So let's wake up our fly.

s45
00:03:06.492 --> 00:03:09.692
Let's bring him into the matrix and let's get him going.

s46
00:03:09.692 --> 00:03:12.452
Now to do that we'll connect him in.

s47
00:03:12.452 --> 00:03:16.132
We'll see his uh connectome and his neurons firing.

s48
00:03:16.132 --> 00:03:17.692
Now I'm not the first one to do this.

s49
00:03:17.692 --> 00:03:20.452
The this is actually been done several times before.

s50
00:03:20.452 --> 00:03:24.292
Uh if you've been on the internet, lots of people are torturing this poor fly.

s51
00:03:24.292 --> 00:03:28.492
They're making it play Doom uh which is a lot of fun, pretty cool.

s52
00:03:28.492 --> 00:03:33.132
Uh they're also doing things like making it play Beat Saber or do things like that.

s53
00:03:33.132 --> 00:03:41.092
Uh now if you think that this is torture to this poor fly's mind in the matrix, uh then you're really going to dislike what I did to it.

s54
00:03:41.092 --> 00:03:44.452
I made it review JavaScript.

s55
00:03:46.532 --> 00:03:50.562
All of JavaScript and its lovely lovely idioms.

s56
00:03:50.562 --> 00:03:50.652
[sighs]

s57
00:03:50.652 --> 00:03:52.652
Ugh.

s58
00:03:52.732 --> 00:03:55.652
I remember when I used to look at this code.

s59
00:03:55.652 --> 00:04:00.652
Uh so let's plug it back in and let's make it learn some code review.

s60
00:04:00.772 --> 00:04:03.334
So this diagram itself, let's talk about what it actually is.

s61
00:04:03.334 --> 00:04:14.654
This is called a connectome and it's the actual neurons of a male fruit fly brain that they meticulously scanned one by one under an electron microscope by slicing this brain

s62
00:04:14.654 --> 00:04:28.294
thousands of times and taking photographs of it and then wiring up all of the connections between those 176,000 neurons to create 6.3 million connections and over 125 million synapses.

s63
00:04:28.294 --> 00:04:35.894
That's what we're passing our code reviews to in this demo and that's what's going to be taking our jobs as code reviewers.

s64
00:04:35.894 --> 00:04:38.214
I didn't really change anything else about it.

s65
00:04:38.214 --> 00:04:42.494
I taught it to run, got it working, played around with it.

s66
00:04:42.494 --> 00:04:43.454
And then I didn't touch it.

s67
00:04:43.454 --> 00:04:50.414
I didn't change any of the neurons except for a small subset of them that I'm using to retain memories.

s68
00:04:50.414 --> 00:04:57.014
And I'll dig into what I did, but I basically have it remembering what's good and what's bad code.

s69
00:04:57.214 --> 00:05:00.414
So, you might be wondering how does it know what code is?

s70
00:05:00.414 --> 00:05:01.574
It's a fly.

s71
00:05:01.574 --> 00:05:02.814
Well, it doesn't.

s72
00:05:02.814 --> 00:05:07.334
It has sensilla which is like little hairs on it that act as its nose.

s73
00:05:07.334 --> 00:05:19.214
And I use AI, specifically I use Jev, a system one model, to act as its nose and ask 30 questions that are non-biased about the code that it's reviewing.

s74
00:05:19.214 --> 00:05:22.854
And then it passes that to the brain as a smell.

s75
00:05:22.854 --> 00:05:27.054
So, we're taking our code and turning it into literal code smells.

s76
00:05:27.054 --> 00:05:31.494
Then that goes against the memory that the fly has about what's good and what's bad code.

s77
00:05:31.494 --> 00:05:39.974
Before it gets to a pull request that it's reviewing, it's been trained on the code in the pull request and it know or sorry, in the code base and it knows

s78
00:05:39.974 --> 00:05:41.054
is this good code?

s79
00:05:41.054 --> 00:05:42.094
Is this bad code?

s80
00:05:42.094 --> 00:05:44.094
And I'll explain how that works.

s81
00:05:44.094 --> 00:05:56.334
And then if it's if it was trained as good code, it got like a sense of smelling sugar water and if it's bad code, it got a very bitter smell and so it's afraid of that and doesn't want to go near it.

s82
00:05:56.334 --> 00:05:59.934
And then a teacher reinforces that over it to give it this taste.

s83
00:05:59.934 --> 00:06:04.934
So, effectively the bug has a taste of is this good or bad code and goes from there.

s84
00:06:04.934 --> 00:06:22.215
And then it just like as the smells come in in different levels because, you know, every coder diff is different, it's going to get different hormones sent to it, either good hormones or bad hormones that, um, uh, like dopamine that, uh, wires like how the neurons are going to connect with each other and then what behavior it's

s85
00:06:22.215 --> 00:06:24.495
going to display.

s86
00:06:24.495 --> 00:06:31.535
And then it'll vote and it'll say, yes, the smell alone makes me think that this is, uh, a good, uh, PR.

s87
00:06:31.535 --> 00:06:35.495
Otherwise, no, I think that changes should be made because I'm terrified of this code.

s88
00:06:35.495 --> 00:06:38.055
It's terrified a lot.

s89
00:06:38.135 --> 00:06:46.759
And all of that just like ends up being like a hash table, uh, where the keys collide on purpose, uh, and we use testing to, um, to write those values.

s90
00:06:46.759 --> 00:06:47.295
[snorts]

s91
00:06:47.295 --> 00:06:50.935
So, every result that it gets from that training is a taste.

s92
00:06:50.935 --> 00:06:55.055
And so, the memories give it a taste of what's good and what's bad code.

s93
00:06:55.055 --> 00:07:03.735
Uh, if the test produces a bug, it gets a bitter punishment dopamine, uh, and that weakens the lit cell wires between the approach.

s94
00:07:03.735 --> 00:07:15.895
And if it's a good thing, it gets sugar, which, uh, fires reward dopamine, uh, and that weakens the wires to avoid, and then the tongue will come out as if it wants to slurp that up.

s95
00:07:16.935 --> 00:07:19.855
Uh, if the tongue comes out, smells good to me.

s96
00:07:19.855 --> 00:07:23.815
That's, uh, that's the indicator that we get from the fly.

s97
00:07:24.135 --> 00:07:31.375
And otherwise, if, uh, it doesn't come out and it retracts, uh, then that is something that we should avoid.

s98
00:07:32.135 --> 00:07:39.575
So, uh, I tested this and like the thing that I was really worried about was making the fly brain kind of redundant in this, right?

s99
00:07:39.575 --> 00:07:46.935
I don't want it to just be like it's Jev that's saying it's good or bad and then the fly is just there to interpret what that means.

s100
00:07:46.935 --> 00:07:48.055
That's not what's actually happening.

s101
00:07:48.055 --> 00:07:58.095
If I remove those memories, then it totally freaks out and doesn't do the right thing because it has no recollection of like what's good and what's bad code and what caused a bitter, uh, bitter taste

s102
00:07:58.095 --> 00:08:01.815
uh, versus a sweet taste, and so it won't behave correctly.

s103
00:08:01.815 --> 00:08:02.895
It'll scramble everything.

s104
00:08:02.895 --> 00:08:06.295
The tongue will come out when it shouldn't, uh, and it overall just doesn't work.

s105
00:08:06.295 --> 00:08:15.185
So, the, um, the fly brain is actually doing the processing on this and it's thinking about what's good and what's bad code.

s106
00:08:15.185 --> 00:08:15.497
[snorts]

s107
00:08:15.497 --> 00:08:18.537
So, the name of this talk was are you smarter than a fruit fly?

s108
00:08:18.537 --> 00:08:21.297
So, we're going to play a game as an audience.

s109
00:08:21.297 --> 00:08:22.577
Uh step right up.

s110
00:08:22.577 --> 00:08:27.497
Let's go ahead and uh hit this button and draw.

s111
00:08:27.497 --> 00:08:33.497
We're just going to pick three random files from the actual code base uh for this fruit fly.

s112
00:08:33.497 --> 00:08:38.097
And show of hands, would you approve this change in the code?

s113
00:08:38.097 --> 00:08:41.377
Raise your hand if you would approve of this change.

s114
00:08:44.017 --> 00:08:44.817
No hands raised.

s115
00:08:44.817 --> 00:08:47.057
That's either no participation or we're not going to.

s116
00:08:47.057 --> 00:08:52.177
So, I'll say that we are uh going to request changes in this.

s117
00:08:52.177 --> 00:08:54.897
Let's see what the fruit fly did.

s118
00:08:55.457 --> 00:08:56.697
It smells good to the fruit fly.

s119
00:08:56.697 --> 00:08:59.417
So, let's see what the actual response was.

s120
00:08:59.417 --> 00:09:00.137
It's a bug.

s121
00:09:00.137 --> 00:09:02.097
The fruit fly was wrong.

s122
00:09:02.097 --> 00:09:04.217
Not looking good so far.

s123
00:09:04.217 --> 00:09:05.257
All right, next one.

s124
00:09:05.257 --> 00:09:13.377
Uh in this PKCE code, would you raise your hand if you would approve this change or if you would reject this change?

s125
00:09:13.377 --> 00:09:13.657
Oh, sorry.

s126
00:09:13.657 --> 00:09:15.857
If you would approve this change, raise your hand.

s127
00:09:15.857 --> 00:09:17.857
Okay?

s128
00:09:17.897 --> 00:09:18.297
See some hands.

s129
00:09:18.297 --> 00:09:20.657
I'll go with approve.

s130
00:09:25.657 --> 00:09:28.617
The fly will also approve.

s131
00:09:28.617 --> 00:09:29.617
Cool.

s132
00:09:29.617 --> 00:09:30.777
It was a harmless change.

s133
00:09:30.777 --> 00:09:32.377
All good.

s134
00:09:32.377 --> 00:09:33.737
Last one.

s135
00:09:33.737 --> 00:09:38.897
Uh on this change, raise your hand if you would approve this change.

s136
00:09:40.897 --> 00:09:42.301
One.

s137
00:09:42.301 --> 00:09:43.097
[laughter]

s138
00:09:43.097 --> 00:09:43.617
All right.

s139
00:09:43.617 --> 00:09:47.577
Yeah, I think this is probably we're going to reject this change.

s140
00:09:47.577 --> 00:09:49.497
Does the fly reject it?

s141
00:09:49.497 --> 00:09:50.177
Ooh, yeah.

s142
00:09:50.177 --> 00:09:51.577
Something stinks.

s143
00:09:51.577 --> 00:09:52.177
Bug.

s144
00:09:52.177 --> 00:09:53.017
All right.

s145
00:09:53.017 --> 00:09:54.737
So, the room went three for three.

s146
00:09:54.737 --> 00:09:56.937
The fly went two for three.

s147
00:09:56.937 --> 00:09:57.977
That's pretty good.

s148
00:09:57.977 --> 00:10:01.057
Uh obviously, you'll you all are smarter than a fruit fly.

s149
00:10:01.057 --> 00:10:02.737
Congratulations.

s150
00:10:02.737 --> 00:10:04.457
Uh but the fruit fly was pretty good.

s151
00:10:04.457 --> 00:10:07.697
It got uh it got it right about 88% of the time when I was testing it.

s152
00:10:07.697 --> 00:10:14.338
So, um overall, is it worth your time when a fruit fly can do it 80% of the time?

s153
00:10:14.338 --> 00:10:17.058
80% of the time it's correct all the time.

s154
00:10:17.058 --> 00:10:20.058
Um that's that's a good question.

s155
00:10:20.058 --> 00:10:22.178
So, let's try it on a on something real.

s156
00:10:22.178 --> 00:10:26.098
Let me paste in a uh a pull request here.

s157
00:10:26.098 --> 00:10:29.418
Uh and this one is just to one of my pie extensions.

s158
00:10:29.418 --> 00:10:35.938
Um and it's something that I approved, but let's see if the fruit fly agrees with that as well.

s159
00:10:35.938 --> 00:10:38.018
So, I'm sending it to Jeff.

s160
00:10:38.018 --> 00:10:40.738
And it's going to do a test.

s161
00:10:40.738 --> 00:10:47.218
Um read that and fetch it and hopefully things cooperate with me in a live demo.

s162
00:10:47.218 --> 00:10:50.538
But, if they don't, that's okay.

s163
00:10:50.538 --> 00:10:52.778
I have a backup.

s164
00:10:54.778 --> 00:10:56.298
We'll go with the backup.

s165
00:10:56.298 --> 00:10:57.218
And it can't tell.

s166
00:10:57.218 --> 00:11:02.458
So, the backup didn't uh didn't make a distinction either on this change.

s167
00:11:02.458 --> 00:11:06.258
Um that's okay.

s168
00:11:06.258 --> 00:11:12.538
Uh you can run this yourself at fruitfly.review uh online and play with it.

s169
00:11:12.538 --> 00:11:17.138
But, you probably don't want to actually use a fruit fly to test your code.

s170
00:11:17.138 --> 00:11:24.482
Um now in this code, um I did give it a lot of hard a lot harder of tests than uh what I showed here.

s171
00:11:24.482 --> 00:11:24.658
[snorts]

s172
00:11:24.658 --> 00:11:32.538
Uh 96% of the time it would correct correctly uh catch the first example here uh where it would request changes.

s173
00:11:32.538 --> 00:11:38.858
At 98% of the time uh a very plausible mistake, it would uh also request changes on that.

s174
00:11:38.858 --> 00:11:42.658
So, it is working most of the time where it can find issues.

s175
00:11:42.658 --> 00:11:47.898
Of course, it's a fruit fly, it can't communicate and it doesn't know what JavaScript actually is, so it's not going to tell you how to change it.

s176
00:11:47.898 --> 00:11:50.218
It's just going to say, "This smells funny to me.

s177
00:11:50.218 --> 00:11:53.178
I don't think I would accept that."

s178
00:11:55.258 --> 00:11:59.858
So, if the fruit fly can do the code review moderately well, then why are humans still doing it?

s179
00:11:59.858 --> 00:12:02.258
That's the question.

s180
00:12:02.658 --> 00:12:16.019
Um what we've learned from this uh is basically it can figure out what's changed, uh figure out conditionals, removed logic, um Um, you know, simple issues like that and it can steer you away from that and it can kind of keep us on track.

s181
00:12:16.019 --> 00:12:19.419
Of course, Reptile, Code Rabbit, all of those can do it much better.

s182
00:12:19.419 --> 00:12:21.259
Um, but nothing knows why.

s183
00:12:21.259 --> 00:12:33.739
And I think that that's the important thing that humans will always have a uh hand on because we need to understand why code is actually being changed or why functionality is changing more specifically.

s184
00:12:33.739 --> 00:12:34.579
So, how do we do that?

s185
00:12:34.579 --> 00:12:45.219
If we were to take humans out of the review process, um, we can start by just escalating to the ones uh escalating the ones that need our reviews and letting everything else pass.

s186
00:12:45.219 --> 00:12:48.219
And I've got some ideas about how we might do that.

s187
00:12:48.219 --> 00:12:51.179
Uh, the first one is a team review skill.

s188
00:12:51.179 --> 00:13:01.139
So, uh I lead the DX team at WorkOS and if a code review is tagged with us, like we use code owners on GitHub, uh if it's tagged with us,

s189
00:13:01.139 --> 00:13:12.499
there should be a reviewer script or skill that we maintain as a team that is basically like the bare minimum of what a DX engineer at WorkOS would be reviewing for.

s190
00:13:12.499 --> 00:13:17.819
And it's something that we meticulously keep up uh and keep track of and make sure it's working correctly.

s191
00:13:17.819 --> 00:13:22.259
Um, if it passes that, then it can move on to the next thing.

s192
00:13:22.259 --> 00:13:23.259
Some kind of classifier.

s193
00:13:23.259 --> 00:13:31.579
This could be Jev or Clef or something else or some lightweight model, excuse me, that is doing uh a check to see, you know, is this too complex?

s194
00:13:31.579 --> 00:13:34.619
Is it actually doing what it says it's going to do?

s195
00:13:34.619 --> 00:13:36.179
Will this cause significant risk?

s196
00:13:36.179 --> 00:13:39.219
It can do like different uh weights on that.

s197
00:13:39.219 --> 00:13:49.939
But assuming it passes all of that, then it can give it to an automated reviewer like Reptile, Code Rabbit, uh Codo, all of the the fantastic companies that are here today.

s198
00:13:49.939 --> 00:13:51.979
And then, from there, it decides.

s199
00:13:51.979 --> 00:13:54.939
If any of those failed, escalate that to a human.

s200
00:13:54.939 --> 00:14:00.939
If none of those failed, it's probably a safe PR to merge on its own and we don't have to bother another human.

s201
00:14:00.939 --> 00:14:03.259
We can get software shipped in faster.

s202
00:14:03.259 --> 00:14:15.259
If something does break, we can review what happened, uh audit the this whole thing, see if maybe we can fix the team review skill to catch it or fix one of the automated code reviewers to catch it uh the next time.

s203
00:14:15.259 --> 00:14:21.741
But more importantly, we can use AI to fix the code uh the uh and go forward from there.

s204
00:14:21.741 --> 00:14:29.661
And then, deliver back a summarized response to the human as to why this change happened uh and you know, what it was.

s205
00:14:29.661 --> 00:14:31.821
Is it business logic that's changing?

s206
00:14:31.821 --> 00:14:35.261
Uh were there all alternatives that were tested and discarded?

s207
00:14:35.261 --> 00:14:45.981
Uh etc. But, the main thing is I think uh as a next step is teams managing that skill where they can keep track of what's changed, uh how they would review

s208
00:14:45.981 --> 00:14:55.221
these skill or review code as a as a team, and then use that to decide if something should be escalated to someone on that team to check.

s209
00:14:56.341 --> 00:15:04.781
And then, of course, on the last thing when it's summarizing, uh if we don't care about the code so much anymore and we only care about does this actually do what we

s210
00:15:04.781 --> 00:15:09.101
uh expect it to do, we're going to want some kind of proof of that or some kind of verification.

s211
00:15:09.101 --> 00:15:17.741
And so, you can prove that the agents actually like did the work to verify this by having it create screenshots, videos, uh put test results or command outputs on there.

s212
00:15:17.741 --> 00:15:22.221
Write and provide smoke tests that I can run and it and see the outputs of that as well.

s213
00:15:22.221 --> 00:15:32.021
Or just give me easy ways to replicate this so that I can do some manual testing if I need to, uh and that will slowly diminish as I trust these other methods to go.

s214
00:15:32.701 --> 00:15:35.221
So, that review might look something like this.

s215
00:15:35.221 --> 00:15:46.021
For the DX team, uh it might say like, "This is anything that a human uh would look at, and these are things that would a human would stop and say, 'No, we need to request changes on from there.'"

s216
00:15:46.021 --> 00:15:49.061
And so, it can just kind of do that review based on that.

s217
00:15:49.061 --> 00:15:54.861
This would be very uh dependent on what your team expects, what your team's looking for, what your team covers.

s218
00:15:54.861 --> 00:15:57.941
You know, a DX team is going to be different from a platform team.

s219
00:15:57.941 --> 00:16:02.141
Um I'm not going to know any what to do with the animal files, for example.

s220
00:16:02.141 --> 00:16:06.461
Uh and so, we're going to have different different things that we're going to look at and focus on.

s221
00:16:06.461 --> 00:16:17.981
And depending on what we're changing, uh those teams can manage different aspects of it and come to a conclusion as to whether we actually need to escalate this to a human in the loop.

s222
00:16:18.301 --> 00:16:21.182
And with that, uh I invite you to try this out.

s223
00:16:21.182 --> 00:16:23.982
You can go run your own pull request there at fruitfly.review.

s224
00:16:23.982 --> 00:16:27.382
You can also see the slides there and uh any uh references.

s225
00:16:27.382 --> 00:16:31.942
So, check it out and uh come see me at the Work OS booth.