I Made a Fruit Fly Review My Code - Nick Nisi, WorkOS

AI Engineer · 16 min · 225 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Welcome.
00:13Thank you for coming to my talk.
00:15My name is Nick Nisi and I am a software engineer and engineering manager at WorkOS where I lead the developer and agent experiences team.
00:23Uh and it's a lot of fun and I do lots of really fun stuff there.
00:27Uh and at WorkOS we do have a booth here.
00:30So if you want to chat with me about this talk or about uh WorkOS itself, come down and see us uh downstairs.
00:37Uh but I want to talk to you about what I see as the big bottleneck in software development right now.
00:42Uh code is being written faster than ever.
00:45Uh and we are putting up more PRs than ever before.
00:48But there's a bottleneck there.
00:51We've got good CI, we've got linting, we've got agents running our
00:55[snorts]
00:55running like finding things on there.
00:57But it's just stuck waiting for me.
00:59So I I've become the the uh I've become the bottleneck.
01:04Uh and that is uh something that hopefully we can solve and I think that we really can.
01:09Um that's just the biggest piece of it.
01:11If we're not writing the code anymore, why do we need a human to review the code?
01:15Uh it doesn't quite make sense when the functionality is what matters.
01:19So let's uh dive into that a little bit.
01:24Um
01:24[snorts]
01:24like I said, I do think that this is a bottleneck that's going to be solved.
01:27Uh code Claude co- Claude Codex um etc. has solved the coding problem.
01:33Uh and we're still there, but we're managing it.
01:36We've become managers, project managers.
01:39Uh we've got agents and we've got CI.
01:41We've always had CI kind of doing checks on our pull requests.
01:43Uh and now we've got agents running on them as well.
01:45Graphite and Code Rabbit and other ones.
01:47If Graphite gives me a five out of five, I am excited and uh I'm pretty confident that I'm not going to break production.
01:54So um it doesn't matter as much to look at the code.
01:58What I really want to know is does it actually do what I expect it to do?
02:04Uh and if code review does stick around, I think that it's just going to change like the reasoning why it's sticking around.
02:10I don't think that it's going to be for me to give a thumbs up, yes I approve this even though that is a part of it.
02:16It's more going to be to bring the context back to the human to say here's the change that is being proposed and why it matters to you and that's going to be the most important piece of it, but it's not going to be please go
02:27make sure that Reptile didn't find uh didn't miss something that a human could find cuz only a human could find it cuz I don't believe that that's the case anymore.
02:35Uh but before we dive into that more, let's take a step back and uh talk about bugs.
02:39Uh so the term bug in software came from a literal moth that was found in a relay uh that was disrupting a program that was running uh way back in on my birthday in uh 1947
02:51although despite the greys I wasn't around then.
02:54Um and that's where the term came from.
02:58Now skip ahead to 2026 and I'm going to use a bug to find bugs in my software.
03:04So let's wake up our fly.
03:06Let's bring him into the matrix and let's get him going.
03:09Now to do that we'll connect him in.
03:12We'll see his uh connectome and his neurons firing.
03:16Now I'm not the first one to do this.
03:17The this is actually been done several times before.
03:20Uh if you've been on the internet, lots of people are torturing this poor fly.
03:24They're making it play Doom uh which is a lot of fun, pretty cool.
03:28Uh they're also doing things like making it play Beat Saber or do things like that.
03:33Uh now if you think that this is torture to this poor fly's mind in the matrix, uh then you're really going to dislike what I did to it.
03:41I made it review JavaScript.
03:46All of JavaScript and its lovely lovely idioms.
03:50[sighs]
03:50Ugh.
03:52I remember when I used to look at this code.
03:55Uh so let's plug it back in and let's make it learn some code review.
04:00So this diagram itself, let's talk about what it actually is.
04:03This is called a connectome and it's the actual neurons of a male fruit fly brain that they meticulously scanned one by one under an electron microscope by slicing this brain
04:14thousands of times and taking photographs of it and then wiring up all of the connections between those 176,000 neurons to create 6.3 million connections and over 125 million synapses.
04:28That's what we're passing our code reviews to in this demo and that's what's going to be taking our jobs as code reviewers.
04:35I didn't really change anything else about it.
04:38I taught it to run, got it working, played around with it.
04:42And then I didn't touch it.
04:43I didn't change any of the neurons except for a small subset of them that I'm using to retain memories.
04:50And I'll dig into what I did, but I basically have it remembering what's good and what's bad code.
04:57So, you might be wondering how does it know what code is?
05:00It's a fly.
05:01Well, it doesn't.
05:02It has sensilla which is like little hairs on it that act as its nose.
05:07And I use AI, specifically I use Jev, a system one model, to act as its nose and ask 30 questions that are non-biased about the code that it's reviewing.
05:19And then it passes that to the brain as a smell.
05:22So, we're taking our code and turning it into literal code smells.
05:27Then that goes against the memory that the fly has about what's good and what's bad code.
05:31Before it gets to a pull request that it's reviewing, it's been trained on the code in the pull request and it know or sorry, in the code base and it knows
05:39is this good code?
05:41Is this bad code?
05:42And I'll explain how that works.
05:44And then if it's if it was trained as good code, it got like a sense of smelling sugar water and if it's bad code, it got a very bitter smell and so it's afraid of that and doesn't want to go near it.
05:56And then a teacher reinforces that over it to give it this taste.
05:59So, effectively the bug has a taste of is this good or bad code and goes from there.
06:04And then it just like as the smells come in in different levels because, you know, every coder diff is different, it's going to get different hormones sent to it, either good hormones or bad hormones that, um, uh, like dopamine that, uh, wires like how the neurons are going to connect with each other and then what behavior it's
06:22going to display.
06:24And then it'll vote and it'll say, yes, the smell alone makes me think that this is, uh, a good, uh, PR.
06:31Otherwise, no, I think that changes should be made because I'm terrified of this code.
06:35It's terrified a lot.
06:38And all of that just like ends up being like a hash table, uh, where the keys collide on purpose, uh, and we use testing to, um, to write those values.
06:46[snorts]
06:47So, every result that it gets from that training is a taste.
06:50And so, the memories give it a taste of what's good and what's bad code.
06:55Uh, if the test produces a bug, it gets a bitter punishment dopamine, uh, and that weakens the lit cell wires between the approach.
07:03And if it's a good thing, it gets sugar, which, uh, fires reward dopamine, uh, and that weakens the wires to avoid, and then the tongue will come out as if it wants to slurp that up.
07:16Uh, if the tongue comes out, smells good to me.
07:19That's, uh, that's the indicator that we get from the fly.
07:24And otherwise, if, uh, it doesn't come out and it retracts, uh, then that is something that we should avoid.
07:32So, uh, I tested this and like the thing that I was really worried about was making the fly brain kind of redundant in this, right?
07:39I don't want it to just be like it's Jev that's saying it's good or bad and then the fly is just there to interpret what that means.
07:46That's not what's actually happening.
07:48If I remove those memories, then it totally freaks out and doesn't do the right thing because it has no recollection of like what's good and what's bad code and what caused a bitter, uh, bitter taste
07:58uh, versus a sweet taste, and so it won't behave correctly.
08:01It'll scramble everything.
08:02The tongue will come out when it shouldn't, uh, and it overall just doesn't work.
08:06So, the, um, the fly brain is actually doing the processing on this and it's thinking about what's good and what's bad code.
08:15[snorts]
08:15So, the name of this talk was are you smarter than a fruit fly?
08:18So, we're going to play a game as an audience.
08:21Uh step right up.
08:22Let's go ahead and uh hit this button and draw.
08:27We're just going to pick three random files from the actual code base uh for this fruit fly.
08:33And show of hands, would you approve this change in the code?
08:38Raise your hand if you would approve of this change.
08:44No hands raised.
08:44That's either no participation or we're not going to.
08:47So, I'll say that we are uh going to request changes in this.
08:52Let's see what the fruit fly did.
08:55It smells good to the fruit fly.
08:56So, let's see what the actual response was.
08:59It's a bug.
09:00The fruit fly was wrong.
09:02Not looking good so far.
09:04All right, next one.
09:05Uh in this PKCE code, would you raise your hand if you would approve this change or if you would reject this change?
09:13Oh, sorry.
09:13If you would approve this change, raise your hand.
09:15Okay?
09:17See some hands.
09:18I'll go with approve.
09:25The fly will also approve.
09:28Cool.
09:29It was a harmless change.
09:30All good.
09:32Last one.
09:33Uh on this change, raise your hand if you would approve this change.
09:40One.
09:42[laughter]
09:43All right.
09:43Yeah, I think this is probably we're going to reject this change.
09:47Does the fly reject it?
09:49Ooh, yeah.
09:50Something stinks.
09:51Bug.
09:52All right.
09:53So, the room went three for three.
09:54The fly went two for three.
09:56That's pretty good.
09:57Uh obviously, you'll you all are smarter than a fruit fly.
10:01Congratulations.
10:02Uh but the fruit fly was pretty good.
10:04It got uh it got it right about 88% of the time when I was testing it.
10:07So, um overall, is it worth your time when a fruit fly can do it 80% of the time?
10:1480% of the time it's correct all the time.
10:17Um that's that's a good question.
10:20So, let's try it on a on something real.
10:22Let me paste in a uh a pull request here.
10:26Uh and this one is just to one of my pie extensions.
10:29Um and it's something that I approved, but let's see if the fruit fly agrees with that as well.
10:35So, I'm sending it to Jeff.
10:38And it's going to do a test.
10:40Um read that and fetch it and hopefully things cooperate with me in a live demo.
10:47But, if they don't, that's okay.
10:50I have a backup.
10:54We'll go with the backup.
10:56And it can't tell.
10:57So, the backup didn't uh didn't make a distinction either on this change.
11:02Um that's okay.
11:06Uh you can run this yourself at fruitfly.review uh online and play with it.
11:12But, you probably don't want to actually use a fruit fly to test your code.
11:17Um now in this code, um I did give it a lot of hard a lot harder of tests than uh what I showed here.
11:24[snorts]
11:24Uh 96% of the time it would correct correctly uh catch the first example here uh where it would request changes.
11:32At 98% of the time uh a very plausible mistake, it would uh also request changes on that.
11:38So, it is working most of the time where it can find issues.
11:42Of course, it's a fruit fly, it can't communicate and it doesn't know what JavaScript actually is, so it's not going to tell you how to change it.
11:47It's just going to say, "This smells funny to me.
11:50I don't think I would accept that."
11:55So, if the fruit fly can do the code review moderately well, then why are humans still doing it?
11:59That's the question.
12:02Um what we've learned from this uh is basically it can figure out what's changed, uh figure out conditionals, removed logic, um Um, you know, simple issues like that and it can steer you away from that and it can kind of keep us on track.
12:16Of course, Reptile, Code Rabbit, all of those can do it much better.
12:19Um, but nothing knows why.
12:21And I think that that's the important thing that humans will always have a uh hand on because we need to understand why code is actually being changed or why functionality is changing more specifically.
12:33So, how do we do that?
12:34If we were to take humans out of the review process, um, we can start by just escalating to the ones uh escalating the ones that need our reviews and letting everything else pass.
12:45And I've got some ideas about how we might do that.
12:48Uh, the first one is a team review skill.
12:51So, uh I lead the DX team at WorkOS and if a code review is tagged with us, like we use code owners on GitHub, uh if it's tagged with us,
13:01there should be a reviewer script or skill that we maintain as a team that is basically like the bare minimum of what a DX engineer at WorkOS would be reviewing for.
13:12And it's something that we meticulously keep up uh and keep track of and make sure it's working correctly.
13:17Um, if it passes that, then it can move on to the next thing.
13:22Some kind of classifier.
13:23This could be Jev or Clef or something else or some lightweight model, excuse me, that is doing uh a check to see, you know, is this too complex?
13:31Is it actually doing what it says it's going to do?
13:34Will this cause significant risk?
13:36It can do like different uh weights on that.
13:39But assuming it passes all of that, then it can give it to an automated reviewer like Reptile, Code Rabbit, uh Codo, all of the the fantastic companies that are here today.
13:49And then, from there, it decides.
13:51If any of those failed, escalate that to a human.
13:54If none of those failed, it's probably a safe PR to merge on its own and we don't have to bother another human.
14:00We can get software shipped in faster.
14:03If something does break, we can review what happened, uh audit the this whole thing, see if maybe we can fix the team review skill to catch it or fix one of the automated code reviewers to catch it uh the next time.
14:15But more importantly, we can use AI to fix the code uh the uh and go forward from there.
14:21And then, deliver back a summarized response to the human as to why this change happened uh and you know, what it was.
14:29Is it business logic that's changing?
14:31Uh were there all alternatives that were tested and discarded?
14:35Uh etc. But, the main thing is I think uh as a next step is teams managing that skill where they can keep track of what's changed, uh how they would review
14:45these skill or review code as a as a team, and then use that to decide if something should be escalated to someone on that team to check.
14:56And then, of course, on the last thing when it's summarizing, uh if we don't care about the code so much anymore and we only care about does this actually do what we
15:04uh expect it to do, we're going to want some kind of proof of that or some kind of verification.
15:09And so, you can prove that the agents actually like did the work to verify this by having it create screenshots, videos, uh put test results or command outputs on there.
15:17Write and provide smoke tests that I can run and it and see the outputs of that as well.
15:22Or just give me easy ways to replicate this so that I can do some manual testing if I need to, uh and that will slowly diminish as I trust these other methods to go.
15:32So, that review might look something like this.
15:35For the DX team, uh it might say like, "This is anything that a human uh would look at, and these are things that would a human would stop and say, 'No, we need to request changes on from there.'"
15:46And so, it can just kind of do that review based on that.
15:49This would be very uh dependent on what your team expects, what your team's looking for, what your team covers.
15:54You know, a DX team is going to be different from a platform team.
15:57Um I'm not going to know any what to do with the animal files, for example.
16:02Uh and so, we're going to have different different things that we're going to look at and focus on.
16:06And depending on what we're changing, uh those teams can manage different aspects of it and come to a conclusion as to whether we actually need to escalate this to a human in the loop.
16:18And with that, uh I invite you to try this out.
16:21You can go run your own pull request there at fruitfly.review.
16:23You can also see the slides there and uh any uh references.
16:27So, check it out and uh come see me at the Work OS booth.