WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=EL123UNokkI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/EL123UNokkI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.600 --> 00:00:14.600 All right, hello everyone. s3 00:00:14.600 --> 00:00:20.960 My name is Will and uh I'm here to talk to you about automated code review. s4 00:00:20.960 --> 00:00:30.000 Uh my teammate Amir and I work at Uber and we're going to be walking through U Review, a system that Uber has built uh to help increase the velocity s5 00:00:30.000 --> 00:00:33.000 of our software engineering teams. s6 00:00:33.160 --> 00:00:46.400 Um for a little bit of context about what software engineering org at Uber looks like, we have thousands of software engineers who work across hundreds of teams uh located across 12 different sites s7 00:00:46.400 --> 00:00:51.400 and uh they work in primarily one of six language-specific monorepos. s8 00:00:51.400 --> 00:00:58.560 As many of you have probably noticed over the past 24 months, the volume of PRs, the size of PRs has been growing. s9 00:00:58.560 --> 00:01:05.239 One of the ways that that's been exposed to us has been through uh the metric that we track of the first time to review. s10 00:01:05.239 --> 00:01:11.080 Back in 2024, we were seeing that engineers would get their first review within 3 hours. s11 00:01:11.080 --> 00:01:16.560 Now in 2026, that has grown to 9 hours uh in addition to all of the volume changes. s12 00:01:16.560 --> 00:01:21.960 So, in short, code review is now the bottleneck that we are running into. s13 00:01:21.960 --> 00:01:34.880 Um specifically around automated code review, uh there are there are various options available in the industry, uh but Uber spent the time to invest in building an in-house solution due to some of the constraints that we have. s14 00:01:34.880 --> 00:01:41.600 One of those is uh we currently use Fabricator and have for a long time and are in the process of migrating to GitHub. s15 00:01:41.600 --> 00:01:45.560 Uh most of the solutions do not provide support for Fabricator. s16 00:01:45.560 --> 00:01:52.320 Um in addition, if you were at the previous talk, you saw Uday and Adam talking about the agentic SDLC. s17 00:01:52.320 --> 00:02:00.280 A big part of what we want to do is bring a consistent code review experience to the inner loop so that our agents are getting the same code review, s18 00:02:00.280 --> 00:02:04.479 the same rules, everything applied as our humans do. s19 00:02:04.479 --> 00:02:15.080 With hundreds of teams across the company, we can't have centralized management of our code reviews, our customizations, and our rules, and even the knowledge that goes into those code reviews. s20 00:02:15.080 --> 00:02:16.240 We need to distribute that. s21 00:02:16.240 --> 00:02:25.600 So, we have a need for plugging into existing team ownership system rather than trying to replicate that externally. s22 00:02:25.600 --> 00:02:38.440 Uh finally, with the volume of code reviews that we perform, we need the ability to take factors like the risk profile and the complexity of a code change and factor that in when deciding s23 00:02:38.440 --> 00:02:40.440 how we're going to run a code review. s24 00:02:40.440 --> 00:02:43.120 Not all code gets the exact same review. s25 00:02:43.120 --> 00:02:44.680 And then finally, consistency. s26 00:02:44.680 --> 00:02:48.760 We need to make sure that we have security and compliance reviews run across everything. s27 00:02:48.760 --> 00:02:54.560 We can't rely on teams hoping to run the skill the code review skill that happens. s28 00:02:54.560 --> 00:02:57.000 We need reliability there. s29 00:02:57.000 --> 00:03:02.600 With all that said, I wanted to give you an overview of the architecture of what you review looks like. s30 00:03:02.600 --> 00:03:07.920 We'll talk about a couple of the big pieces, and then we're going to dive into a few focus areas. s31 00:03:07.920 --> 00:03:14.800 At the top, you'll notice that we have our code review surface areas, GitHub, Fabricator, and the agent loop. s32 00:03:14.800 --> 00:03:18.760 These all feed into you review service. s33 00:03:18.760 --> 00:03:21.519 These This takes in requests for reviews. s34 00:03:21.519 --> 00:03:25.000 It brings in feedback from users, and it routes it. s35 00:03:25.000 --> 00:03:27.239 We have a number of different generators. s36 00:03:27.239 --> 00:03:33.959 Now, these generators are tuned for different performance and cost avenues. s37 00:03:33.959 --> 00:03:43.239 There are We also have the ability to plug into third-party code review systems so that we can compare ourselves to what's available more broadly. s38 00:03:43.239 --> 00:03:51.720 Finally, with all these different generators, we might be might be duplicating comments, and we can actually create quite a high volume of comments. s39 00:03:51.720 --> 00:03:55.400 If you've ever used AI to to run a code review, you've probably seen that. s40 00:03:55.400 --> 00:04:08.040 So, we run through a number of steps in the post-processing where we both rate, categorize, filter, and deduplicate comments so that our engineers get only the highest confidence comments s41 00:04:08.040 --> 00:04:10.880 that are actionable for them to work on. s42 00:04:10.880 --> 00:04:15.520 You'll also notice along the bottom we talk a little bit about feedback in our evaluation. s43 00:04:15.520 --> 00:04:22.519 But, with this context of the overall system, I'm now going to hand it off to Ameya to dive into our first focus area. s44 00:04:24.200 --> 00:04:25.200 Hello. s45 00:04:25.200 --> 00:04:26.280 Hello, everyone. s46 00:04:26.280 --> 00:04:33.200 So, I will be talking about how we evolve U review with observability and evaluation. s47 00:04:33.520 --> 00:04:35.720 So, U review had a very humble beginning. s48 00:04:35.720 --> 00:04:42.320 Basically, it was a single prompt that you should do logic checks per file, a simple agent which used to do thorough review. s49 00:04:42.320 --> 00:04:48.000 And we had a dispatcher to decide whether to go which generator to choose. s50 00:04:48.000 --> 00:04:51.840 Even what we used to collect as observability was very surface-level. s51 00:04:51.840 --> 00:04:53.240 We used to collect cost. s52 00:04:53.240 --> 00:04:57.680 We used to run an NPS survey, have Google Forms being filled, Slack support. s53 00:04:57.680 --> 00:05:03.280 And with all of this, we saw that our quality to cost ratio was like all over the place. s54 00:05:03.280 --> 00:05:11.200 Like, our goal is to be in the second quadrant, that is the top left quadrant, but you can see we were all over the place. s55 00:05:11.200 --> 00:05:15.560 Then what we did is that we started collecting more data. s56 00:05:15.560 --> 00:05:28.080 So, we started collecting the sentiments of the replies that were made to the U review that the U review uh call you know, the U review agent got from the developers. s57 00:05:28.080 --> 00:05:30.600 So, we categorized them into positive, negative. s58 00:05:30.600 --> 00:05:39.120 We classified them into various categories, and we found a bunch a lot of classes of bugs and issues that we could actually solve. s59 00:05:39.120 --> 00:05:47.560 And with that, we improved the system, and we were able to move a large number of PRs to a high quality to cost ratio. s60 00:05:47.560 --> 00:05:51.240 Um but, we still felt that this was not enough. s61 00:05:51.240 --> 00:05:54.480 We need to know more of how the review is done. s62 00:05:54.480 --> 00:05:57.040 So we started tracking things like address rate. s63 00:05:57.040 --> 00:06:03.800 So basically when a U review comment is made, does the developer go and actually address the comment? s64 00:06:03.800 --> 00:06:05.640 We started tracking that. s65 00:06:05.640 --> 00:06:16.840 And then we also started doing more like a runtime profile, which is like the agent trajectory, which told us why the agent is doing what it what it did. s66 00:06:16.840 --> 00:06:19.760 We get to know what tools calls it made. s67 00:06:19.760 --> 00:06:22.080 We get to know what thinking process it had. s68 00:06:22.080 --> 00:06:35.520 And then with that insight, we were able to actually tune our runtime, tune our performance such that the agent could very quickly give us high-quality results at a low cost. s69 00:06:35.520 --> 00:06:40.000 One of the biggest learnings in this process was like the model doesn't know that it's wrong. s70 00:06:40.000 --> 00:06:45.240 It always confidently says 100% sure that yeah, this is the review for your code. s71 00:06:45.240 --> 00:06:45.800 Go ahead. s72 00:06:45.800 --> 00:06:57.400 But we saw that no, it actually needs a lot of guidance from the teams because each team has its own style guide, its own patterns or like anti-patterns that they want to look for. s73 00:06:57.400 --> 00:07:00.440 So that all should be like baked into the agent. s74 00:07:00.440 --> 00:07:04.320 And we also realized that we need to have guardrails for the agent. s75 00:07:04.320 --> 00:07:07.640 So we need to tell the agent what not to waste turns doing. s76 00:07:07.640 --> 00:07:12.919 Like code review is something that has to happen in like a specific time span. s77 00:07:12.919 --> 00:07:20.760 And then if it starts spending time doing things that it should not be doing, uh leads to a bad quality code review. s78 00:07:20.760 --> 00:07:26.160 Uh second focus area for U review has been nations. s79 00:07:26.160 --> 00:07:37.280 We We went very deep on team customizations because as we'll presented that we have hundreds of teams and everyone has like their own way or their own thing for code review. s80 00:07:37.280 --> 00:07:39.440 So our review stack is pretty straightforward. s81 00:07:39.440 --> 00:07:43.200 We have single-file reviewers and multi-file reviewers. s82 00:07:43.200 --> 00:07:49.640 Uh, we basically do a general purpose "Hey, find me all logic bugs per file" uh, kind of a review. s83 00:07:49.640 --> 00:07:54.040 And uh, then we also do a deep review because we have like six mono repos. s84 00:07:54.040 --> 00:08:02.400 So, all these mono repos have their own anti-pattern style guides and all baked into this agent review which does have a nice multi-file review. s85 00:08:02.400 --> 00:08:06.640 But, then we extended it further uh, basically to AI linters. s86 00:08:06.640 --> 00:08:20.800 These are basically few shot uh, AI problem uh, or like a few shots uh, system where uh, developers can basically kind of deterministically get more context and then run rules s87 00:08:20.800 --> 00:08:26.800 with that context and like a file and find some uh, systematic and mechanical issues. s88 00:08:26.800 --> 00:08:38.080 And finally uh, the most powerful thing is the custom agent uh, where the teams could basically define their own custom agent, link it to like a knowledge base, uh, link it to their past PRs, s89 00:08:38.080 --> 00:08:41.919 have like a skill to do the review, and so on. s90 00:08:41.919 --> 00:08:54.120 But, uh, all of this was not simple because we had to actually uh, piggyback on our uh, ownership model which is at Uber uh, so that we can like very logically roll out to all the teams. s91 00:08:54.120 --> 00:08:57.960 Uh, we had to basically do a uh, what do you say? s92 00:08:57.960 --> 00:09:05.920 Co-locate the customizations next to where the developers write their code so that they can like quickly uh, keep updating these customizations. s93 00:09:05.920 --> 00:09:16.480 We had to implement a smart deterministic uh, routing so that we could route which team gets what kind of review with which model, what kind of generators, and so on. s94 00:09:16.480 --> 00:09:28.040 And finally uh, the hard thing was like we had to actually surface all of this observability that I talked before, like the agent trajectory, addressal rate, uh, sentiment analysis back to the teams. s95 00:09:28.040 --> 00:09:37.040 So, so that the teams could actually understand that "Oh, I wrote this rule, but maybe not a lot of developers are liking it in my team, so let me go and update it." s96 00:09:37.040 --> 00:09:43.680 And then we had to give Bubble up that kind of observability to all the people who are contributing to the platform. s97 00:09:43.680 --> 00:09:49.000 Uh one thing that we learned is that actually writing the skill was very easy. s98 00:09:49.000 --> 00:09:58.040 Like teams just very quickly wrote a skill by asking Claude to write one, go over my previous PR reviews and write me a skill. s99 00:09:58.040 --> 00:10:04.320 But the hard part was how to run these skills at scale with consistent quality and low cost. s100 00:10:04.320 --> 00:10:12.120 And that required a lot of iterations not only from the U-Review team side, but also like for each team who was trying to write these rules. s101 00:10:12.120 --> 00:10:20.680 Uh in results, we basically uh see that, you know, uh U-Review does like around 25,000 comments a week. s102 00:10:20.680 --> 00:10:24.480 And uh we get 10% of them actually get some feedback. s103 00:10:24.480 --> 00:10:28.280 And only 4% of the PRs actually get some negative feedback. s104 00:10:28.280 --> 00:10:43.520 Uh we also saw that um the overall addressal rate was uh around 67% and almost three quarters of the high severity issues uh were usually addressed by the developers, which shows that U-Review actually s105 00:10:43.520 --> 00:10:46.680 adds some value to the entire development life cycle. s106 00:10:46.680 --> 00:10:57.160 And then uh with all the observability and uh evals that I showed that I went through, we saw that against like a very naive implementation, our costs were down by 60% s107 00:10:57.160 --> 00:11:02.160 and our quality and our accuracy was up by uh around 70%. s108 00:11:02.160 --> 00:11:10.080 Uh for a last focus area, I'll give the mic back to Will and he will go over the inner versus outer loop. s109 00:11:11.600 --> 00:11:11.960 Awesome. s110 00:11:11.960 --> 00:11:24.080 So, now that we've talked about uh some of the details of actually implementing high-quality reviews, it kind of brings us to the last area, which is where we start talking about s111 00:11:24.080 --> 00:11:25.640 where things are going, right? s112 00:11:25.640 --> 00:11:35.400 With moving to the Agentech SDLC, we're moving software into a model where engineers are interacting with the code less. s113 00:11:35.400 --> 00:11:38.840 They're often times not as involved in authoring the code. s114 00:11:38.840 --> 00:11:52.680 Uh currently, we still have uh humans approving the code, uh but we see a a short path in the near future to a percentage of our code landing automatically, having automatic approvals, s115 00:11:52.680 --> 00:11:53.160 right? s116 00:11:53.160 --> 00:11:56.400 The various parts of the industry are already moving there. s117 00:11:56.400 --> 00:12:10.040 Um Part of the way along the process was figuring out by having our single code review platform, what did we need to tune for the various audiences that are actually getting these code reviews? s118 00:12:10.040 --> 00:12:14.480 Um you know, the interface, that's one area that's sort of intuitive there. s119 00:12:14.480 --> 00:12:18.400 Uh one thing that might be less intuitive is around accuracy. s120 00:12:18.400 --> 00:12:33.840 Uh with the inner loop, our accuracy needs actually need to go up, or else we can result in uh dealing with cavitation of an agent where it fixes something, goes back, gets another code review, and has to kind of like fix backwards because the quality of the comment s121 00:12:33.840 --> 00:12:35.000 was low. s122 00:12:35.000 --> 00:12:47.040 Um The one of the other interesting things is agents are more than happy to go through and fix 100 nits on a pull request where your engineers really get frustrated in situations like that. s123 00:12:47.040 --> 00:12:53.320 Um but probably the most interesting aspect of this transition is the feedback. s124 00:12:53.320 --> 00:13:08.080 As you can see, quite a bit of what went into getting high-quality code reviews at Uber was bringing the human feedback into the system and using that to figure out how to tune our prompts, how to tune our agents. s125 00:13:08.080 --> 00:13:22.800 Uh and so as we move to a model where humans are less in the loop, where software engineering is moving to an agentic model, we're effectively going to a place where we're starting to talk about s126 00:13:22.800 --> 00:13:24.560 are we going to kill the outer loop? s127 00:13:24.560 --> 00:13:29.400 Is the human engineer not going to be involved in the code review. s128 00:13:29.400 --> 00:13:31.560 Some people are already here. s129 00:13:31.560 --> 00:13:39.240 Now, with the feedback taken into consideration, you start wondering, all right, what could this result in, right? s130 00:13:39.240 --> 00:13:43.760 I'll let your imagination go there in terms of quality degradation, slop, and so forth. s131 00:13:43.760 --> 00:13:54.200 But, rather than killing the outer loop, I think that we believe and the industry has just started to really kind of coalesce on this idea that we're really expanding s132 00:13:54.200 --> 00:13:55.000 the outer loop. s133 00:13:55.000 --> 00:14:03.560 Rather than removing humans from the code review process, we are moving their responsibilities up a layer. s134 00:14:03.560 --> 00:14:09.400 Rather than them dealing with the details of the implementation, the agent is great at writing the software. s135 00:14:09.400 --> 00:14:14.240 The agent is getting much, much better at reviewing the software as a human would. s136 00:14:14.240 --> 00:14:17.640 But now, as software engineers, we still are going to have an outer loop. s137 00:14:17.640 --> 00:14:19.400 It's just going to look a little different. s138 00:14:19.400 --> 00:14:28.600 Instead of you worrying about the optimization of the performance and the API compatibility, you're going to be thinking more about architecture in your code reviews. s139 00:14:28.600 --> 00:14:34.440 You're going to have time to focus on the domain expertise that you have and product thinking. s140 00:14:34.440 --> 00:14:46.040 So, we believe that as we adopt this automated uh code review, this is going to be the result of how our engineers are interacting with the system and guiding it. s141 00:14:47.360 --> 00:14:48.120 And that's it. s142 00:14:48.120 --> 00:14:49.440 Thank you so much for coming. s143 00:14:49.440 --> 00:14:51.440 Thanks.