WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=474j-n1Ltxc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/474j-n1Ltxc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.400 --> 00:00:14.799 How's everyone doing? s2 00:00:14.799 --> 00:00:15.440 Awesome. s3 00:00:15.440 --> 00:00:16.480 Uh, my name is Du. s4 00:00:16.480 --> 00:00:19.760 I'm one of the co-founders of a company called Gravile. s5 00:00:19.760 --> 00:00:27.199 And at Grapile, we're working on AI agents that validate poll requests with full context of the codebase. s6 00:00:27.199 --> 00:00:37.600 The way Graval does this is every time there's a pull request, we have a swarm of agents and they go and look at every file that's changed, every file that's related to the files that changed to figure out if there's bugs. s7 00:00:37.600 --> 00:00:49.840 And it also spins up your code in a sandbox, installs the dependencies, spins up local host, clicks around to try to break things, mocks inputs, and everything else to figure out if the code is broken. s8 00:00:49.840 --> 00:00:56.239 But I'm going to talk about something a little bit different, which is fully autonomous coding agents. s9 00:00:56.239 --> 00:01:00.559 So I first moved to San Francisco three years ago to work on AI coding. s10 00:01:00.559 --> 00:01:09.280 And this is because GBT 3.5 which came out in 2022 was the first model that seemed like it was really good at programming at the time. s11 00:01:09.280 --> 00:01:13.040 Code complete tab complete was like the main paradigm for coding. s12 00:01:13.040 --> 00:01:22.240 The most exciting thing that was happening was you had tab complete from cursor and you had co-pilots code complete and that was like the way in which AI coding works. s13 00:01:22.240 --> 00:01:26.880 In 2024, for the first time, multifile editing started to work. s14 00:01:26.880 --> 00:01:30.320 Cursor was the first to do this and then a couple of other products followed. s15 00:01:30.320 --> 00:01:35.280 But now, for the first time, AI could simultaneously edit multiple files at once. s16 00:01:35.280 --> 00:01:39.360 But things got really interesting in 2025 because for the first time, we had autonomous agents. s17 00:01:39.360 --> 00:01:43.840 They could be given a task and they could go and fulfill the task and create entire pull requests all at once. s18 00:01:43.840 --> 00:01:49.840 And this came to a precipice in December of last year when coding agents became literally completely autonomous. s19 00:01:49.840 --> 00:02:00.399 And anyone that's working in AI coding knows that December when the new models came out was sort of this like watershed moment in the history of AI coding because these products for the first time were like truly autonomous in how they programmed. s20 00:02:00.399 --> 00:02:07.920 And all over Twitter were all these crazy stories of people that were having these agents spin off and open a 100 pull requests a day. s21 00:02:07.920 --> 00:02:14.400 They were experimenting with polyphasic sleep so they could stay awake to re-trigger their agents from time to time. s22 00:02:14.400 --> 00:02:21.520 And as someone that started programming before agents, I was very skeptical that real companies could program this way. s23 00:02:21.520 --> 00:02:25.520 So I was really really curious, are these fully vi PRs really actually good? s24 00:02:25.520 --> 00:02:32.480 Is this something that's sort of a Twitter hype and you can have independent developers and and maybe like small startups with no real customers do this? s25 00:02:32.480 --> 00:02:42.239 But anyone with real customers with an actually commercially viable code base, could they still use these end-to-end coding agents? s26 00:02:42.239 --> 00:02:46.000 And at Grubile, we're fortunate to work with some very large companies. s27 00:02:46.000 --> 00:02:49.920 We work with Nvidia, with Coinbase, with Scale, with Data Dog, with American Express. s28 00:02:49.920 --> 00:03:01.360 And we got really excited to see if these coding agents were being used really at these large companies and if they were any good at doing real world coding. s29 00:03:02.640 --> 00:03:06.319 So, as an amateur data scientist, I decided to go and dive into our data. s30 00:03:06.319 --> 00:03:12.400 We review more than a million poll requests a month and we tend to get a lot of interesting data on what's good and bad about these poll requests. s31 00:03:12.400 --> 00:03:19.440 We do this for thousands and thousands of companies and they're usually enterprises or at least companies with serious products with real customers. s32 00:03:19.440 --> 00:03:23.599 And so we figured actually studying this data would yield some really interesting results. s33 00:03:23.599 --> 00:03:30.959 Was a surprisingly hard to figure out from all the poll requests that GR was reviewing which of the poll requests was actually generated by AI. s34 00:03:30.959 --> 00:03:32.879 Surprisingly difficult to do this. s35 00:03:32.879 --> 00:03:37.440 The first thing that I tried was I started looking at the GitHub author field. s36 00:03:37.440 --> 00:03:47.760 GitHub every commit has an author field and so went through the authors and turns out that less than 1% of pull requests had codeex or clawed or cursor as the author. s37 00:03:47.760 --> 00:03:51.360 But it wasn't intuitive to me that only but 1% of all code was fully AI generated. s38 00:03:51.360 --> 00:03:53.680 It just seemed intuitive that that number would be much higher. s39 00:03:53.680 --> 00:03:55.760 So I started to look for other signals. s40 00:03:55.760 --> 00:04:01.439 Now thankfully Claude and Codex and some other products leave a PR description in the footer. s41 00:04:01.439 --> 00:04:06.480 In the PR description, they put something along the lines of co-authored by claude or co-authored by cursor and so on. s42 00:04:06.480 --> 00:04:13.519 And this gave me a little bit more signal, a little bit more data on which PRs were ostensibly largely, if not completely AI generated. s43 00:04:13.519 --> 00:04:15.680 The third thing I looked at was branch name prefixes. s44 00:04:15.680 --> 00:04:18.479 If you use Codex, you know that Codex names as branches. s45 00:04:18.479 --> 00:04:26.960 And it seemed reasonable that if someone was using these coding agents in such a sense that they were writing the branch names or writing the PR descriptions that the PRs were largely AI generated. s46 00:04:26.960 --> 00:04:28.800 That seemed like a reasonable assumption to me. s47 00:04:28.800 --> 00:04:36.000 So now I had a good set of signals to indicate whether a poll request was in fact completely or largely AI generated or not. s48 00:04:36.000 --> 00:04:45.919 And I came to the result that about a quarter of all the poll requests that Graal was reviewing in any given month were completely or at least largely generated by AI. s49 00:04:46.560 --> 00:04:51.360 Then I backtracked this data to the last 12 months and it turns out that this number is growing really fast. s50 00:04:51.360 --> 00:04:57.199 In fact, early last year, fewer than 1% of pull requests had any evidence of being completely AI generated. s51 00:04:57.199 --> 00:05:00.639 So this number is going very very fast as model performance is getting better. s52 00:05:00.639 --> 00:05:05.280 What's also interesting is that you can't tell when new models came out on this chart. s53 00:05:05.280 --> 00:05:07.520 It seemed like the progress is very continuous. s54 00:05:07.520 --> 00:05:13.120 These things are just generally diffusing into the economy at a generally high pace. s55 00:05:13.120 --> 00:05:14.960 So then came the next question. s56 00:05:14.960 --> 00:05:16.080 Everyone's vibe coding. s57 00:05:16.080 --> 00:05:18.320 Everyone's producing these PRs that are antiendentic. s58 00:05:18.320 --> 00:05:20.240 Not human plus AI but literally AI. s59 00:05:20.240 --> 00:05:23.120 The question is are these PRs any good? s60 00:05:23.120 --> 00:05:24.960 First question, what does it mean for PR to be good? s61 00:05:24.960 --> 00:05:30.240 That seemed like actually a very important question to answer before we got into judging these AI generated PRs. s62 00:05:30.240 --> 00:05:32.160 And so I took a few methods to try this. s63 00:05:32.160 --> 00:05:34.400 The first one I tried was revert rates. s64 00:05:34.400 --> 00:05:37.120 If a pull request reverted, it was probably pretty bad. s65 00:05:37.120 --> 00:05:42.880 And so that seemed like a pretty sincere sort of guess at what a bad PR could be. s66 00:05:42.880 --> 00:05:46.320 And so I started tracking, okay, which PRs were in fact reverted. s67 00:05:46.320 --> 00:05:51.440 GitHub conveniently names its branches with a revert dash PR number and PR name. s68 00:05:51.440 --> 00:05:54.479 And so I was able to track the rate at which these things were being reverted. s69 00:05:54.479 --> 00:05:56.320 And I found some interesting data. s70 00:05:56.320 --> 00:05:59.600 Codex PR is reverted about one out of every thousand poll requests. s71 00:05:59.600 --> 00:06:03.440 Devon once every three and a half times every thousand poll requests. s72 00:06:03.440 --> 00:06:05.360 Humans right in the middle at about two and a half. s73 00:06:05.360 --> 00:06:12.080 So there doesn't seem to be very big difference between the rate at which poll requests were reverted from people versus agents in my study. s74 00:06:12.080 --> 00:06:26.639 Now I was very skeptical of this and I figured okay this is probably because humans are making agents do the easier work the simpler well scoped pull requests and the more complex work was being done by humans and of course complex work had a higher propensity to be reverted because the poll requests were more complicated had larger s75 00:06:26.639 --> 00:06:28.800 surface areas risks and so on. s76 00:06:28.800 --> 00:06:36.880 So I decided to measure the average size of PR and whether the revert rates were sort of aligned for both of those and I found really interesting data. s77 00:06:36.880 --> 00:06:44.560 Turns out there's actually very little correlation, if any, between the size of PRs of humans were getting reverted versus size of PRs from agents that were being reverted. s78 00:06:44.560 --> 00:06:52.000 So this does not seem like actually very strong evidence that the human PRs are any better than the agent PRs. s79 00:06:52.000 --> 00:06:54.720 The second set of signals I started looking at was Graile comments. s80 00:06:54.720 --> 00:07:07.120 Now, now Grapple is reviewing all these poll requests and Graile finds P 0's, P1's, P2s across all these changes and it seemed reasonable that if Grapile was finding more bugs in the code that the code was most likely worse. s81 00:07:07.120 --> 00:07:12.720 So then I started tracking the counts of P 0, P1's and P2s in all these changes. s82 00:07:12.720 --> 00:07:15.840 And interestingly, once again, there was not that much of a difference. s83 00:07:15.840 --> 00:07:25.199 In fact, three out of the four agents that we tested performed better than humans in terms of the rate at which they were producing P 0. they were producing fewer P 0 than humans were. s84 00:07:25.199 --> 00:07:28.240 Similarly the case with P1's and P2s. s85 00:07:28.240 --> 00:07:36.080 Broadly speaking, human generated PRs were about equal in quality to agent generated PRs based on this data. s86 00:07:43.840 --> 00:07:49.520 have the agent go and look at those comments, address them, and make a new commit on the pull request branch. s87 00:07:49.520 --> 00:07:51.360 That is the most common way to use guptile. s88 00:07:51.360 --> 00:07:57.120 And so it seems reasonable that if the pull requests were higher quality, it would require fewer iterations before they would be merged. s89 00:07:57.120 --> 00:08:04.639 And so I started tracking the number of iterations, the number of review rounds between when the poll request was opened and when it was merged. s90 00:08:04.639 --> 00:08:06.879 Sure enough, very little difference. s91 00:08:06.879 --> 00:08:14.639 Devon's PRs 2.1, Codex PR is 2.45, four, five number of review cycles to merge and humans right in the middle. s92 00:08:14.639 --> 00:08:24.000 Once again, very little if any statistical difference between human generated and AI generated pull request in terms of how many iterations before they were ready to merge. s93 00:08:24.400 --> 00:08:31.840 So now that there wasn't any strong evidence that agent generated PRs were worse than human generated PRs, I got curious if there were qualitative differences. s94 00:08:31.840 --> 00:08:36.880 Maybe the types of ways in which agents failed were different from the types of ways that humans failed. s95 00:08:36.880 --> 00:08:39.839 And so then I looked at the corpus of reptiles comments. s96 00:08:39.839 --> 00:08:42.320 It makes on average about four comments per pull request. s97 00:08:42.320 --> 00:08:46.160 So we had this corpus of several million comments across the last several months. s98 00:08:46.160 --> 00:08:49.680 And so I started scanning them for specific phrases and words. s99 00:08:49.680 --> 00:08:53.120 For instance, SQL injection or N plus1 query. s100 00:08:53.120 --> 00:08:58.480 And I plotted the frequency with which these terms occur in graphile comments for these various agents. s101 00:08:58.959 --> 00:09:03.040 And so I made this chart of patterns of failure. s102 00:09:03.279 --> 00:09:09.600 To interpret this chart, you can assume that 1x is the human propensity for producing that type of error across the entire chart. s103 00:09:09.600 --> 00:09:16.320 And you can kind of see that there's actually quite a lot of variation in the types of failures that these agents seem to produce. s104 00:09:16.320 --> 00:09:22.000 For instance, Claude is one and a half times more likely to produce a SQL injection error than humans. s105 00:09:22.000 --> 00:09:29.120 Devon is about half as likely as humans to produce a off bypass issue. s106 00:09:29.120 --> 00:09:37.200 I found it very interesting that there was this much variation in how these agents were performing and how different their failure modes were from humans. s107 00:09:38.080 --> 00:09:52.160 So it turns out that in spite of my initial skepticism around the enterprise usability of endto-end coding agents, the evidence seems to suggest that they're here and they probably can contribute in real meaningful ways to enterprise s108 00:09:52.160 --> 00:09:53.920 coding environments. s109 00:09:53.920 --> 00:09:57.760 And so we started to think a little bit more about what code review would look like in such a world. s110 00:09:57.760 --> 00:10:00.320 Here's an interesting stat. s111 00:10:00.320 --> 00:10:05.839 Today, Graal is used by several tens of thousands of engineers every single week to review all of their code. s112 00:10:05.839 --> 00:10:11.200 So, we also know generally speaking the number of pull requests that each of these people are writing. s113 00:10:11.200 --> 00:10:14.800 The median grubile user writes 50 pull requests a month. s114 00:10:14.800 --> 00:10:17.360 So, call it about two plus per workday. s115 00:10:17.360 --> 00:10:20.720 The 90th percentile writes 500 poll requests per month. s116 00:10:20.720 --> 00:10:24.079 That is a drastic difference between the median and the P90. s117 00:10:24.079 --> 00:10:27.200 The P99 is in the thousands of pull requests. s118 00:10:27.200 --> 00:10:34.800 That is very interesting in its own sense because that means that the people on the margins are actually producing poll requests at the rate at which they're coming up with new ideas. s119 00:10:34.800 --> 00:10:47.120 And beyond being interesting, it also makes you wonder well there existing systems for validating this code which is manual code review of course testing maybe you have a QA firm that you work with naturally can scale to that same degree. s120 00:10:47.120 --> 00:10:52.000 And at Grav, we decided to take sort of a first principles view at what really good validation could look like. s121 00:10:52.000 --> 00:11:05.279 Instead of saying that we wanted to automate QA or automate testing or automate code review, we took a step back and said, what would we need to do for anyone to be able to merge hundreds of pull requests a month in an enterprise environment where it really matters that the code is correct? s122 00:11:05.279 --> 00:11:11.200 What would need to happen between when the code was expressed into a pull request and when it was merged and deployed safely? s123 00:11:11.200 --> 00:11:14.160 We figured we only actually had to answer three questions. s124 00:11:14.160 --> 00:11:17.519 The first one is does this change violate the user contract? s125 00:11:17.519 --> 00:11:24.880 The second one, does it increase the propensity of a future violation of the user contract, whatever the user contract might be for that application? s126 00:11:24.880 --> 00:11:27.600 And third, does it fulfill the intent that the author described? s127 00:11:27.600 --> 00:11:31.360 Did the poll request do the thing that the author wanted to do? s128 00:11:31.360 --> 00:11:41.120 And so we started approaching this problem from base ground level and said, okay, agents can probably figure out if something's going to violate the user contract and detect bugs. s129 00:11:41.120 --> 00:11:51.040 If you let it spin up the code in a sandbox, have it install the dependencies, mock the inputs, run the browser agents, you can probably start to discover most of the issues that could occur, s130 00:11:51.040 --> 00:11:53.680 you get a pretty high degree of confidence on merge. s131 00:11:53.680 --> 00:12:04.000 Today, almost a fifth of all the poll request or gravile reviews are merged without any human review or without any human testing, which I find very interesting and that is a number that we care a lot about and want to bring higher and higher, s132 00:12:04.000 --> 00:12:08.720 of course, within the guard rails of producing really high quality code. s133 00:12:10.000 --> 00:12:10.800 Thank you so much. s134 00:12:10.800 --> 00:12:13.519 My name is D, one of the co-founders of Graptile. s135 00:12:13.519 --> 00:12:15.360 We have a booth here which you should come to. s136 00:12:15.360 --> 00:12:21.040 And if you're interested in trying Graptile, you can find us at gretile.com and try it today for free. s137 00:12:21.040 --> 00:12:22.800 And we'd love to hear your feedback. s138 00:12:22.800 --> 00:12:25.200 Thank you so much.