WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Ib5GBkD555M NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Ib5GBkD555M.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:25.280 --> 00:00:26.960 What's up everybody? s3 00:00:26.960 --> 00:00:29.160 How we doing? s4 00:00:29.480 --> 00:00:33.280 Guys, give it up for all the great speakers today so far. s5 00:00:33.640 --> 00:00:34.715 Um s6 00:00:34.715 --> 00:00:36.040 [applause] s7 00:00:36.040 --> 00:00:42.120 All right, this is Harness Engineering is not enough and why software factories fail. s8 00:00:42.120 --> 00:00:44.760 And we're going to click, maybe. s9 00:00:44.760 --> 00:00:46.760 Oop. s10 00:00:47.520 --> 00:00:48.920 Oh, that's way too many slides. s11 00:00:48.920 --> 00:00:50.160 Hold on, guys. s12 00:00:50.160 --> 00:00:51.480 Okay. s13 00:00:51.480 --> 00:00:55.960 Um So, we're all racing to put AI coding into production. s14 00:00:55.960 --> 00:00:59.040 And uh there's been lots been said about loop engineering. s15 00:00:59.040 --> 00:01:01.560 And uh we should probably write more loops. s16 00:01:01.560 --> 00:01:03.160 And uh yeah, I don't know. s17 00:01:03.160 --> 00:01:05.840 I guess we're doing loops now. s18 00:01:06.200 --> 00:01:10.520 Uh StrongDM built a lights-out software factory where nobody even reads the code. s19 00:01:10.520 --> 00:01:15.800 And the prevailing narrative is we should just spend more tokens. s20 00:01:15.800 --> 00:01:17.640 You are the bottleneck. s21 00:01:17.640 --> 00:01:18.760 The models are good enough. s22 00:01:18.760 --> 00:01:20.080 Code is free. s23 00:01:20.080 --> 00:01:22.240 Just ship more stuff. s24 00:01:22.240 --> 00:01:26.040 But at the same time, we are starting to see the cracks. s25 00:01:26.040 --> 00:01:36.960 Our friend Mario at AI Engineer Europe begged us to slow down because companies that should not be having outages because of coding agents are having outages due to coding agent mishaps. s26 00:01:36.960 --> 00:01:41.120 Um codebases are falling apart faster than they ever have before. s27 00:01:41.120 --> 00:01:48.680 And our friends at Faros AI actually even did a report since we all picked up all these AI coding tools in January, maybe February. s28 00:01:48.680 --> 00:01:51.840 Um pull request code review quality is way down. s29 00:01:51.840 --> 00:01:57.520 We're having more comments, longer comments, and tons of PRs being merged without any review at all. s30 00:01:57.520 --> 00:02:04.120 Incidents are way up, bugs per developer are way up, and uh many people will tell you that you're holding it wrong. s31 00:02:04.120 --> 00:02:05.640 That's the only reason. s32 00:02:05.640 --> 00:02:07.720 You're not. s33 00:02:07.880 --> 00:02:11.000 Well, maybe you are, but that's not the point. s34 00:02:11.000 --> 00:02:15.520 Um I've spoken a lot about how to hold it better when it comes to working with AI. s35 00:02:15.520 --> 00:02:20.000 Uh probably a million views on YouTube at this point across a bunch of different talks. s36 00:02:20.000 --> 00:02:27.000 Um and the basic thing is like as engineers, we've been told that if token maxing isn't working, then it's a skill issue. s37 00:02:27.000 --> 00:02:28.840 You just need to spend more tokens. s38 00:02:28.840 --> 00:02:31.200 Uh let go of reading the code. s39 00:02:31.200 --> 00:02:42.200 That with enough harness engineering, if we maybe sprinkle some magic words, adversarial review on enough of our uh PR bots, that we can get the best of both worlds. s40 00:02:42.200 --> 00:02:48.800 10 to 100x faster, high quality, and nobody has to do that thing we all hate called code review. s41 00:02:48.800 --> 00:02:53.040 Uh I'm here to convince you today that this is in fact not a skill issue. s42 00:02:53.040 --> 00:03:00.280 That no amount of harness engineering or loops maxing can solve what is fundamentally a model training issue. s43 00:03:00.280 --> 00:03:02.800 That's why we say the harness is not enough. s44 00:03:02.800 --> 00:03:08.600 Um and to understand this, we kind of have to grapple and dig into how coding models are trained. s45 00:03:08.600 --> 00:03:13.720 I'm going to talk about what I think the shortcomings are with some of the current benchmarks and what better ones might look like. s46 00:03:13.720 --> 00:03:17.480 And we'll talk about how to move faster safely in the meantime. s47 00:03:17.480 --> 00:03:20.080 Um it's going to sound like a rant, uh but there is hope here. s48 00:03:20.080 --> 00:03:23.840 Uh I'm going to talk about our journey and a bunch of the landmines we've hit uh building in this world. s49 00:03:23.840 --> 00:03:28.800 A bunch of exciting new techniques that we've been working with uh a lot of our users and customers to develop. s50 00:03:28.800 --> 00:03:36.280 And I think how we all as a community get to the next chapter of agentic engineering after whatever this thing that we're in. s51 00:03:36.280 --> 00:03:37.360 Um so we use a lot of words here. s52 00:03:37.360 --> 00:03:38.239 I'm going to zoom out a little bit. s53 00:03:38.239 --> 00:03:41.560 I want to give you kind of like a brief history of the software factory. s54 00:03:41.560 --> 00:03:44.360 Um and it's actually I don't I I I just learned this last week. s55 00:03:44.360 --> 00:03:49.400 It was the term software factory was defined at a NATO conference in 1968. s56 00:03:49.400 --> 00:03:53.160 Uh we're going to start around 2022, like right before AI started coming around. s57 00:03:53.160 --> 00:03:59.600 Um and basically in a typical 2022 software factory, you will have some people building stuff. s58 00:03:59.600 --> 00:04:04.920 You'll have engineers, you'll have PMs, maybe you have some sort of leadership team that is driving the vision here. s59 00:04:04.920 --> 00:04:07.120 And they all decide that stuff needs to get done. s60 00:04:07.120 --> 00:04:13.720 And so you put it in a tracker, a linear, a Jira, a beads, some sort of state machine that tracks what needs to be done. s61 00:04:13.720 --> 00:04:17.160 And then someone goes and grabs something off of there, and they build the thing. s62 00:04:17.160 --> 00:04:21.760 And there may be some automated testing in that process, maybe some manual manual testing in that process. s63 00:04:21.760 --> 00:04:24.400 At a certain point, we make this pull request thing. s64 00:04:24.400 --> 00:04:25.160 Says, "Okay, cool. s65 00:04:25.160 --> 00:04:27.200 We got to run a bunch of checks, automated stuff. s66 00:04:27.200 --> 00:04:29.800 A human's going to review the change and review the code. s67 00:04:29.800 --> 00:04:34.040 And perhaps we might even have uh a human pull it down and test it somehow." s68 00:04:34.040 --> 00:04:37.880 And if anything goes wrong here, we loop back to someone builds the thing. s69 00:04:37.880 --> 00:04:39.560 Uh and eventually we're ready for prod. s70 00:04:39.560 --> 00:04:41.160 And so we ship it to production. s71 00:04:41.160 --> 00:04:45.160 And once it's in prod, it makes contact with our users. s72 00:04:45.160 --> 00:04:48.240 And users do uh a thing that we all love. s73 00:04:48.240 --> 00:04:51.000 Uh users love to complain. s74 00:04:51.000 --> 00:04:52.160 I I love our users. s75 00:04:52.160 --> 00:04:56.280 Uh but yeah, they're going to ask for things, they're going to find bugs, they're going to file feature requests. s76 00:04:56.280 --> 00:04:57.919 Uh and that goes back to your team. s77 00:04:57.919 --> 00:05:00.040 You might also add monitoring. s78 00:05:00.040 --> 00:05:03.160 And so uh you know, what do we want more than anything else? s79 00:05:03.160 --> 00:05:06.360 We want to wake up engineers at 3:00 in the morning when something breaks. s80 00:05:06.360 --> 00:05:08.760 So they can get dragged out of bed to try to go fix it. s81 00:05:08.760 --> 00:05:10.919 Uh and we go on and on in this loop. s82 00:05:10.919 --> 00:05:13.240 Uh and we ship a bunch of code. s83 00:05:13.240 --> 00:05:24.800 Um and one thing that we noticed here is that uh teams figured this out decades ago is that this someone builds the thing step is usually going to take hours or days in most cases. s84 00:05:24.800 --> 00:05:28.440 And the review part will also take hours or days for large things. s85 00:05:28.440 --> 00:05:40.520 And so teams started doing these upfront planning, architectural proposals, sprint planning, and they would collaborate this on on these things as a team uh with the hopes that we might decrease the percent chance that something would need to be reworked. s86 00:05:40.520 --> 00:05:47.480 That we would be able to reduce the time spent in reviewing every line of code because we aligned on everything ahead of time. s87 00:05:47.480 --> 00:05:50.720 This brings us to the agentic software factory. s88 00:05:50.720 --> 00:05:58.800 Uh every company and their mother is talking about how they built a coding agent factory that ships 75% of their code now. s89 00:05:58.800 --> 00:06:01.000 Uh literally everybody. s90 00:06:01.000 --> 00:06:07.840 Uh and so if we look at the software factory from 2022, uh we just replace someone builds the thing with an agent builds the thing. s91 00:06:07.840 --> 00:06:12.840 And we have an orchestration and a harness and a sandbox and a model and computer use and I'm not going to get into like the details of that. s92 00:06:12.840 --> 00:06:15.600 You can watch 100 talks about that this week, I'm sure. s93 00:06:15.600 --> 00:06:23.320 Um but now the building part takes minutes or hours, but this human part still takes hours or days if you're going to review the code and you're going to test the changes. s94 00:06:23.320 --> 00:06:24.960 And so we bring in agentic code review. s95 00:06:24.960 --> 00:06:31.600 We bring in agentic regression testing, uh and it makes this part faster, but it's probably still the bottleneck. s96 00:06:31.600 --> 00:06:32.680 But we can do more loops here. s97 00:06:32.680 --> 00:06:33.120 Why not? s98 00:06:33.120 --> 00:06:34.360 Let's do some more loops. s99 00:06:34.360 --> 00:06:36.440 So we can route all incidents straight into the factory. s100 00:06:36.440 --> 00:06:43.560 Why does someone need to get woken up uh and try to fix it when they could just wake up to a pull request and uh maybe that fixes the issue for you. s101 00:06:43.560 --> 00:06:49.919 You can take all the user feedback and just stick it straight into the factory so that people ask for stuff and it gets built. s102 00:06:49.919 --> 00:06:57.880 And now your only job is how much things can you stuff into the queue of stuff to do and how fast can you review and test the changes? s103 00:06:57.880 --> 00:07:05.080 Which brings us of course to, I'm sure you know, the lights off software factory where basically Dentsu Bureau coined this is we no longer read the code. s104 00:07:05.080 --> 00:07:06.080 We say, "You know what? s105 00:07:06.080 --> 00:07:07.240 This is going great. s106 00:07:07.240 --> 00:07:08.560 That code review thing? s107 00:07:08.560 --> 00:07:09.440 No, thanks. s108 00:07:09.440 --> 00:07:11.400 We're just not going to do that anymore." s109 00:07:11.400 --> 00:07:14.440 Uh and we invest into all these other parts of the system. s110 00:07:14.440 --> 00:07:17.040 Your your testing, your monitoring, your rollout, everything else. s111 00:07:17.040 --> 00:07:19.800 We just write more code and build those systems better. s112 00:07:19.800 --> 00:07:25.080 And now our job really is just how much how much stuff can we ask the agent to build? s113 00:07:25.320 --> 00:07:28.440 I am going to posit that this does not work. s114 00:07:28.440 --> 00:07:31.080 Uh and this is why software factories fail. s115 00:07:31.080 --> 00:07:34.480 Um as as an aside, what I'm going to say has nothing to do with vibe coding. s116 00:07:34.480 --> 00:07:36.400 So Addy had this uh great post. s117 00:07:36.400 --> 00:07:38.440 I'm going to just going to literally take his quote verbatim. s118 00:07:38.440 --> 00:07:49.880 A developer vibe coding a side project a dozen people will ever run, and a team keeping a 10-year-old enterprise system alive for another quarter share almost no constraints worth naming. s119 00:07:49.880 --> 00:07:56.480 And most of what you hear on the internet is one of these groups of people telling the other group of people how to live their lives. s120 00:07:56.480 --> 00:07:59.760 So, if you love vibe coding, please go on. s121 00:07:59.760 --> 00:08:05.200 Um at Human Layer, what we care about is how do we help people solve hard problems in complex codebases. s122 00:08:05.200 --> 00:08:10.760 Um we use the word brownfield a lot, which historically has meant like some 10-year-old Java thing. s123 00:08:10.760 --> 00:08:16.840 I actually think agents really start to struggle after maybe 3 to 6 months, especially with the pace at which we can ship now. s124 00:08:16.840 --> 00:08:21.880 Um you can ask me how I know this, and I will tell you that it is because in July 2025, we tried this. s125 00:08:21.880 --> 00:08:31.600 We went full lights off, and uh if you have tried this seriously for a number of months, you probably found at least one issue that the agent couldn't solve. s126 00:08:31.600 --> 00:08:42.560 Even with your most advanced prompting, you do research, you do reproductions, you just you have to go and dig into that codebase that you stopped reading 3 months ago to try to figure out what's broken. s127 00:08:42.560 --> 00:08:50.040 And in the meantime, your site was down, your users were pissed, and you were If you were like me, you were probably miserable reading all this slop code that you let s128 00:08:50.040 --> 00:08:51.880 slip into your system. s129 00:08:51.880 --> 00:08:55.720 And what I want to get to is basically models have a shortcoming. s130 00:08:55.720 --> 00:09:01.560 Um they can't maintain and improve codebase quality over time, not without a a decent amount of human steering. s131 00:09:01.560 --> 00:09:10.120 Um and when I say maintainability, I'm basically talking about issues like it becomes really, really hard to make a change in one part of the codebase without breaking other parts of the codebase. s132 00:09:10.120 --> 00:09:13.640 This is Martin Fowler's shotgun surgery, textbook code smell. s133 00:09:13.640 --> 00:09:15.360 Um I'm not going to say much more about maintainability. s134 00:09:15.360 --> 00:09:16.880 There's a bunch of books that you can go read about it. s135 00:09:16.880 --> 00:09:23.680 In fact, John Osterhout is actually here speaking this week, so you can go ask him in person about the philosophy of software design if you want to. s136 00:09:23.680 --> 00:09:28.280 Um but it brings us to this question of like why can't models do software maintainability? s137 00:09:28.280 --> 00:09:33.080 Um and you may also be saying, but Dex, you know, surely the models have gotten much better since then. s138 00:09:33.080 --> 00:09:36.880 Um they've gotten better in some ways, but they're still about the same in others. s139 00:09:36.880 --> 00:09:44.280 Um if you want to solve one-off problems or vibe code a new marketing side, yes, they got way better since 2025 and 2024. s140 00:09:44.280 --> 00:09:49.240 But as far as improving code base quality, I think uh they have not gotten much better. s141 00:09:49.240 --> 00:09:57.600 Now, I cannot prove this because there are no good benchmarks for a model's ability to maintain code base quality, and I'll get into like where we're going with that. s142 00:09:57.600 --> 00:10:01.440 Um, but if you've worked with coding agents for a while, a lot of people are posting about this. s143 00:10:01.440 --> 00:10:08.360 It's just like you probably have this vibe that they they generally make things worse over time and make the code base harder to work in. s144 00:10:08.360 --> 00:10:13.040 And to figure out why this happens, I want to zoom out to the first great coding agent. s145 00:10:13.040 --> 00:10:21.680 Why did Claude Code go from nothing to 4 billion and I think now they're at 9 billion in revenue in under a year? s146 00:10:21.680 --> 00:10:23.880 Cuz they were great CLI agents before Claude Code. s147 00:10:23.880 --> 00:10:25.120 You had Aider, you had Code Buff. s148 00:10:25.120 --> 00:10:27.080 There was a bunch of tools in this category. s149 00:10:27.080 --> 00:10:30.160 They had all the same tools, read, write, edit, grep, bash. s150 00:10:30.160 --> 00:10:31.800 Um, so what was the difference? s151 00:10:31.800 --> 00:10:40.720 The difference was was that the this was the first time that a model lab trained a model against the harness that they were going to distribute it to users in. s152 00:10:40.720 --> 00:10:47.680 Um, and it got really, really good at this is just some of the tools, but it got really, really good at calling these sorts of tools in an agentic loop. s153 00:10:47.680 --> 00:10:59.360 In fact, the OpenAI team did a talk in November about basically if you are a uh harness builder and you don't own the model weights and you can't RL the model in your harness, s154 00:10:59.360 --> 00:11:05.160 you will always be at a disadvantage compared to somebody who owns both the model and the harness. s155 00:11:05.160 --> 00:11:11.480 Um, and I'm going to decide a couple slides from my buddy Calvin French Owen, who was a MTS on Codex during the initial launch. s156 00:11:11.480 --> 00:11:13.760 Um, but LLMs are just next token predictors. s157 00:11:13.760 --> 00:11:20.160 Uh, this is a slide from over a year ago where basically as you're doing your agentic loop, context window goes in, next step comes out. s158 00:11:20.160 --> 00:11:21.320 And uh we're going to try to do this. s159 00:11:21.320 --> 00:11:26.360 I haven't actually timed this, but we're going to see if we can do coding agent reinforcement learning in 60 seconds. s160 00:11:26.360 --> 00:11:34.400 So, what we're going to do if we want to train a model to get better at tool calling, better at solving software problems, we're going to generate a bunch of We're going to give it a problem and we're going to generate a bunch of traces. s161 00:11:34.400 --> 00:11:36.280 Try to solve the problem a bunch of different times. s162 00:11:36.280 --> 00:11:40.160 We're going to score them all on correctness and did the test pass and all this stuff. s163 00:11:40.160 --> 00:11:41.280 Uh and then we're going to reinforce. s164 00:11:41.280 --> 00:11:47.280 We're going to make the bad behavior less likely and we're going to update the weights to make the good behavior more likely. s165 00:11:47.280 --> 00:11:50.680 Um this one of the classic ones here is SweetBench multilingual. s166 00:11:50.680 --> 00:11:52.080 Uh they're about 15-minute tasks. s167 00:11:52.080 --> 00:11:55.880 They're from open-source repos like Redis, JQ, and Django and all this stuff. s168 00:11:55.880 --> 00:12:00.480 And they have binary one or zero rewards on did you fix the problem you were trying to fix? s169 00:12:00.480 --> 00:12:02.920 And did you do it without breaking anything else? s170 00:12:02.920 --> 00:12:05.240 Um and we look at actually a real problem from one of these benchmarks. s171 00:12:05.240 --> 00:12:07.600 This is Fastlane, which is a Ruby project. s172 00:12:07.600 --> 00:12:13.440 Um basically, there was some issue where we weren't checking for nil and we have a stack trace blow up because you have a null pointer exception. s173 00:12:13.440 --> 00:12:20.920 And in this um in this benchmark, you have a base commit that we're going to check out before the issue was solved by a human in the past. s174 00:12:20.920 --> 00:12:25.480 We're going to give it a test patch that says here's what the behavior should be afterwards. s175 00:12:25.480 --> 00:12:26.440 We have a golden patch. s176 00:12:26.440 --> 00:12:28.240 Both these are hidden from the model. s177 00:12:28.240 --> 00:12:30.920 Uh and so we have the agent go try to solve the problem. s178 00:12:30.920 --> 00:12:32.160 We store its patch. s179 00:12:32.160 --> 00:12:38.560 We undo all the changes it made to any test files cuz I'm sure you've seen models comment out tests just to get things working. s180 00:12:38.560 --> 00:12:41.920 And then um we're going to apply our golden test patch. s181 00:12:41.920 --> 00:12:43.200 Uh and then we're going to run the test. s182 00:12:43.200 --> 00:12:44.960 Old test and did the new test pass? s183 00:12:44.960 --> 00:12:47.839 And if they both pass, then uh then we get the reward. s184 00:12:47.839 --> 00:12:49.240 Otherwise, we don't. s185 00:12:49.240 --> 00:12:51.520 Um and so models are trying to get the test to pass. s186 00:12:51.520 --> 00:12:57.600 There's no way in this system that we can penalize it for poor program design or for eroding the maintainability of our systems. s187 00:12:57.600 --> 00:12:59.280 That's why we get things like this. s188 00:12:59.280 --> 00:13:02.880 Try catches around things that probably don't need a try catch. s189 00:13:02.880 --> 00:13:04.000 Or things like this. s190 00:13:04.000 --> 00:13:11.000 I think Bybop gave us this example earlier of casting things to other things just so the model can just just it just wants to get the test to pass. s191 00:13:11.000 --> 00:13:16.920 Um and so if you can't verify the uh maintainability of the code, it gets way harder to train on this stuff. s192 00:13:16.920 --> 00:13:18.640 Um so you remember this picture? s193 00:13:18.640 --> 00:13:24.320 Verifying code quality and maintainability is orders of magnitude harder than the code runs and the test pass. s194 00:13:24.320 --> 00:13:28.960 Because the cost function of bad architecture is measured in months and years. s195 00:13:28.960 --> 00:13:39.280 If you have a coding episode and then you only find out months later that like somebody vibed this a little bit too hard, it's really hard to propagate that reward signal back across the gap. s196 00:13:39.280 --> 00:13:41.640 And now the frontier is getting better, slowly. s197 00:13:41.640 --> 00:13:52.080 And since I know someone's going to be in the YouTube comments about this, yes, I know benchmarks and verifiers are different and they actually have to be separate data sets, but they're shaped the same and the the structure of these benchmarks is directionally correct. s198 00:13:52.080 --> 00:13:56.600 So we're going to look at these as like what is the future of evaluating code maintainability. s199 00:13:56.600 --> 00:14:04.400 Um there's a really cool one called Sweep Marathon from Abundant AI where they do like 400-hour tasks of like clone all of Microsoft Excel, every single feature. s200 00:14:04.400 --> 00:14:07.040 Uh and they have some sophisticated reward channel stuff. s201 00:14:07.040 --> 00:14:16.080 Uh Deep Sweep from Data Curve is also like large tasks on OSS repos that are not actually in the training set cuz they were never actually built in the real world. s202 00:14:16.080 --> 00:14:20.280 Uh and then you have Frontier Code from Cognition, um which is multi-PR tasks. s203 00:14:20.280 --> 00:14:25.920 They do interesting things like hey, if the model writes tests that don't fail on the pre-patch code, then it gets penalized. s204 00:14:25.920 --> 00:14:30.760 And then we have a judge model that says, "Okay, uh did this follow all of our code quality rules?" s205 00:14:30.760 --> 00:14:37.720 Um so we're getting better, but I think models judging quality can only go so far uh cuz if the new model if the model knew what good code looks like, s206 00:14:37.720 --> 00:14:40.400 it would probably write it in the first place. s207 00:14:40.400 --> 00:14:48.680 Uh and review agents and throwing more tokens at the problem, it can raise the floor, um but we're still constrained by what we can teach during RL. s208 00:14:48.680 --> 00:14:54.360 Um and so I will I will posit that for now we're stuck reading the code, uh but we can still move pretty fast. s209 00:14:54.360 --> 00:15:02.960 And of course there's a world where this is solved uh in the future, and if you want to just keep yolowing prompts until you get to GPT-7 and you don't have to think about this, by all means, please. s210 00:15:02.960 --> 00:15:05.480 Uh but bitter lesson be damned, we've got some problems to solve. s211 00:15:05.480 --> 00:15:07.880 So let's engineer our way out of this. s212 00:15:07.880 --> 00:15:11.720 Um so turning the lights back on, we're going to put the code review back. s213 00:15:11.720 --> 00:15:18.800 Uh we're going to embrace this approach of like how do we plan up front to reduce the chance that we have a long or difficult review process. s214 00:15:18.800 --> 00:15:19.600 We're going to find leverage. s215 00:15:19.600 --> 00:15:21.520 We're going to use AI to help with this. s216 00:15:21.520 --> 00:15:28.760 Um the first thing we're going to do is we're going to do some sort of product review, understanding what problem we're solving, what's the desired behavior, maybe looking at mock-ups. s217 00:15:28.760 --> 00:15:32.560 Here's a product review I was working on yesterday with a mock-up of a new feature. s218 00:15:32.560 --> 00:15:38.200 Once we have our product review, we're going to By the way, we don't small stuff still just go straight to the agent. s219 00:15:38.200 --> 00:15:41.080 But once we have the product review, we're going to also do architecture, system architecture. s220 00:15:41.080 --> 00:15:46.160 A lot of people have been doing this for a while, component contracts, data models, constraints. s221 00:15:46.160 --> 00:15:52.800 This is an example of a doc that we build to understand how these systems are going to fit together and what's like the high-level picture of it. s222 00:15:52.800 --> 00:15:59.280 From there, we do something that I think is really under-emphasized in agentic coding these days, which is program design. s223 00:15:59.280 --> 00:16:03.120 I think people assume that once you get the architecture right, the model can just cook. s224 00:16:03.120 --> 00:16:09.320 But we we often look into the types and the method signatures, the program layout and the call stacks. s225 00:16:09.320 --> 00:16:10.240 So here's some examples. s226 00:16:10.240 --> 00:16:13.160 I don't think you'll be able to read this one, but this is like the level of abstraction we're at. s227 00:16:13.160 --> 00:16:17.640 It's how we're actually going to lay this stuff out and how these systems are going to interact. s228 00:16:17.640 --> 00:16:22.960 Dylan Mulroy from Cloudflare talks a lot about how he's using these call graphs as part of his planning process. s229 00:16:22.960 --> 00:16:25.080 I think this is exactly right. s230 00:16:25.080 --> 00:16:36.400 And then once we've done the product program design, we can do this thing called vertical slices, which is the order of implementation, multi-repo coordination, how we're going to build this across our entire system, and how are we going to check it along the way. s231 00:16:36.400 --> 00:16:39.280 I've talked a little bit about how models have horizontal plans. s232 00:16:39.280 --> 00:16:40.440 I won't go too deep into it. s233 00:16:40.440 --> 00:16:45.200 If you want to learn more about this, you can go watch our talk from AI Engineer Miami. s234 00:16:45.200 --> 00:16:50.360 Couple shots of a doc like this going through the tests and the steps in between each phase. s235 00:16:50.360 --> 00:16:56.720 The main idea here is 30 minutes over here in pre-planning and alignment can save you hours in review. s236 00:16:56.720 --> 00:17:01.280 And so it's actually feasible to still read every line of code. s237 00:17:01.280 --> 00:17:03.080 We'll skip this part. s238 00:17:03.080 --> 00:17:07.520 Basically, the the summary here is like you don't have too many PRs. s239 00:17:07.520 --> 00:17:12.360 If you're drowning in PRs, you actually have too many bad PRs. s240 00:17:12.360 --> 00:17:15.079 Because a good PR is a joy to to review. s241 00:17:15.079 --> 00:17:17.319 It's it's you're just reading through it like, "Yep, this is great. s242 00:17:17.319 --> 00:17:18.240 This is what we discussed. s243 00:17:18.240 --> 00:17:20.000 This is what we talked about." s244 00:17:20.000 --> 00:17:32.520 Um but even if a PR needs 20% rework, which is generous for a lot of AI AI vibe coded slop, um it's an It's an emotional and intellectual burden on both the reviewer and the submitter. s245 00:17:32.520 --> 00:17:39.000 Um and so if you use model assistant planning and alignment, your alignment is shorter cuz you used AI to get all the information at once. s246 00:17:39.000 --> 00:17:44.040 Your code review is faster because you aligned up front, and your coding is faster cuz AI did it. s247 00:17:44.040 --> 00:17:49.000 And so you're now you're actually really moving faster, but you're still reading everything and you're still owning the code. s248 00:17:49.000 --> 00:17:54.440 So, closing advice, um is easy to hear all this and be a little bummed out. s249 00:17:54.440 --> 00:18:00.960 Uh I really like the world where we just YOLO everything, and we can just like not have to ever read code ever again. s250 00:18:00.960 --> 00:18:06.760 But uh we're engineers, and these are just constraints, and models are good at certain things, and they're not good at other things. s251 00:18:06.760 --> 00:18:11.040 And so go figure out how to solve problems given a set of constraints. s252 00:18:11.040 --> 00:18:12.560 Uh use loops, they're great. s253 00:18:12.560 --> 00:18:14.120 Go solve hard problems. s254 00:18:14.120 --> 00:18:15.240 Seek leverage. s255 00:18:15.240 --> 00:18:17.800 Um if you want to help with this, um we're building Human Layer. s256 00:18:17.800 --> 00:18:20.880 Human Layer is an AI IDE and collaboration platform. s257 00:18:20.880 --> 00:18:27.520 It's building blocks for your software factory, um and soon to be better vi- verifiers for software quality. s258 00:18:27.520 --> 00:18:31.760 Um we've got sort of a Figma for cloud code and Codex-style collaborative workspace. s259 00:18:31.760 --> 00:18:35.760 It walks you through the workflows for doing this sort of work. s260 00:18:35.760 --> 00:18:38.520 And uh we are talking to design partners. s261 00:18:38.520 --> 00:18:41.840 We are hiring founding engineers here in San Francisco. s262 00:18:41.840 --> 00:18:43.640 And uh these slides are live. s263 00:18:43.640 --> 00:18:45.080 You can go get them right now. s264 00:18:45.080 --> 00:18:47.920 You can try Human Layer at humanlayer.com. s265 00:18:47.920 --> 00:18:49.480 Uh it's free for small teams. s266 00:18:49.480 --> 00:18:52.440 Go solve hard problems in complex codebases. s267 00:18:52.440 --> 00:18:54.473 Thank you all for your energy. s268 00:18:54.473 --> 00:18:56.473 [applause] s269 00:19:12.813 --> 00:19:14.813 [music] s270 00:19:16.960 --> 00:19:18.960 Hey.