WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=AnsXWH2_EBU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/AnsXWH2_EBU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.760 --> 00:00:17.680 I'm pretty excited to be here and talk about who verifies the software factory. s3 00:00:17.680 --> 00:00:22.240 Uh before we get started, uh can you give me a quick show of hands? s4 00:00:22.240 --> 00:00:28.800 Uh raise your hand if your coding agent has pulled uh opened any pull request in the last 1 week. s5 00:00:28.800 --> 00:00:30.480 Almost everyone, thanks. s6 00:00:30.480 --> 00:00:39.200 And keep your hands raised up if you have merged pull requests and reviewed every line of code that was merged. s7 00:00:39.400 --> 00:00:40.760 Oh, I do see a few people. s8 00:00:40.760 --> 00:00:42.080 That's excellent. s9 00:00:42.080 --> 00:00:54.800 So, what I'm going to talk about is the gap between the uh two uh camps, uh merging requests without reviewing manually, and uh you know, merging requests uh after doing some reviews. s10 00:00:54.800 --> 00:01:06.040 So, my name is Manish Kapoor, and I lead the product and solution uh developer relations uh team at Sonar, and I'm excited to be here uh with my guest s11 00:01:06.040 --> 00:01:10.080 speaker, Addy Osmani from Anthropic. s12 00:01:10.080 --> 00:01:13.800 Uh Addy has been uh had been at Google for about 14 years. s13 00:01:13.800 --> 00:01:15.560 Most of you probably know him. s14 00:01:15.560 --> 00:01:18.640 And he's written lots and lots of content and articles. s15 00:01:18.640 --> 00:01:25.920 Uh I learn a lot from Addy, so I'm uh really fortunate to have him here, stand next to me, and uh share his wisdom with all of us here today. s16 00:01:25.920 --> 00:01:33.560 Uh he recently joined Anthropic, and Addy, so congratulations on your uh role at uh Anthropic. s17 00:01:33.560 --> 00:01:37.000 Would you like to tell the team what you're working on at Anthropic, what your role is? s18 00:01:37.000 --> 00:01:37.560 Yeah, sure. s19 00:01:37.560 --> 00:01:47.200 So, I'm responsible for making sure that you all have a really great developer experience building with Cloud Code, and anytime that we discover there are opportunities to make it even smoother, s20 00:01:47.200 --> 00:01:48.840 we bring that right back into the product. s21 00:01:48.840 --> 00:01:56.080 We ship a couple of times every single day, and so we're very interested in trying to make sure that you have the best possible experience as often as possible. s22 00:01:56.080 --> 00:01:56.400 Cool. s23 00:01:56.400 --> 00:01:57.680 Thank you. s24 00:01:57.680 --> 00:01:58.240 All right. s25 00:01:58.240 --> 00:02:02.452 Uh so, let's start with some problem and context setting slide. s26 00:02:02.452 --> 00:02:04.652 There are two numbers on the slide. s27 00:02:04.652 --> 00:02:09.692 CircleCI analyzed 28 million CI workflows this year. s28 00:02:09.692 --> 00:02:13.532 And they noticed that the throughput jumped by about 59%. s29 00:02:13.532 --> 00:02:19.052 But yet three in 10 attempts to merge that code into production are failing. s30 00:02:19.052 --> 00:02:23.332 And this is a stat from one independent stat from CircleCI. s31 00:02:23.332 --> 00:02:29.892 Uh And I think I'm seeing the same trend from most of our customers that I speak to every day. s32 00:02:29.892 --> 00:02:32.332 Uh this is a quote from Adi. s33 00:02:32.332 --> 00:02:35.812 A software factory is harnessing loops at scale. s34 00:02:35.812 --> 00:02:37.772 This resonated quite a bit for me. s35 00:02:37.772 --> 00:02:39.852 And if you have not read that article, you can look it up. s36 00:02:39.852 --> 00:02:48.412 It's pretty educational article and talking about future how the future will look like in terms of soft software factories. s37 00:02:48.412 --> 00:02:55.252 Uh so, Adi, would you like to expand on what you were thinking when you wrote this article about soft loops and harnessing the loops at scale? s38 00:02:55.252 --> 00:02:56.052 Yeah, absolutely. s39 00:02:56.052 --> 00:03:02.732 So, a lot of folks here probably heard this term software factories, and there's two types of factories that are at least worth keeping in mind. s40 00:03:02.732 --> 00:03:05.692 Well, you know, software factory is harnessing loops at scale. s41 00:03:05.692 --> 00:03:19.412 A lights-on factory is one where you have agents that are helping you with planning, building, checking, and shipping code again and again, but you as a person still apply your judgment and taste inside the picture. s42 00:03:19.412 --> 00:03:28.252 And then we have a dark software factory, which is a little bit closer to how, you know, a car factory with robotics might be working where it's constantly doing the work. s43 00:03:28.252 --> 00:03:38.932 So, it's scoping, building, checking, and you as the person move down further to thinking about intent, and you're pulled in when they're really the greatest of exceptions to the case. s44 00:03:38.932 --> 00:03:40.612 But, that's sort of what I was thinking. s45 00:03:40.612 --> 00:03:49.972 I think a lot of teams are somewhere on this spectrum and are on this journey to figure out like how how do I get further and further into automation in a way that I can trust it. s46 00:03:49.972 --> 00:03:51.572 Cool. s47 00:03:51.572 --> 00:03:55.572 So, uh AI coding is now basically a solved problem. s48 00:03:55.572 --> 00:03:57.212 We all know agents are there. s49 00:03:57.212 --> 00:04:00.052 85% of the organizations are already using coding agents. s50 00:04:00.052 --> 00:04:03.334 It will go to 95% and very soon to 100%. s51 00:04:03.334 --> 00:04:14.534 Uh 75% of the uh 76% of the developers that generate AI code, uh they are saying that it it's being code is being produced at a speed that is impossible for them to verify it. s52 00:04:14.534 --> 00:04:17.374 So, they are spending more and more time reviewing the code. s53 00:04:17.374 --> 00:04:23.214 So, what do you think is the most pressing question that engineering leaders have right now? s54 00:04:23.214 --> 00:04:27.614 So, I think that teams have gotten better working with agents over the last year. s55 00:04:27.614 --> 00:04:30.454 Um models, harnesses have also gotten better. s56 00:04:30.454 --> 00:04:38.214 And what I see a lot of leaders being interested in is how do we enable you to ship code faster um with confidence. s57 00:04:38.214 --> 00:04:40.574 And that confidence piece is very important. s58 00:04:40.574 --> 00:04:45.814 Agents can now generate ways for you to check your code much more cheaply than ever before. s59 00:04:45.814 --> 00:04:58.094 And if you think about the things that enabled us to call good code, you know, high quality, uh those were things like good tests, benchmarks, uh specs, anything that gives us confidence. s60 00:04:58.094 --> 00:05:02.494 So, agents are able to help us generate that a little bit more easily these days, a little bit more quickly. s61 00:05:02.494 --> 00:05:09.734 We can of course refine those things, but I think that those checks are one of the most important things we should be spending our time on these days. s62 00:05:09.734 --> 00:05:18.774 Like if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct. s63 00:05:18.774 --> 00:05:27.694 If you've got that in place and it you can make sure that the code is both correct and complete, that can easily two to three by like the quality of your outcomes. s64 00:05:27.694 --> 00:05:28.054 Yep. s65 00:05:28.054 --> 00:05:28.294 Yep. s66 00:05:28.294 --> 00:05:28.974 Absolutely right. s67 00:05:28.974 --> 00:05:29.494 Spot on. s68 00:05:29.494 --> 00:05:34.334 I think code confidence uh we have been running an ad code confidence is scarce. s69 00:05:34.334 --> 00:05:38.054 And you really need to build it up now that the coding problem is solved. s70 00:05:38.054 --> 00:05:42.174 So, I actually I yesterday night, so this might be a surprise question for you. s71 00:05:42.174 --> 00:05:47.654 Uh yesterday night I read a post uh from Anthropic, and I encourage all of you to read it. s72 00:05:47.654 --> 00:05:50.694 It talks about the scale at which Anthropic is operating. s73 00:05:50.694 --> 00:05:54.894 Uh they are merging about 30,000 code changes every single day. s74 00:05:54.894 --> 00:06:00.294 They used to do about 3,000 or so, 2,500 or so code changes just a few months back. s75 00:06:00.294 --> 00:06:05.655 They have scaled from 2,500 code changes to 3,000 uh 30,000 code changes per day. s76 00:06:05.655 --> 00:06:06.975 That is mind-boggling. s77 00:06:06.975 --> 00:06:13.375 So, I didn't What does code review look like at uh Claude Code at Anthropic and with the Claude Code team that you're working with? s78 00:06:13.375 --> 00:06:14.335 Yeah, these are great questions. s79 00:06:14.335 --> 00:06:19.215 So, we were we were merging about 2,500 changes all the way back in May. s80 00:06:19.215 --> 00:06:26.055 Um and even for our recent refresh of Claude.ai, we put out a case study saying that we kind of three-by'd the performance of it. s81 00:06:26.055 --> 00:06:35.215 That was accomplished by merging about 3,000 pull requests in a very short space of time with engineers, product designers, a number of different functions contributing to it. s82 00:06:35.215 --> 00:06:44.175 So, our workflow for being able to get verification and code review in place is that we first of all have Claude do a first-pass review. s83 00:06:44.175 --> 00:06:50.655 What's very important in there is making sure that it has a way to actually try the experience out similar to a real user. s84 00:06:50.655 --> 00:07:01.175 So, that can mean end-to-end tasks, you know, opening up the experience, checking it in a browser, running any tools, doing any linting, doing any other checks that are necessary to build up that confidence. s85 00:07:01.175 --> 00:07:07.615 Um people usually ask, "Is it okay, well, what happens if, you know, you discover a bug or a gap in functionality?" s86 00:07:07.615 --> 00:07:11.375 Well, we have hooks that will then go and run on any edits. s87 00:07:11.375 --> 00:07:18.095 They will study the diffs and allow us to get this loop in place where we're correcting the PR as quickly as possible. s88 00:07:18.095 --> 00:07:21.135 So, we have uh Claude Code doing the review. s89 00:07:21.135 --> 00:07:24.535 We have a um code review product uh that you can also check out. s90 00:07:24.535 --> 00:07:25.135 It's pretty good. s91 00:07:25.135 --> 00:07:29.055 And the idea there is we're trying to automate code review um as much as possible. s92 00:07:29.055 --> 00:07:31.135 This is something that we use ourselves. s93 00:07:31.135 --> 00:07:40.095 Um and increasingly what's important is for your teams to be thinking about how can you cut We talk about taste and judgment so much these days, right? s94 00:07:40.095 --> 00:07:51.895 It's important for you to be trying to codify what you consider best practice and taste in your team's shared like Claude.md, agents.md, anywhere where the agent can then consult during code review. s95 00:07:51.895 --> 00:07:55.695 And then people are responsible for checking on the agent's evidence. s96 00:07:55.695 --> 00:07:56.655 So, how do we know? s97 00:07:56.655 --> 00:08:10.337 Some If I'm building out um a piece of UI that's going to be, you know, used by a user in the browser, I want to make sure that it's tested in the browser, maybe generated me a video of that experience where it's been clicking around and gives me confidence. s98 00:08:10.337 --> 00:08:11.857 Well, okay, well, this works. s99 00:08:11.857 --> 00:08:16.417 If I care about the mobile or desktop experience, maybe it generates me screenshots or videos of that. s100 00:08:16.417 --> 00:08:21.257 But basically, you want to get to a place where the agent's giving you as much verification evidence as possible. s101 00:08:21.257 --> 00:08:21.777 Cool. s102 00:08:21.777 --> 00:08:23.137 Yeah, verification is key. s103 00:08:23.137 --> 00:08:29.337 I think you need to verify everything that the software factories will produce and that's what we are going to talk about today. s104 00:08:29.337 --> 00:08:37.057 Uh so Uh another these stats by the way are from uh the state of code survey report that will be coming out very soon from Sonar. s105 00:08:37.057 --> 00:08:46.337 Uh we surveyed about a thousand more than a thousand developers and we found some interesting data which will give us insights as to what's happening in the industry today and where we are headed. s106 00:08:46.337 --> 00:08:52.777 So, 84% say AI features uh uh gets gets features and fixes put to production faster. s107 00:08:52.777 --> 00:08:55.777 The pull requests are getting larger and more complex. s108 00:08:55.777 --> 00:09:02.417 Uh and you know, yet uh only uh 31% more code is reaching reaching the production without review. s109 00:09:02.417 --> 00:09:14.217 So, that's a little little bit of a scary fact particularly for uh production uh applications in the regulated industries in the mission-critical applications where you absolutely make need to make sure s110 00:09:14.217 --> 00:09:20.057 uh that you know, uh you have to verify that code and you're confident in the code that is being shipped. s111 00:09:20.057 --> 00:09:23.817 So, uh what are the where are the next gains going to come from, Addy? s112 00:09:23.817 --> 00:09:29.017 I think the next gains are going to come from a combination of us really nailing verification, right? s113 00:09:29.017 --> 00:09:30.657 As well as model quality. s114 00:09:30.657 --> 00:09:36.897 But models are already like if you you take a look at Opus 55, model quality has already gotten to a place where it's pretty good. s115 00:09:36.897 --> 00:09:47.377 Um and I would say that us being able to share best practices about that verification step is going to become increasingly more important because if you tell people, yeah, just like verify the work, you know, their next question is, what does that even mean? s116 00:09:47.377 --> 00:09:48.097 Yeah. s117 00:09:48.097 --> 00:09:53.417 And so, I think that the conversation needs to evolve into specifically what kinds of tests are useful. s118 00:09:53.417 --> 00:10:02.297 We lean a lot on end-to-end tests, property tests, any visual verification tests, things that can help us just get that confidence to merge those changes in automatically. s119 00:10:02.297 --> 00:10:03.057 Cool. s120 00:10:03.057 --> 00:10:04.737 You mentioned light factories. s121 00:10:04.737 --> 00:10:07.777 Want to go down that and that's the way uh direction we are headed. s122 00:10:07.777 --> 00:10:13.538 Uh our survey results show that, you know, today 5% of the companies are using uh software factories. s123 00:10:13.538 --> 00:10:17.898 It will get to 34% and surprisingly 39% in the financial services. s124 00:10:17.898 --> 00:10:19.578 How many of you are using software factories today? s125 00:10:19.578 --> 00:10:21.818 Just curious. s126 00:10:22.458 --> 00:10:23.538 A handful of you. s127 00:10:23.538 --> 00:10:28.058 And how many are planning to use software factories by next year? s128 00:10:29.098 --> 00:10:30.218 Yeah, that's a good question. s129 00:10:30.218 --> 00:10:31.818 How do you define a software factory? s130 00:10:31.818 --> 00:10:32.938 Would you like to take that question? s131 00:10:32.938 --> 00:10:33.418 Yeah, yeah, sure. s132 00:10:33.418 --> 00:10:37.298 So, we we talked about definition of software factories just a little bit earlier. s133 00:10:37.298 --> 00:10:39.538 A software factory is basically loops at scale. s134 00:10:39.538 --> 00:10:47.538 And when we think about this, it's all about like what end of the spectrum are you at in terms of how much people are still involved in the process. s135 00:10:47.538 --> 00:10:53.938 The more that the agents are able to generate verification and give you confidence, the more that you can get to that place where you go dark. s136 00:10:53.938 --> 00:11:00.298 And I think that yeah, most people are still somewhere in that spectrum and are not quite as this data shows at that dark point just yet. s137 00:11:00.298 --> 00:11:00.778 Yep. s138 00:11:00.778 --> 00:11:09.138 Just to add to what Addy said, I think software factory, think of it like a production line where you start defining the intent and your specifications. s139 00:11:09.138 --> 00:11:10.778 Then you start writing the code. s140 00:11:10.778 --> 00:11:15.618 So, it's an assembly line where each of these software development life cycle steps happen. s141 00:11:15.618 --> 00:11:22.618 And automating them at scale where there's pretty much in a dark factory, there would be no lights on. s142 00:11:22.618 --> 00:11:25.458 It's autonomously working, fully autonomously working. s143 00:11:25.458 --> 00:11:32.338 A light factory has uh is where people are involved in the process at every step or most steps. s144 00:11:32.338 --> 00:11:38.098 Uh whereas a dark factory, the goal would be to turn off the lights and let the factory run independently 24/7. s145 00:11:38.098 --> 00:11:39.218 Uh all right. s146 00:11:39.218 --> 00:11:41.618 So, you talked about verification. s147 00:11:41.618 --> 00:11:46.338 Uh I'll talk a little bit about what we think at Sonar about verification. s148 00:11:46.338 --> 00:11:52.018 If you saw our keynote earlier today from our CEO, he talked about verification should be multi-layered. s149 00:11:52.018 --> 00:11:58.778 Uh we absolutely 100% believe that each company should adopt a zero trust multi-layered verification approach. s150 00:11:58.778 --> 00:12:01.818 And verification should be done in every loop. s151 00:12:01.818 --> 00:12:09.738 Whether you're operating inside the agentic loop or whether you're working inside the CI verification loop or whether you're doing code maintenance. s152 00:12:09.738 --> 00:12:17.059 In each and every loop, it is important to verify the output of the factories or the agents that code that the agents are producing. s153 00:12:17.059 --> 00:12:28.819 So, we look at it in terms of layers and think of it like, you know, the there's first layer which is a deterministic layer which catches uh bugs, security issues, s154 00:12:28.819 --> 00:12:33.459 things like SQL injection, buffer overflow bugs, and all of those are caught. s155 00:12:33.459 --> 00:12:41.579 And then there's a second layer where agentic review happens where agents are used to like Adi was talking about, agents review review that code. s156 00:12:41.579 --> 00:12:49.419 And in that agentic review, they catch logic and intent issues which the deterministic layer is not well suited to catch. s157 00:12:49.419 --> 00:12:54.179 So, we we use each layer is targeted at catching a specific types of issues. s158 00:12:54.179 --> 00:13:05.099 And then the third layer could be a software supply chain security layer where you look at all the CVEs, all your malware detection happens, all your malicious package detection happens. s159 00:13:05.099 --> 00:13:06.739 All of that happens in the third layer. s160 00:13:06.739 --> 00:13:12.979 So, it's a layered approach where each layer is tuned and optimized to catch a certain type of issue. s161 00:13:12.979 --> 00:13:16.859 So, we absolutely recommend doing a multi-layered verification. s162 00:13:16.859 --> 00:13:18.859 Oops. s163 00:13:19.579 --> 00:13:21.619 So, there's another stat here. s164 00:13:21.619 --> 00:13:29.699 28% you know, companies have approved agents and shared controls and very few shared controls. s165 00:13:29.699 --> 00:13:33.299 And 5% are operating software factories autonomously. s166 00:13:33.299 --> 00:13:34.699 What are your thoughts there, Adi? s167 00:13:34.699 --> 00:13:37.299 Uh in terms of s168 00:13:37.299 --> 00:13:46.899 So, I've talked to so many companies where a couple of months ago one of their biggest worries was they were allowing their engineering teams to go off and play around with agents and they got to a place where s169 00:13:46.899 --> 00:13:51.379 there were hundreds, thousands of agents in some cases and they just didn't know what they were doing. s170 00:13:51.379 --> 00:13:52.699 Like, what are these agents actually doing? s171 00:13:52.699 --> 00:13:57.379 They're touching all of our systems, accessing all kinds of data, maybe they're deploying to production. s172 00:13:57.379 --> 00:14:05.819 And so, a lot of enterprises think about concerns like, you know, agent identity and telemetry and being able to track what's actually happening. s173 00:14:05.819 --> 00:14:06.019 Yeah. s174 00:14:06.019 --> 00:14:13.019 And I'd say that the three biggest things for a lot of companies are how do we think about control, how do we think about permissions, and how do we think about the sandbox? s175 00:14:13.019 --> 00:14:17.141 Like, how are we making sure that the agent isn't jumping out of the sandbox and doing things that it shouldn't be doing? s176 00:14:17.141 --> 00:14:17.661 Yeah. s177 00:14:17.661 --> 00:14:20.541 Yeah, we we at sonar we call it guide rails. s178 00:14:20.541 --> 00:14:27.181 So, instead of guard rails, we're thinking of calling it as guide rails, where we guide the agents to follow the real tracks, right? s179 00:14:27.181 --> 00:14:34.741 A specific path to take and prevent them from going out of bounds or so to say going out of the sandbox like Addie was saying. s180 00:14:34.741 --> 00:14:40.381 So, it's very very important to have guide rails for the agent and govern the agents that you're working with. s181 00:14:40.381 --> 00:14:47.221 Uh So, let's uh we're coming to a close now as my timer is telling me we just have a few minutes left. s182 00:14:47.221 --> 00:14:49.381 Let's go get into a lightning round. s183 00:14:49.381 --> 00:14:51.941 Uh one sentence response for each of these questions. s184 00:14:51.941 --> 00:14:53.941 And you've seen the questions already on the slide, so s185 00:14:53.941 --> 00:14:55.541 Can these be long sentences? s186 00:14:55.541 --> 00:14:58.581 They They can be as long as the time permits. s187 00:14:58.581 --> 00:14:59.261 Perfect. s188 00:14:59.261 --> 00:15:03.581 Uh light or dark factory for most teams by end of 2027, yes or no? s189 00:15:03.581 --> 00:15:06.741 I think we're going to see uh light factories with some dark corners. s190 00:15:06.741 --> 00:15:12.661 I don't think we're going to get to everybody in this room using dark factories entirely, but it's going to be something close to that. s191 00:15:12.661 --> 00:15:17.461 I I would guess Anthropic would do it this year or next year. s192 00:15:18.320 --> 00:15:20.320 [laughter] s193 00:15:20.501 --> 00:15:21.261 All right. s194 00:15:21.261 --> 00:15:22.701 Second question. s195 00:15:22.701 --> 00:15:26.021 Should agent-written code be labeled in the PR? s196 00:15:26.021 --> 00:15:27.381 I personally think it should. s197 00:15:27.381 --> 00:15:33.901 Historically, we have felt a great deal of ownership around our code, pride around the fact that, "Hey, I wrote this thing. s198 00:15:33.901 --> 00:15:35.501 I'm putting my name on it." s199 00:15:35.501 --> 00:15:38.581 And at Anthropic, like we we think about this a little bit differently. s200 00:15:38.581 --> 00:15:45.341 We've evolved to the point where we don't really mind so much if Claude is like labeled as, "Hey, I'm I'm the one that wrote this thing." s201 00:15:45.341 --> 00:15:52.661 because we're a little bit more focused on the outcomes and making sure that the code, the features, like what we're shipping to users is the right thing. s202 00:15:52.661 --> 00:15:58.101 And so, I personally think it's fine for the code to be written labeled as being coming from an agent. s203 00:15:58.101 --> 00:16:04.021 And that can also help us with things like figuring out, "Okay, well, if the agent wrote it, what are some of the constraints and extra checks that we need to keep in place?" s204 00:16:04.021 --> 00:16:05.981 I I have the same point of view, Addie. s205 00:16:05.981 --> 00:16:14.821 I think in the short run, at least in the foreseeable you know, 6 to 9 months, we should label the PRs till we are confident enough in our factories. s206 00:16:14.821 --> 00:16:20.342 We do want to know and track it, and it's also important for the financial customers here in terms of auditability. s207 00:16:20.342 --> 00:16:26.302 If you if you know, if you label it, you know who's writing the code because we are at the end of the day still accountable for s208 00:16:26.302 --> 00:16:26.862 Yeah, absolutely. s209 00:16:26.862 --> 00:16:29.022 Accountability is the most important part here. s210 00:16:29.022 --> 00:16:32.022 Like we still label like who is the owner for this change. s211 00:16:32.022 --> 00:16:36.942 I might have kicked off a cloud cloud run to complete a piece of work, but I'm I'm still on the hook. s212 00:16:36.942 --> 00:16:38.982 If something goes wrong, I have to go and fix it. s213 00:16:38.982 --> 00:16:39.582 Yep, yep. s214 00:16:39.582 --> 00:16:40.102 Key point. s215 00:16:40.102 --> 00:16:40.782 Yep. s216 00:16:40.782 --> 00:16:45.462 Uh the next question, actually I'll improvise it here because this is a leadership session. s217 00:16:45.462 --> 00:16:51.182 What one skill a junior developer or leader should build on this year? s218 00:16:51.182 --> 00:16:57.382 I think that you should push your teams, if you're a leader, you should push your teams to really think hard about what verification means. s219 00:16:57.382 --> 00:17:00.342 Um how are you already approaching things like testing? s220 00:17:00.342 --> 00:17:08.902 What is the gap between that and you getting to a place where you're very comfortable with the agents being able to sign off on the code being good enough to merge on their own. s221 00:17:08.902 --> 00:17:12.102 And so figuring that out, if you're a junior, it's the other side of that. s222 00:17:12.102 --> 00:17:18.502 Like actually getting more experience in the different flavors of tests, whether it's like end-to-end tests, property tests, etc. s223 00:17:18.502 --> 00:17:19.142 Yep. s224 00:17:19.142 --> 00:17:24.222 And the last uh you you actually not following the rules, you're going beyond one sentence, but that's cool. s225 00:17:24.222 --> 00:17:27.022 Uh The last one s226 00:17:27.022 --> 00:17:28.022 30 seconds left. s227 00:17:28.022 --> 00:17:28.422 I'm fine. s228 00:17:28.422 --> 00:17:30.422 We can go a minute over, I'm sure. s229 00:17:30.422 --> 00:17:32.542 Uh the agent opened the PR. s230 00:17:32.542 --> 00:17:35.182 It is safe to merge when? s231 00:17:35.182 --> 00:17:39.542 You have full confidence nothing's going to go wrong or that you have mitigated that as much as possible. s232 00:17:39.542 --> 00:17:40.702 Love it. s233 00:17:40.702 --> 00:17:41.582 All right. s234 00:17:41.582 --> 00:17:44.182 Uh here's my last slide and then we'll wrap it up. s235 00:17:44.182 --> 00:17:50.062 Uh so what separates uh teams that are shipping uh fast but not shipping blind? s236 00:17:50.062 --> 00:17:52.422 I think that three key takeaways for you. s237 00:17:52.422 --> 00:17:55.222 Uh writing code is the fastest part and that is solved. s238 00:17:55.222 --> 00:18:00.102 Strong review and verification for our teams is what will will help you maintain that speed. s239 00:18:00.102 --> 00:18:03.182 Fast to write is not fast to ship. s240 00:18:03.182 --> 00:18:11.422 Uh you have to basically count what is getting merged, looked at what is getting merged, and make sure it doesn't escape the sandbox and the things like that before it gets into the production. s241 00:18:11.422 --> 00:18:12.782 Verify everything. s242 00:18:12.782 --> 00:18:14.622 And autonomy is earned. s243 00:18:14.622 --> 00:18:18.782 Uh you We will get to software factories very soon if not next year. s244 00:18:18.782 --> 00:18:24.583 Uh you need to put in independent automated verification inside each loop of your factory. s245 00:18:24.583 --> 00:18:28.143 Make sure that you can trust the output that the factory is producing. s246 00:18:28.143 --> 00:18:30.583 Uh, with that, I'll wrap it up. s247 00:18:30.583 --> 00:18:34.263 Thanks for attending our talk today, uh, a conversation with Adi. s248 00:18:34.263 --> 00:18:38.663 Uh, again, Adi, I appreciate you taking the time and joining us here in this conversation. s249 00:18:38.663 --> 00:18:45.223 Uh, visit us at the big red booth, uh, the sonar booth in the expo hall if you have any further questions or would like to, uh, talk about it. s250 00:18:45.223 --> 00:18:47.503 Thank you again.