How Anthropic Merges 30,000 Code Changes a Day - Manish Kapur, Sonar & Addy Osmani, Anthropic

AI Engineer · 18 min · 250 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12I'm pretty excited to be here and talk about who verifies the software factory.
00:17Uh before we get started, uh can you give me a quick show of hands?
00:22Uh raise your hand if your coding agent has pulled uh opened any pull request in the last 1 week.
00:28Almost everyone, thanks.
00:30And keep your hands raised up if you have merged pull requests and reviewed every line of code that was merged.
00:39Oh, I do see a few people.
00:40That's excellent.
00:42So, what I'm going to talk about is the gap between the uh two uh camps, uh merging requests without reviewing manually, and uh you know, merging requests uh after doing some reviews.
00:54So, my name is Manish Kapoor, and I lead the product and solution uh developer relations uh team at Sonar, and I'm excited to be here uh with my guest
01:06speaker, Addy Osmani from Anthropic.
01:10Uh Addy has been uh had been at Google for about 14 years.
01:13Most of you probably know him.
01:15And he's written lots and lots of content and articles.
01:18Uh I learn a lot from Addy, so I'm uh really fortunate to have him here, stand next to me, and uh share his wisdom with all of us here today.
01:25Uh he recently joined Anthropic, and Addy, so congratulations on your uh role at uh Anthropic.
01:33Would you like to tell the team what you're working on at Anthropic, what your role is?
01:37Yeah, sure.
01:37So, I'm responsible for making sure that you all have a really great developer experience building with Cloud Code, and anytime that we discover there are opportunities to make it even smoother,
01:47we bring that right back into the product.
01:48We ship a couple of times every single day, and so we're very interested in trying to make sure that you have the best possible experience as often as possible.
01:56Cool.
01:56Thank you.
01:57All right.
01:58Uh so, let's start with some problem and context setting slide.
02:02There are two numbers on the slide.
02:04CircleCI analyzed 28 million CI workflows this year.
02:09And they noticed that the throughput jumped by about 59%.
02:13But yet three in 10 attempts to merge that code into production are failing.
02:19And this is a stat from one independent stat from CircleCI.
02:23Uh And I think I'm seeing the same trend from most of our customers that I speak to every day.
02:29Uh this is a quote from Adi.
02:32A software factory is harnessing loops at scale.
02:35This resonated quite a bit for me.
02:37And if you have not read that article, you can look it up.
02:39It's pretty educational article and talking about future how the future will look like in terms of soft software factories.
02:48Uh so, Adi, would you like to expand on what you were thinking when you wrote this article about soft loops and harnessing the loops at scale?
02:55Yeah, absolutely.
02:56So, a lot of folks here probably heard this term software factories, and there's two types of factories that are at least worth keeping in mind.
03:02Well, you know, software factory is harnessing loops at scale.
03:05A lights-on factory is one where you have agents that are helping you with planning, building, checking, and shipping code again and again, but you as a person still apply your judgment and taste inside the picture.
03:19And then we have a dark software factory, which is a little bit closer to how, you know, a car factory with robotics might be working where it's constantly doing the work.
03:28So, it's scoping, building, checking, and you as the person move down further to thinking about intent, and you're pulled in when they're really the greatest of exceptions to the case.
03:38But, that's sort of what I was thinking.
03:40I think a lot of teams are somewhere on this spectrum and are on this journey to figure out like how how do I get further and further into automation in a way that I can trust it.
03:49Cool.
03:51So, uh AI coding is now basically a solved problem.
03:55We all know agents are there.
03:5785% of the organizations are already using coding agents.
04:00It will go to 95% and very soon to 100%.
04:03Uh 75% of the uh 76% of the developers that generate AI code, uh they are saying that it it's being code is being produced at a speed that is impossible for them to verify it.
04:14So, they are spending more and more time reviewing the code.
04:17So, what do you think is the most pressing question that engineering leaders have right now?
04:23So, I think that teams have gotten better working with agents over the last year.
04:27Um models, harnesses have also gotten better.
04:30And what I see a lot of leaders being interested in is how do we enable you to ship code faster um with confidence.
04:38And that confidence piece is very important.
04:40Agents can now generate ways for you to check your code much more cheaply than ever before.
04:45And if you think about the things that enabled us to call good code, you know, high quality, uh those were things like good tests, benchmarks, uh specs, anything that gives us confidence.
04:58So, agents are able to help us generate that a little bit more easily these days, a little bit more quickly.
05:02We can of course refine those things, but I think that those checks are one of the most important things we should be spending our time on these days.
05:09Like if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct.
05:18If you've got that in place and it you can make sure that the code is both correct and complete, that can easily two to three by like the quality of your outcomes.
05:27Yep.
05:28Yep.
05:28Absolutely right.
05:28Spot on.
05:29I think code confidence uh we have been running an ad code confidence is scarce.
05:34And you really need to build it up now that the coding problem is solved.
05:38So, I actually I yesterday night, so this might be a surprise question for you.
05:42Uh yesterday night I read a post uh from Anthropic, and I encourage all of you to read it.
05:47It talks about the scale at which Anthropic is operating.
05:50Uh they are merging about 30,000 code changes every single day.
05:54They used to do about 3,000 or so, 2,500 or so code changes just a few months back.
06:00They have scaled from 2,500 code changes to 3,000 uh 30,000 code changes per day.
06:05That is mind-boggling.
06:06So, I didn't What does code review look like at uh Claude Code at Anthropic and with the Claude Code team that you're working with?
06:13Yeah, these are great questions.
06:14So, we were we were merging about 2,500 changes all the way back in May.
06:19Um and even for our recent refresh of Claude.ai, we put out a case study saying that we kind of three-by'd the performance of it.
06:26That was accomplished by merging about 3,000 pull requests in a very short space of time with engineers, product designers, a number of different functions contributing to it.
06:35So, our workflow for being able to get verification and code review in place is that we first of all have Claude do a first-pass review.
06:44What's very important in there is making sure that it has a way to actually try the experience out similar to a real user.
06:50So, that can mean end-to-end tasks, you know, opening up the experience, checking it in a browser, running any tools, doing any linting, doing any other checks that are necessary to build up that confidence.
07:01Um people usually ask, "Is it okay, well, what happens if, you know, you discover a bug or a gap in functionality?"
07:07Well, we have hooks that will then go and run on any edits.
07:11They will study the diffs and allow us to get this loop in place where we're correcting the PR as quickly as possible.
07:18So, we have uh Claude Code doing the review.
07:21We have a um code review product uh that you can also check out.
07:24It's pretty good.
07:25And the idea there is we're trying to automate code review um as much as possible.
07:29This is something that we use ourselves.
07:31Um and increasingly what's important is for your teams to be thinking about how can you cut We talk about taste and judgment so much these days, right?
07:40It's important for you to be trying to codify what you consider best practice and taste in your team's shared like Claude.md, agents.md, anywhere where the agent can then consult during code review.
07:51And then people are responsible for checking on the agent's evidence.
07:55So, how do we know?
07:56Some If I'm building out um a piece of UI that's going to be, you know, used by a user in the browser, I want to make sure that it's tested in the browser, maybe generated me a video of that experience where it's been clicking around and gives me confidence.
08:10Well, okay, well, this works.
08:11If I care about the mobile or desktop experience, maybe it generates me screenshots or videos of that.
08:16But basically, you want to get to a place where the agent's giving you as much verification evidence as possible.
08:21Cool.
08:21Yeah, verification is key.
08:23I think you need to verify everything that the software factories will produce and that's what we are going to talk about today.
08:29Uh so Uh another these stats by the way are from uh the state of code survey report that will be coming out very soon from Sonar.
08:37Uh we surveyed about a thousand more than a thousand developers and we found some interesting data which will give us insights as to what's happening in the industry today and where we are headed.
08:46So, 84% say AI features uh uh gets gets features and fixes put to production faster.
08:52The pull requests are getting larger and more complex.
08:55Uh and you know, yet uh only uh 31% more code is reaching reaching the production without review.
09:02So, that's a little little bit of a scary fact particularly for uh production uh applications in the regulated industries in the mission-critical applications where you absolutely make need to make sure
09:14uh that you know, uh you have to verify that code and you're confident in the code that is being shipped.
09:20So, uh what are the where are the next gains going to come from, Addy?
09:23I think the next gains are going to come from a combination of us really nailing verification, right?
09:29As well as model quality.
09:30But models are already like if you you take a look at Opus 55, model quality has already gotten to a place where it's pretty good.
09:36Um and I would say that us being able to share best practices about that verification step is going to become increasingly more important because if you tell people, yeah, just like verify the work, you know, their next question is, what does that even mean?
09:47Yeah.
09:48And so, I think that the conversation needs to evolve into specifically what kinds of tests are useful.
09:53We lean a lot on end-to-end tests, property tests, any visual verification tests, things that can help us just get that confidence to merge those changes in automatically.
10:02Cool.
10:03You mentioned light factories.
10:04Want to go down that and that's the way uh direction we are headed.
10:07Uh our survey results show that, you know, today 5% of the companies are using uh software factories.
10:13It will get to 34% and surprisingly 39% in the financial services.
10:17How many of you are using software factories today?
10:19Just curious.
10:22A handful of you.
10:23And how many are planning to use software factories by next year?
10:29Yeah, that's a good question.
10:30How do you define a software factory?
10:31Would you like to take that question?
10:32Yeah, yeah, sure.
10:33So, we we talked about definition of software factories just a little bit earlier.
10:37A software factory is basically loops at scale.
10:39And when we think about this, it's all about like what end of the spectrum are you at in terms of how much people are still involved in the process.
10:47The more that the agents are able to generate verification and give you confidence, the more that you can get to that place where you go dark.
10:53And I think that yeah, most people are still somewhere in that spectrum and are not quite as this data shows at that dark point just yet.
11:00Yep.
11:00Just to add to what Addy said, I think software factory, think of it like a production line where you start defining the intent and your specifications.
11:09Then you start writing the code.
11:10So, it's an assembly line where each of these software development life cycle steps happen.
11:15And automating them at scale where there's pretty much in a dark factory, there would be no lights on.
11:22It's autonomously working, fully autonomously working.
11:25A light factory has uh is where people are involved in the process at every step or most steps.
11:32Uh whereas a dark factory, the goal would be to turn off the lights and let the factory run independently 24/7.
11:38Uh all right.
11:39So, you talked about verification.
11:41Uh I'll talk a little bit about what we think at Sonar about verification.
11:46If you saw our keynote earlier today from our CEO, he talked about verification should be multi-layered.
11:52Uh we absolutely 100% believe that each company should adopt a zero trust multi-layered verification approach.
11:58And verification should be done in every loop.
12:01Whether you're operating inside the agentic loop or whether you're working inside the CI verification loop or whether you're doing code maintenance.
12:09In each and every loop, it is important to verify the output of the factories or the agents that code that the agents are producing.
12:17So, we look at it in terms of layers and think of it like, you know, the there's first layer which is a deterministic layer which catches uh bugs, security issues,
12:28things like SQL injection, buffer overflow bugs, and all of those are caught.
12:33And then there's a second layer where agentic review happens where agents are used to like Adi was talking about, agents review review that code.
12:41And in that agentic review, they catch logic and intent issues which the deterministic layer is not well suited to catch.
12:49So, we we use each layer is targeted at catching a specific types of issues.
12:54And then the third layer could be a software supply chain security layer where you look at all the CVEs, all your malware detection happens, all your malicious package detection happens.
13:05All of that happens in the third layer.
13:06So, it's a layered approach where each layer is tuned and optimized to catch a certain type of issue.
13:12So, we absolutely recommend doing a multi-layered verification.
13:16Oops.
13:19So, there's another stat here.
13:2128% you know, companies have approved agents and shared controls and very few shared controls.
13:29And 5% are operating software factories autonomously.
13:33What are your thoughts there, Adi?
13:34Uh in terms of
13:37So, I've talked to so many companies where a couple of months ago one of their biggest worries was they were allowing their engineering teams to go off and play around with agents and they got to a place where
13:46there were hundreds, thousands of agents in some cases and they just didn't know what they were doing.
13:51Like, what are these agents actually doing?
13:52They're touching all of our systems, accessing all kinds of data, maybe they're deploying to production.
13:57And so, a lot of enterprises think about concerns like, you know, agent identity and telemetry and being able to track what's actually happening.
14:05Yeah.
14:06And I'd say that the three biggest things for a lot of companies are how do we think about control, how do we think about permissions, and how do we think about the sandbox?
14:13Like, how are we making sure that the agent isn't jumping out of the sandbox and doing things that it shouldn't be doing?
14:17Yeah.
14:17Yeah, we we at sonar we call it guide rails.
14:20So, instead of guard rails, we're thinking of calling it as guide rails, where we guide the agents to follow the real tracks, right?
14:27A specific path to take and prevent them from going out of bounds or so to say going out of the sandbox like Addie was saying.
14:34So, it's very very important to have guide rails for the agent and govern the agents that you're working with.
14:40Uh So, let's uh we're coming to a close now as my timer is telling me we just have a few minutes left.
14:47Let's go get into a lightning round.
14:49Uh one sentence response for each of these questions.
14:51And you've seen the questions already on the slide, so
14:53Can these be long sentences?
14:55They They can be as long as the time permits.
14:58Perfect.
14:59Uh light or dark factory for most teams by end of 2027, yes or no?
15:03I think we're going to see uh light factories with some dark corners.
15:06I don't think we're going to get to everybody in this room using dark factories entirely, but it's going to be something close to that.
15:12I I would guess Anthropic would do it this year or next year.
15:18[laughter]
15:20All right.
15:21Second question.
15:22Should agent-written code be labeled in the PR?
15:26I personally think it should.
15:27Historically, we have felt a great deal of ownership around our code, pride around the fact that, "Hey, I wrote this thing.
15:33I'm putting my name on it."
15:35And at Anthropic, like we we think about this a little bit differently.
15:38We've evolved to the point where we don't really mind so much if Claude is like labeled as, "Hey, I'm I'm the one that wrote this thing."
15:45because we're a little bit more focused on the outcomes and making sure that the code, the features, like what we're shipping to users is the right thing.
15:52And so, I personally think it's fine for the code to be written labeled as being coming from an agent.
15:58And that can also help us with things like figuring out, "Okay, well, if the agent wrote it, what are some of the constraints and extra checks that we need to keep in place?"
16:04I I have the same point of view, Addie.
16:05I think in the short run, at least in the foreseeable you know, 6 to 9 months, we should label the PRs till we are confident enough in our factories.
16:14We do want to know and track it, and it's also important for the financial customers here in terms of auditability.
16:20If you if you know, if you label it, you know who's writing the code because we are at the end of the day still accountable for
16:26Yeah, absolutely.
16:26Accountability is the most important part here.
16:29Like we still label like who is the owner for this change.
16:32I might have kicked off a cloud cloud run to complete a piece of work, but I'm I'm still on the hook.
16:36If something goes wrong, I have to go and fix it.
16:38Yep, yep.
16:39Key point.
16:40Yep.
16:40Uh the next question, actually I'll improvise it here because this is a leadership session.
16:45What one skill a junior developer or leader should build on this year?
16:51I think that you should push your teams, if you're a leader, you should push your teams to really think hard about what verification means.
16:57Um how are you already approaching things like testing?
17:00What is the gap between that and you getting to a place where you're very comfortable with the agents being able to sign off on the code being good enough to merge on their own.
17:08And so figuring that out, if you're a junior, it's the other side of that.
17:12Like actually getting more experience in the different flavors of tests, whether it's like end-to-end tests, property tests, etc.
17:18Yep.
17:19And the last uh you you actually not following the rules, you're going beyond one sentence, but that's cool.
17:24Uh The last one
17:2730 seconds left.
17:28I'm fine.
17:28We can go a minute over, I'm sure.
17:30Uh the agent opened the PR.
17:32It is safe to merge when?
17:35You have full confidence nothing's going to go wrong or that you have mitigated that as much as possible.
17:39Love it.
17:40All right.
17:41Uh here's my last slide and then we'll wrap it up.
17:44Uh so what separates uh teams that are shipping uh fast but not shipping blind?
17:50I think that three key takeaways for you.
17:52Uh writing code is the fastest part and that is solved.
17:55Strong review and verification for our teams is what will will help you maintain that speed.
18:00Fast to write is not fast to ship.
18:03Uh you have to basically count what is getting merged, looked at what is getting merged, and make sure it doesn't escape the sandbox and the things like that before it gets into the production.
18:11Verify everything.
18:12And autonomy is earned.
18:14Uh you We will get to software factories very soon if not next year.
18:18Uh you need to put in independent automated verification inside each loop of your factory.
18:24Make sure that you can trust the output that the factory is producing.
18:28Uh, with that, I'll wrap it up.
18:30Thanks for attending our talk today, uh, a conversation with Adi.
18:34Uh, again, Adi, I appreciate you taking the time and joining us here in this conversation.
18:38Uh, visit us at the big red booth, uh, the sonar booth in the expo hall if you have any further questions or would like to, uh, talk about it.
18:45Thank you again.