Rebuilding CI for AI-Scale Commit Volume — Gautam Korlam, Sonar

AI Engineer · 19 min · 186 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Hello everyone.
00:00Uh welcome to my talk.
00:02Uh today we're going to be talking about how we are rebuilding CI for AI scale commit volume.
00:09And this comes from a lot of experience from build and merge infrastructure that I have been working on for a decade plus at this point.
00:17And kind of want to walk you through all the new bottlenecks we're seeing in software factories and also specifically for highly regulated industries and banks.
00:26How does that translate?
00:27How do we keep things more compliant and auditable while being able to move fast?
00:32So let's dive in.
00:34My name is Goautam.
00:35I am a co-founder and CTO at QAR and recently joined Sonar as a principal engineer.
00:40After the acquisition pre previous to I was at Uber for almost a decade working on build and merge infrastructure for mono repos.
00:48So everything from build, CI/CD, ids, you name it to make sure developers are productive.
00:55And now at QAR as part of Solar I'm trying to make sure that code that's been checked in is verified and all the code written by agents is of high quality before it goes in.
01:08So before I get started I want to just paint a bit of history picture.
01:13Uh at Uber we had agent scale coming volume before they were agents.
01:17So what I mean by this is we had one giant monor repo and we were doing like thousands of changes a day and with very long running builds.
01:25This meant that if you just kept everything serialized and merged them one at a time so as not to break your uh main branch you would take a really long time up to days so 20 days in this case to make sure that everything was green.
01:40And if he skipped everything and just let everyone merge to me, we would see red main all the time because changes would conflict with each other and even though the CI would be green on one of them
01:51combined, they might actually break.
01:53So we would see like our iOS monitor would be like red all the time and that was not productive for our developers.
01:59To fix this, what we did was we made a system called summit cube which we have a paper on and we tested all the likely possibilities of commits that are coming in uh and making sure that we can like reduce the wait time so we can test these things in parallel.
02:14We could also predict what could break by testing all the combinations.
02:18So once we know that a PR might break, we might remove it back from the queue and let the author know that they have to go fix it before they can push it back up after rebasing.
02:28And finally like to keep the trunk clean only if it passes with all the combination of commits coming in it would be let into go directly onto the trunk.
02:38So you want to make sure that the overhead was worth it for keeping our trunk clean.
02:42And this is still running today at Uber.
02:44I'm sure now we have like merge cues that GitHub came up with and all the other code hosts but for back then there was not a system.
02:51So we developed this.
02:52[snorts]
02:53So the problem with this approach is currently all these cues that we have.
02:58So that's either PCI or review are a centralized bottleneck in the SDLC.
03:02So all these cues were built for human speed and once you build it from human speed what happens is they are not able to keep up once you go to agents because humans generate changes
03:12used to generate changes like few times an hour.
03:15So we could do testing and review one at a time.
03:19Now with agents developers are writing like several PRs uh in many in a day and they don't always wait for the queue they all want to like come in at the same time which means your CI and review cues are piling up and having pressure
03:34and add to that like the patterns of software factories and background agents triggering on like various workflows suddenly you see like everything slowing down to a crawl because your bottlenecks are still going BCI in review and they're over capacity in this
03:50So what does this mean uh for teams today?
03:53Right?
03:54So just to put things into perspective uh our own uh rate of commits has almost like 10xed since we started guitar uh about 3 years ago and it's just like uh unbelievable with the same size team we're still very small we're able to like push so much more code
04:10uh and we even see this in the industry trends on guitar.
04:13So this graph shows you the number of PRs that we see that guitar reviews over time.
04:18And if you look at the growth trajectory, it's pretty exponential.
04:22Every month almost we're growing 25% month over month of just the amount of PRs that have to review fixed CI and other things.
04:30So this is also I think like corroborated by evidence that industry players like GitHub have been putting out where they're seeing like uh staggering growth in PRs and this is now coming for every single team not if our small team is seeing this I'm sure like the larger teams and banks are seeing it like at a much
04:48higher degree as they are going through their AI adoption journeys.
04:52[snorts]
04:52So the problem I was talking about right so if agents are opening PRs and if you make them uh check everything through the CI system uh individually they might pass but once you kind of merge them back on domain
05:05your truck might still be broken.
05:06So how do you like solve for this like how do you keep things green uh at this high rate of pace right so this is very important kind of like how we solve submit Q the changes themselves are not
05:18uh testing against each other so there needs to be some sort of change in either the CI system as well as how you architecture code and in the cues to be smarter to be able to solve this
05:28and the problem we're also seeing with everyone trying to adopt like software factories which is having agents write all the code uh from the spec and then just having humans be responsible for the spec and review
05:41is that the factory line that exists today was built for human speed as I mentioned which means that as agents push more changes uh every part of the factory line is going to be slowing down.
05:52So for example with build and test you might have lots of rebuilds whenever uh compilation breaks or you might have integration tests you might have flaky tests that are not reproducible locally
06:03and that just slows everything down.
06:05So because you have to retry them and it could take hours in many cases with a complex build.
06:10Add to the fact that now you have review and approval.
06:14So a lot of you probably already use like AI to review your code.
06:17Hopefully uh you're not doing every single review with with humans.
06:21You ideally are reviewing with AI as a first pass.
06:24Now the reviewers become the cue if you don't have that system in place.
06:28And even after you review, there might be some criteria you might want additionally to make sure that it's approved fully because what's happening now in the industry is a lot of folks are just rubber stamping them because they cannot keep up with the high rate of PR.
06:41They were relying on other signals now to be able to just get through.
06:44So the process has to change.
06:46And the other problem right now we're having is like a lot of these checks are running.
06:50There's uh a lot of records that we keep that are expiring.
06:54So like your old logs don't exist as they expire out.
06:57There is no stamp of like what was actually checked because there is like just an approval stamp.
07:03So there needs to be some record of these things for auditability and compliance checks to happen.
07:10The other issue we're seeing is that as the code volume increases the time taken for repair also is adding a ton of overhead.
07:18Uh what we see on our platform at least 13% of PRs had a recorded CI failure.
07:24This is just one or more CI failures.
07:26Which means in all these cases there is a separate round trip to go either back to the developer machine or for an agent to understand the logs root cause of failure
07:36fix it push another commit and then keep iterating till it's green.
07:40And this is just CI because most of the time if a human reviewer wants to look at a PR they're not going to take a look unless CI is green because that's the first line of defense.
07:50So that also like slows uh the review process because they can't go in parallel because usually we want to look at something once it's finalized and every lab is now paying for
08:00review.
08:01So if your review comments come in and you're doing another pass same at break again and it kind of like repeats the cycle.
08:09So how do we make this better like what what do we do to make this problem more manageable as agents are pushing down more code.
08:16So we have discovered this uh obviously back to but also in guitar steam right now.
08:23But the most obvious thing is of course like make your CI really quick and fast which means that you may want to like remove a lot of the expensive checks.
08:30Try to offload them to like post merge even and try to make the CI step itself really quick because your repairs will be much more efficient.
08:39So you can paralyze things, you can chart, you can do like uh caching.
08:43And we'll talk about some of the techniques that we have used as well.
08:46You also want to make sure the review is fast.
08:49So not just like a deep review, but you also want to have maybe multiple passes that are like giving quicker signal to your developers.
08:56So they're able to like use them as a way to understand in what scenarios they may want to go deeper for higher risk changes or maybe for like lower risk ones even like let the AI approve and merge them.
09:07And for the checks that are already running in CI, if you can move them to the left, it's way cheaper to catch them.
09:13And this actually uh is very relevant to checks that happen even past CI.
09:18For example, you might have like expensive regression tests or like things that might fail in production.
09:24Uh not every time it's possible to like move things to the left, but as much as possible, if you can move them, it means you have less bottlenecks down the line.
09:31And locally, agents have done more context anyway to fix the issue that they're facing.
09:36And you also want to have policies.
09:37This is very important for regulated industries.
09:40So a story we have at Back at Uber is we lost like the license to operate in London because there was a glitch in the software that allowed drivers to get on the platform without all the proper checks and that was because of a feature flag change.
09:54So we had to put in a process in place so that every change affecting London that was going through the SDLC would need some sort of like a a lineage of okay does this change affect London?
10:05If so, what are the controls in place and we were able to win our license back?
10:09Now, a lot of you might be facing these kind of like challenges where agents just make it much harder to understand what's happening in your codebase.
10:16So, how do you ensure your policies are actually being respected?
10:19So, you need to have a way to like enforce these rules beyond CI like just because the code compiles doesn't mean that it's following a particular specifications.
10:28It could be like HIPPA.
10:29It could be like uh Tora.
10:31There's so many other policies that these agents are not going to be aware of unless your agents are enforcing it on the other end.
10:38And also you want to make sure that you record that it was enforced for auditability.
10:44So just a quick story about how we made our CI faster as well.
10:48So we were at a baseline, we were I would say like still fast but like very annoying when you're trying to like push more code.
10:55uh we were around like 10 minutes for our CI and we wanted to like bring that down significantly.
11:00So we started think looking at how can we cache outputs uh we use rust for our build system.
11:05So we started like caching as much as possible and reusing them.
11:08And we also looked at ways to like reduce churn in the compiler.
11:12So we added some more fixes to like ensure that the compilation part is more cachable and is invalidated less often.
11:19And by adding more fixes on top of this, by sharding and by removing like these overheads, we're able to bring the time of CI almost like less than a minute for incremental
11:30steps, which means once the cold CI is warmed up, incremental builds are way way quick quicker and we can push a ton more code.
11:37The other issue is also review.
11:39So just because the CI is green, what was funny was when we were measuring our uh P90 time of like PRs making into production, the vast majority of time was spent in review back at Uber and even today to be honest like
11:54most teams spend more time reviewing code and doing other kinds of testing before they are comfortable merging the code into main.
12:01So this is something that needs to be addressed on top of CI as well because right now what's happening is agents are of course like generating code at the same time the AI agents are also like finding issues on on your
12:16uh code review system and these automated reviews are also findings that you would have to now go back and address.
12:22So either you have to pull context locally for your agent or have an agent on CI iterate and fix and humans are able to review whatever is left after the AI took its first pass.
12:32And this needs to be part of the equation so you're not wasting precious human cycles on more and more agent generated PRs.
12:42Uh here's an example for us.
12:43Uh we were refactoring our code base to have faster CI.
12:47So we were using next text to parallelize our uh tests and we had this like shared database that we were spinning up to make sure that we could set it up once and all our database tests would reuse it.
13:00But the problem was that there were some guards we didn't put in place which meant that when CI gets very busy uh there is like uh crosscontamination and test might become flaky.
13:10Guitar actually is a code reviewer.
13:12It caught it on our PR before we merged it.
13:14So we didn't have to waste time after.
13:16And GAR also fixed the issue for us because it suggested the fix, committed it, and added the engineer as a co-author.
13:23By doing this, we're able to like prevent expensive mistakes before the code made it all the way into main.
13:29The other thing that we do at Qar uh interestingly and as we see more and more of our customers start doing is we're using policies and criteria to decide
13:39when is it appropriate to actually even like approve changes.
13:43There's a lot of changes that are like routine or lower risk documentation dependency bumps of course if they're not breaking any APIs security patches uh even things like uh lower risk changes for internal tooling that can potentially be auto approved.
13:57Uh so guitar supports these things out of the box and you can also add additional custom rules which are plain English that you can use to like create the approval and merge on and we are seeing like teams uh who are like very AI native starting to adopt these more and more.
14:12We see I think on our platform the rate at which folks are adopting like the auto approval and automerge grow really quickly across teams even from large teams to like 100 plus,000
14:23plus developers.
14:24So here's an example of approval policies.
14:26So if it's a dependency bump and doc changes, we should be approving them.
14:30Any changes under sensitive stuff like payments or o might need a human approver on top.
14:36And then of course you don't want to have the agent who made the change approve its change because that would not be uh zero trust.
14:44So in this case as agents are generating PRs the PRs are getting approved.
14:49When there is a new change that doesn't meet the criteria then we would have to escalate to another approver.
14:55And this can be done with your own agents of course but also gitas out of the box if you want to try it.
15:03And the other problem with uh just relying on CI today is that just because CI is green as I mentioned doesn't mean the code is compliant in many cases.
15:11So in this case uh the CI is passing and there is some metadata that we have about it on the rerun a flaky test uh is going to fail which means that when you force push to fix that thing now the branch is pointing at another piece of code.
15:28So the problem with this is that over time most CI data is going to be expired.
15:33So unless you have a record of what happened and what was fixed, you will not be able to like be compliant or be able to audit what actually happened.
15:41Which is why I think things like agent trajectories and like your uh outcomes after each push are very very important.
15:49So whatever information you can record on the PR or any other system of record is going to be a source of truth as your artifacts expire.
15:57So the control here is making sure that the change was approved and tested by someone who did not write it.
16:04If you don't have that information, it could be a problem.
16:07So here's an example.
16:08We have a pipeline that failed in the left and a pipeline that passed.
16:12But the difference here is that if you record properly why the failure occurred and someone else was able to approve it because it was a known failure mode and
16:21[snorts]
16:21it was saved with the change that made it in.
16:24This thing actually passes your checks versus something where you don't have a record of what happened but you just blindly trust the agent pushing code because it could be a test that was actually
16:35uh not high quality.
16:36There was no record of all the compliance checks that were like recorded.
16:39So you want to make sure that on the uh more sensitive uh industries you want to have all of these decisions recorded as rules or compliance checks on top of your PR in addition to code review.
16:53So the what are some of the kinds of records that we want to keep.
16:57So here's a few that we keep on which I think are just examples.
17:01It could be different for folks.
17:03So for example, which coding agent made uh the PR?
17:07Where did the PR originate from?
17:09What was the session that originated it?
17:11And who actually started it?
17:13Could it be a trigger from like automated workflow or could it be just an engineer doing prompting?
17:19And for each of these like CIS that run, what is a commit where CI was trusted and a save test report, not just a status?
17:27And making sure that each finding has like a record of how it was resolved.
17:31Was it dismissed?
17:32Was it followed up?
17:34Was it like fixed on the PR?
17:36And then making sure the approver is someone a apart from the bot that was prompted by the same person.
17:42And making sure that any overrides are explicitly named.
17:44So if there is a red build or like a control code coverage skipped, you want to make sure there's a reason for when why that check failed and why we're okay with that in that scenario.
17:54And making sure that the code that merges actually has all this and it's not like a small change on top that was merged.
18:01So all the checks are done every single time.
18:04So in this case we want to make sure that agents cannot approve their own changes.
18:08Engineer who prompted it is also not an author.
18:11And if you have these practices in place now we can actually start removing some of these bottlenecks because the CI being faster will unblock your builds.
18:22So your flow is much nicer and making sure you can like move some checks left will also make sure that your test load is going to be reduced quite a bit.
18:30On the CI side, if you can automate review for the most part and only escalate to humans for exceptional cases and potentially even auto approve lowrisk or internal changes,
18:42then you remove further bottlenecks and then you have less that has reaching humans uh for a software factory to perform to its uh potential.
18:52And of course after every single approval and merge you want to record what happened because as factories are going to ramp up if you don't have an idea of what happened in the past that context is going to be very useful on how to like iterate and make your software better over time.
19:07Uh that's it for me.
19:09Uh would love for you folks to also try out guitar and see how you can improve your software factories.
19:15And with that uh thank you so much for for your