WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7b9tov5Uc5s

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7b9tov5Uc5s.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.080 --> 00:00:00.880
Hello everyone.

s2
00:00:00.880 --> 00:00:02.560
Uh welcome to my talk.

s3
00:00:02.560 --> 00:00:09.040
Uh today we're going to be talking about how we are rebuilding CI for AI scale commit volume.

s4
00:00:09.040 --> 00:00:17.359
And this comes from a lot of experience from build and merge infrastructure that I have been working on for a decade plus at this point.

s5
00:00:17.359 --> 00:00:26.720
And kind of want to walk you through all the new bottlenecks we're seeing in software factories and also specifically for highly regulated industries and banks.

s6
00:00:26.720 --> 00:00:27.840
How does that translate?

s7
00:00:27.840 --> 00:00:32.079
How do we keep things more compliant and auditable while being able to move fast?

s8
00:00:32.079 --> 00:00:34.239
So let's dive in.

s9
00:00:34.239 --> 00:00:35.360
My name is Goautam.

s10
00:00:35.360 --> 00:00:40.640
I am a co-founder and CTO at QAR and recently joined Sonar as a principal engineer.

s11
00:00:40.640 --> 00:00:48.960
After the acquisition pre previous to I was at Uber for almost a decade working on build and merge infrastructure for mono repos.

s12
00:00:48.960 --> 00:00:55.520
So everything from build, CI/CD, ids, you name it to make sure developers are productive.

s13
00:00:55.520 --> 00:01:07.920
And now at QAR as part of Solar I'm trying to make sure that code that's been checked in is verified and all the code written by agents is of high quality before it goes in.

s14
00:01:08.000 --> 00:01:13.280
So before I get started I want to just paint a bit of history picture.

s15
00:01:13.280 --> 00:01:17.040
Uh at Uber we had agent scale coming volume before they were agents.

s16
00:01:17.040 --> 00:01:25.920
So what I mean by this is we had one giant monor repo and we were doing like thousands of changes a day and with very long running builds.

s17
00:01:25.920 --> 00:01:40.240
This meant that if you just kept everything serialized and merged them one at a time so as not to break your uh main branch you would take a really long time up to days so 20 days in this case to make sure that everything was green.

s18
00:01:40.240 --> 00:01:51.600
And if he skipped everything and just let everyone merge to me, we would see red main all the time because changes would conflict with each other and even though the CI would be green on one of them

s19
00:01:51.600 --> 00:01:53.200
combined, they might actually break.

s20
00:01:53.200 --> 00:01:59.200
So we would see like our iOS monitor would be like red all the time and that was not productive for our developers.

s21
00:01:59.200 --> 00:02:14.640
To fix this, what we did was we made a system called summit cube which we have a paper on and we tested all the likely possibilities of commits that are coming in uh and making sure that we can like reduce the wait time so we can test these things in parallel.

s22
00:02:14.640 --> 00:02:18.560
We could also predict what could break by testing all the combinations.

s23
00:02:18.560 --> 00:02:28.560
So once we know that a PR might break, we might remove it back from the queue and let the author know that they have to go fix it before they can push it back up after rebasing.

s24
00:02:28.560 --> 00:02:38.160
And finally like to keep the trunk clean only if it passes with all the combination of commits coming in it would be let into go directly onto the trunk.

s25
00:02:38.160 --> 00:02:42.640
So you want to make sure that the overhead was worth it for keeping our trunk clean.

s26
00:02:42.640 --> 00:02:44.560
And this is still running today at Uber.

s27
00:02:44.560 --> 00:02:51.519
I'm sure now we have like merge cues that GitHub came up with and all the other code hosts but for back then there was not a system.

s28
00:02:51.519 --> 00:02:52.921
So we developed this.

s29
00:02:52.921 --> 00:02:53.680
[snorts]

s30
00:02:53.680 --> 00:02:58.080
So the problem with this approach is currently all these cues that we have.

s31
00:02:58.080 --> 00:03:02.959
So that's either PCI or review are a centralized bottleneck in the SDLC.

s32
00:03:02.959 --> 00:03:12.959
So all these cues were built for human speed and once you build it from human speed what happens is they are not able to keep up once you go to agents because humans generate changes

s33
00:03:12.959 --> 00:03:15.519
used to generate changes like few times an hour.

s34
00:03:15.519 --> 00:03:19.040
So we could do testing and review one at a time.

s35
00:03:19.040 --> 00:03:34.239
Now with agents developers are writing like several PRs uh in many in a day and they don't always wait for the queue they all want to like come in at the same time which means your CI and review cues are piling up and having pressure

s36
00:03:34.239 --> 00:03:49.920
and add to that like the patterns of software factories and background agents triggering on like various workflows suddenly you see like everything slowing down to a crawl because your bottlenecks are still going BCI in review and they're over capacity in this

s37
00:03:50.799 --> 00:03:53.760
So what does this mean uh for teams today?

s38
00:03:53.760 --> 00:03:54.080
Right?

s39
00:03:54.080 --> 00:04:10.799
So just to put things into perspective uh our own uh rate of commits has almost like 10xed since we started guitar uh about 3 years ago and it's just like uh unbelievable with the same size team we're still very small we're able to like push so much more code

s40
00:04:10.799 --> 00:04:13.519
uh and we even see this in the industry trends on guitar.

s41
00:04:13.519 --> 00:04:18.880
So this graph shows you the number of PRs that we see that guitar reviews over time.

s42
00:04:18.880 --> 00:04:22.479
And if you look at the growth trajectory, it's pretty exponential.

s43
00:04:22.479 --> 00:04:30.320
Every month almost we're growing 25% month over month of just the amount of PRs that have to review fixed CI and other things.

s44
00:04:30.320 --> 00:04:48.000
So this is also I think like corroborated by evidence that industry players like GitHub have been putting out where they're seeing like uh staggering growth in PRs and this is now coming for every single team not if our small team is seeing this I'm sure like the larger teams and banks are seeing it like at a much

s45
00:04:48.000 --> 00:04:52.133
higher degree as they are going through their AI adoption journeys.

s46
00:04:52.133 --> 00:04:52.639
[snorts]

s47
00:04:52.639 --> 00:05:05.199
So the problem I was talking about right so if agents are opening PRs and if you make them uh check everything through the CI system uh individually they might pass but once you kind of merge them back on domain

s48
00:05:05.199 --> 00:05:06.720
your truck might still be broken.

s49
00:05:06.720 --> 00:05:18.320
So how do you like solve for this like how do you keep things green uh at this high rate of pace right so this is very important kind of like how we solve submit Q the changes themselves are not

s50
00:05:18.320 --> 00:05:28.720
uh testing against each other so there needs to be some sort of change in either the CI system as well as how you architecture code and in the cues to be smarter to be able to solve this

s51
00:05:28.720 --> 00:05:41.680
and the problem we're also seeing with everyone trying to adopt like software factories which is having agents write all the code uh from the spec and then just having humans be responsible for the spec and review

s52
00:05:41.680 --> 00:05:52.240
is that the factory line that exists today was built for human speed as I mentioned which means that as agents push more changes uh every part of the factory line is going to be slowing down.

s53
00:05:52.240 --> 00:06:03.600
So for example with build and test you might have lots of rebuilds whenever uh compilation breaks or you might have integration tests you might have flaky tests that are not reproducible locally

s54
00:06:03.600 --> 00:06:05.600
and that just slows everything down.

s55
00:06:05.600 --> 00:06:10.720
So because you have to retry them and it could take hours in many cases with a complex build.

s56
00:06:10.720 --> 00:06:14.160
Add to the fact that now you have review and approval.

s57
00:06:14.160 --> 00:06:17.840
So a lot of you probably already use like AI to review your code.

s58
00:06:17.840 --> 00:06:21.600
Hopefully uh you're not doing every single review with with humans.

s59
00:06:21.600 --> 00:06:24.639
You ideally are reviewing with AI as a first pass.

s60
00:06:24.639 --> 00:06:28.560
Now the reviewers become the cue if you don't have that system in place.

s61
00:06:28.560 --> 00:06:41.280
And even after you review, there might be some criteria you might want additionally to make sure that it's approved fully because what's happening now in the industry is a lot of folks are just rubber stamping them because they cannot keep up with the high rate of PR.

s62
00:06:41.280 --> 00:06:44.319
They were relying on other signals now to be able to just get through.

s63
00:06:44.319 --> 00:06:46.639
So the process has to change.

s64
00:06:46.639 --> 00:06:50.560
And the other problem right now we're having is like a lot of these checks are running.

s65
00:06:50.560 --> 00:06:54.560
There's uh a lot of records that we keep that are expiring.

s66
00:06:54.560 --> 00:06:57.919
So like your old logs don't exist as they expire out.

s67
00:06:57.919 --> 00:07:03.440
There is no stamp of like what was actually checked because there is like just an approval stamp.

s68
00:07:03.440 --> 00:07:10.000
So there needs to be some record of these things for auditability and compliance checks to happen.

s69
00:07:10.479 --> 00:07:18.800
The other issue we're seeing is that as the code volume increases the time taken for repair also is adding a ton of overhead.

s70
00:07:18.800 --> 00:07:24.240
Uh what we see on our platform at least 13% of PRs had a recorded CI failure.

s71
00:07:24.240 --> 00:07:26.720
This is just one or more CI failures.

s72
00:07:26.720 --> 00:07:36.160
Which means in all these cases there is a separate round trip to go either back to the developer machine or for an agent to understand the logs root cause of failure

s73
00:07:36.160 --> 00:07:40.960
fix it push another commit and then keep iterating till it's green.

s74
00:07:40.960 --> 00:07:50.560
And this is just CI because most of the time if a human reviewer wants to look at a PR they're not going to take a look unless CI is green because that's the first line of defense.

s75
00:07:50.560 --> 00:08:00.879
So that also like slows uh the review process because they can't go in parallel because usually we want to look at something once it's finalized and every lab is now paying for

s76
00:08:00.879 --> 00:08:01.199
review.

s77
00:08:01.199 --> 00:08:09.680
So if your review comments come in and you're doing another pass same at break again and it kind of like repeats the cycle.

s78
00:08:09.840 --> 00:08:16.639
So how do we make this better like what what do we do to make this problem more manageable as agents are pushing down more code.

s79
00:08:16.639 --> 00:08:23.440
So we have discovered this uh obviously back to but also in guitar steam right now.

s80
00:08:23.440 --> 00:08:30.800
But the most obvious thing is of course like make your CI really quick and fast which means that you may want to like remove a lot of the expensive checks.

s81
00:08:30.800 --> 00:08:39.760
Try to offload them to like post merge even and try to make the CI step itself really quick because your repairs will be much more efficient.

s82
00:08:39.760 --> 00:08:43.519
So you can paralyze things, you can chart, you can do like uh caching.

s83
00:08:43.519 --> 00:08:46.720
And we'll talk about some of the techniques that we have used as well.

s84
00:08:46.720 --> 00:08:49.040
You also want to make sure the review is fast.

s85
00:08:49.040 --> 00:08:56.080
So not just like a deep review, but you also want to have maybe multiple passes that are like giving quicker signal to your developers.

s86
00:08:56.080 --> 00:09:07.440
So they're able to like use them as a way to understand in what scenarios they may want to go deeper for higher risk changes or maybe for like lower risk ones even like let the AI approve and merge them.

s87
00:09:07.440 --> 00:09:13.920
And for the checks that are already running in CI, if you can move them to the left, it's way cheaper to catch them.

s88
00:09:13.920 --> 00:09:18.320
And this actually uh is very relevant to checks that happen even past CI.

s89
00:09:18.320 --> 00:09:24.160
For example, you might have like expensive regression tests or like things that might fail in production.

s90
00:09:24.160 --> 00:09:31.519
Uh not every time it's possible to like move things to the left, but as much as possible, if you can move them, it means you have less bottlenecks down the line.

s91
00:09:31.519 --> 00:09:36.320
And locally, agents have done more context anyway to fix the issue that they're facing.

s92
00:09:36.320 --> 00:09:37.839
And you also want to have policies.

s93
00:09:37.839 --> 00:09:40.160
This is very important for regulated industries.

s94
00:09:40.160 --> 00:09:54.080
So a story we have at Back at Uber is we lost like the license to operate in London because there was a glitch in the software that allowed drivers to get on the platform without all the proper checks and that was because of a feature flag change.

s95
00:09:54.080 --> 00:10:05.360
So we had to put in a process in place so that every change affecting London that was going through the SDLC would need some sort of like a a lineage of okay does this change affect London?

s96
00:10:05.360 --> 00:10:09.040
If so, what are the controls in place and we were able to win our license back?

s97
00:10:09.040 --> 00:10:16.320
Now, a lot of you might be facing these kind of like challenges where agents just make it much harder to understand what's happening in your codebase.

s98
00:10:16.320 --> 00:10:19.600
So, how do you ensure your policies are actually being respected?

s99
00:10:19.600 --> 00:10:28.560
So, you need to have a way to like enforce these rules beyond CI like just because the code compiles doesn't mean that it's following a particular specifications.

s100
00:10:28.560 --> 00:10:29.519
It could be like HIPPA.

s101
00:10:29.519 --> 00:10:31.519
It could be like uh Tora.

s102
00:10:31.519 --> 00:10:38.880
There's so many other policies that these agents are not going to be aware of unless your agents are enforcing it on the other end.

s103
00:10:38.880 --> 00:10:44.000
And also you want to make sure that you record that it was enforced for auditability.

s104
00:10:44.399 --> 00:10:48.079
So just a quick story about how we made our CI faster as well.

s105
00:10:48.079 --> 00:10:55.600
So we were at a baseline, we were I would say like still fast but like very annoying when you're trying to like push more code.

s106
00:10:55.600 --> 00:11:00.399
uh we were around like 10 minutes for our CI and we wanted to like bring that down significantly.

s107
00:11:00.399 --> 00:11:05.200
So we started think looking at how can we cache outputs uh we use rust for our build system.

s108
00:11:05.200 --> 00:11:08.959
So we started like caching as much as possible and reusing them.

s109
00:11:08.959 --> 00:11:12.399
And we also looked at ways to like reduce churn in the compiler.

s110
00:11:12.399 --> 00:11:19.279
So we added some more fixes to like ensure that the compilation part is more cachable and is invalidated less often.

s111
00:11:19.279 --> 00:11:30.560
And by adding more fixes on top of this, by sharding and by removing like these overheads, we're able to bring the time of CI almost like less than a minute for incremental

s112
00:11:30.560 --> 00:11:37.920
steps, which means once the cold CI is warmed up, incremental builds are way way quick quicker and we can push a ton more code.

s113
00:11:37.920 --> 00:11:39.440
The other issue is also review.

s114
00:11:39.440 --> 00:11:54.079
So just because the CI is green, what was funny was when we were measuring our uh P90 time of like PRs making into production, the vast majority of time was spent in review back at Uber and even today to be honest like

s115
00:11:54.079 --> 00:12:01.839
most teams spend more time reviewing code and doing other kinds of testing before they are comfortable merging the code into main.

s116
00:12:01.839 --> 00:12:16.480
So this is something that needs to be addressed on top of CI as well because right now what's happening is agents are of course like generating code at the same time the AI agents are also like finding issues on on your

s117
00:12:16.480 --> 00:12:22.079
uh code review system and these automated reviews are also findings that you would have to now go back and address.

s118
00:12:22.079 --> 00:12:32.880
So either you have to pull context locally for your agent or have an agent on CI iterate and fix and humans are able to review whatever is left after the AI took its first pass.

s119
00:12:32.880 --> 00:12:42.000
And this needs to be part of the equation so you're not wasting precious human cycles on more and more agent generated PRs.

s120
00:12:42.000 --> 00:12:43.680
Uh here's an example for us.

s121
00:12:43.680 --> 00:12:47.680
Uh we were refactoring our code base to have faster CI.

s122
00:12:47.680 --> 00:13:00.160
So we were using next text to parallelize our uh tests and we had this like shared database that we were spinning up to make sure that we could set it up once and all our database tests would reuse it.

s123
00:13:00.160 --> 00:13:10.639
But the problem was that there were some guards we didn't put in place which meant that when CI gets very busy uh there is like uh crosscontamination and test might become flaky.

s124
00:13:10.639 --> 00:13:12.240
Guitar actually is a code reviewer.

s125
00:13:12.240 --> 00:13:14.639
It caught it on our PR before we merged it.

s126
00:13:14.639 --> 00:13:16.399
So we didn't have to waste time after.

s127
00:13:16.399 --> 00:13:23.200
And GAR also fixed the issue for us because it suggested the fix, committed it, and added the engineer as a co-author.

s128
00:13:23.200 --> 00:13:29.279
By doing this, we're able to like prevent expensive mistakes before the code made it all the way into main.

s129
00:13:29.279 --> 00:13:39.120
The other thing that we do at Qar uh interestingly and as we see more and more of our customers start doing is we're using policies and criteria to decide

s130
00:13:39.120 --> 00:13:43.120
when is it appropriate to actually even like approve changes.

s131
00:13:43.120 --> 00:13:57.680
There's a lot of changes that are like routine or lower risk documentation dependency bumps of course if they're not breaking any APIs security patches uh even things like uh lower risk changes for internal tooling that can potentially be auto approved.

s132
00:13:57.680 --> 00:14:12.399
Uh so guitar supports these things out of the box and you can also add additional custom rules which are plain English that you can use to like create the approval and merge on and we are seeing like teams uh who are like very AI native starting to adopt these more and more.

s133
00:14:12.399 --> 00:14:23.199
We see I think on our platform the rate at which folks are adopting like the auto approval and automerge grow really quickly across teams even from large teams to like 100 plus,000

s134
00:14:23.199 --> 00:14:24.560
plus developers.

s135
00:14:24.560 --> 00:14:26.720
So here's an example of approval policies.

s136
00:14:26.720 --> 00:14:30.800
So if it's a dependency bump and doc changes, we should be approving them.

s137
00:14:30.800 --> 00:14:36.399
Any changes under sensitive stuff like payments or o might need a human approver on top.

s138
00:14:36.399 --> 00:14:44.240
And then of course you don't want to have the agent who made the change approve its change because that would not be uh zero trust.

s139
00:14:44.240 --> 00:14:49.120
So in this case as agents are generating PRs the PRs are getting approved.

s140
00:14:49.120 --> 00:14:55.920
When there is a new change that doesn't meet the criteria then we would have to escalate to another approver.

s141
00:14:55.920 --> 00:15:03.040
And this can be done with your own agents of course but also gitas out of the box if you want to try it.

s142
00:15:03.040 --> 00:15:11.680
And the other problem with uh just relying on CI today is that just because CI is green as I mentioned doesn't mean the code is compliant in many cases.

s143
00:15:11.680 --> 00:15:28.240
So in this case uh the CI is passing and there is some metadata that we have about it on the rerun a flaky test uh is going to fail which means that when you force push to fix that thing now the branch is pointing at another piece of code.

s144
00:15:28.240 --> 00:15:33.760
So the problem with this is that over time most CI data is going to be expired.

s145
00:15:33.760 --> 00:15:41.440
So unless you have a record of what happened and what was fixed, you will not be able to like be compliant or be able to audit what actually happened.

s146
00:15:41.440 --> 00:15:49.360
Which is why I think things like agent trajectories and like your uh outcomes after each push are very very important.

s147
00:15:49.360 --> 00:15:57.839
So whatever information you can record on the PR or any other system of record is going to be a source of truth as your artifacts expire.

s148
00:15:57.839 --> 00:16:04.079
So the control here is making sure that the change was approved and tested by someone who did not write it.

s149
00:16:04.079 --> 00:16:07.279
If you don't have that information, it could be a problem.

s150
00:16:07.279 --> 00:16:08.560
So here's an example.

s151
00:16:08.560 --> 00:16:12.240
We have a pipeline that failed in the left and a pipeline that passed.

s152
00:16:12.240 --> 00:16:21.856
But the difference here is that if you record properly why the failure occurred and someone else was able to approve it because it was a known failure mode and

s153
00:16:21.856 --> 00:16:21.920
[snorts]

s154
00:16:21.920 --> 00:16:24.480
it was saved with the change that made it in.

s155
00:16:24.480 --> 00:16:35.120
This thing actually passes your checks versus something where you don't have a record of what happened but you just blindly trust the agent pushing code because it could be a test that was actually

s156
00:16:35.120 --> 00:16:36.320
uh not high quality.

s157
00:16:36.320 --> 00:16:39.680
There was no record of all the compliance checks that were like recorded.

s158
00:16:39.680 --> 00:16:53.040
So you want to make sure that on the uh more sensitive uh industries you want to have all of these decisions recorded as rules or compliance checks on top of your PR in addition to code review.

s159
00:16:53.680 --> 00:16:57.279
So the what are some of the kinds of records that we want to keep.

s160
00:16:57.279 --> 00:17:01.440
So here's a few that we keep on which I think are just examples.

s161
00:17:01.440 --> 00:17:03.040
It could be different for folks.

s162
00:17:03.040 --> 00:17:07.520
So for example, which coding agent made uh the PR?

s163
00:17:07.520 --> 00:17:09.600
Where did the PR originate from?

s164
00:17:09.600 --> 00:17:11.679
What was the session that originated it?

s165
00:17:11.679 --> 00:17:13.760
And who actually started it?

s166
00:17:13.760 --> 00:17:19.439
Could it be a trigger from like automated workflow or could it be just an engineer doing prompting?

s167
00:17:19.439 --> 00:17:27.600
And for each of these like CIS that run, what is a commit where CI was trusted and a save test report, not just a status?

s168
00:17:27.600 --> 00:17:31.840
And making sure that each finding has like a record of how it was resolved.

s169
00:17:31.840 --> 00:17:32.720
Was it dismissed?

s170
00:17:32.720 --> 00:17:34.000
Was it followed up?

s171
00:17:34.000 --> 00:17:36.160
Was it like fixed on the PR?

s172
00:17:36.160 --> 00:17:42.000
And then making sure the approver is someone a apart from the bot that was prompted by the same person.

s173
00:17:42.000 --> 00:17:44.559
And making sure that any overrides are explicitly named.

s174
00:17:44.559 --> 00:17:54.640
So if there is a red build or like a control code coverage skipped, you want to make sure there's a reason for when why that check failed and why we're okay with that in that scenario.

s175
00:17:54.640 --> 00:18:01.840
And making sure that the code that merges actually has all this and it's not like a small change on top that was merged.

s176
00:18:01.840 --> 00:18:04.559
So all the checks are done every single time.

s177
00:18:04.559 --> 00:18:08.160
So in this case we want to make sure that agents cannot approve their own changes.

s178
00:18:08.160 --> 00:18:11.840
Engineer who prompted it is also not an author.

s179
00:18:11.840 --> 00:18:22.160
And if you have these practices in place now we can actually start removing some of these bottlenecks because the CI being faster will unblock your builds.

s180
00:18:22.160 --> 00:18:30.720
So your flow is much nicer and making sure you can like move some checks left will also make sure that your test load is going to be reduced quite a bit.

s181
00:18:30.720 --> 00:18:42.559
On the CI side, if you can automate review for the most part and only escalate to humans for exceptional cases and potentially even auto approve lowrisk or internal changes,

s182
00:18:42.559 --> 00:18:52.080
then you remove further bottlenecks and then you have less that has reaching humans uh for a software factory to perform to its uh potential.

s183
00:18:52.080 --> 00:19:07.760
And of course after every single approval and merge you want to record what happened because as factories are going to ramp up if you don't have an idea of what happened in the past that context is going to be very useful on how to like iterate and make your software better over time.

s184
00:19:07.760 --> 00:19:09.440
Uh that's it for me.

s185
00:19:09.440 --> 00:19:15.200
Uh would love for you folks to also try out guitar and see how you can improve your software factories.

s186
00:19:15.200 --> 00:19:19.600
And with that uh thank you so much for for your