AI Writes More PRs. Who Validates Them? — Ali-Reza Adl-Tabatabai, Sonar

AI Engineer · 11 min · 90 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:13Let's get started.
00:14Um, good afternoon everyone.
00:16I'm Ali and um, I'm one of the founders of guitar.ai AI and previously the CEO up until about a month ago where we were acquired by Sonar and now we are part of Sonar
00:32and you've heard a lot in the past few days about the importance of verification in the age of AI and so today I'm going to focus my talk on code and verifying code.
00:44So in in almost all software teams there is this centralized validation phase that is basically CI plus code review and as you scale your engineering this plays this phase plays a very critical part
00:59especially in an AI world.
01:01Uh first it enforces all your quality gates centrally.
01:05So these are security compliance code quality gates.
01:10It's on the critical path of getting your changes into production.
01:14So the faster is always the better.
01:16But as you scale engineering, this becomes kind of a challenge.
01:20As as you have longer running builds and tasks, you have asynchronous team review flows across sides and geographies and things just get slower naturally as you grow.
01:32Um, this is also a phase where you end up spending a lot of developer infrastructure dollars.
01:38So there's a lot of expensive tools here that need to be integrated and scaled up as you um scale your organization and it all runs on very expensive infrastructure.
01:50Uh for these reasons it's often times the main focus of your platform organization team or your platform team.
01:58Um but any kind of failure in this phase is very expensive.
02:03It introduces failures introduce an outer loop that are typically measured in hours and days.
02:11So failures are expensive.
02:13They costly.
02:13They really slow down your velocity.
02:16Uh you lose developer time and that's compounded by disruptions to your flow state.
02:23It's also very expensive to optimize.
02:26So you've got a lot of fragmented systems that have been integrated together with bespoke configuration and scripting.
02:32and you have a usually headcount restricted, constrained platform organization trying to keep it all up and running for you.
02:39And trying to address that all with AI is also kind of complex because it requires complex workflow orchestration involving various inter asynchronous interaction, team workflows, integrations across tools and so on.
02:56In fact, AI kind of makes the problem even worse.
03:00uh you've got a lot of PRs now being generated, more PRs than before uh thanks to AI and bigger PRs uh all of which can have defects and so your costs in your development life cycle are kind of shifting to the right.
03:14You have a lot more code to review.
03:16You can have potential for a lot more failures in production and and so your choices are kind of tough here.
03:22either you slow down and ask developers to review every PR very carefully or you rubber stamp PRs and you risk incidents in production.
03:31Either way, you've got less happy developers, lower sentiment, and much less productivity than than you'd expect.
03:40So, what's the answer here?
03:43Of course, it's more AI.
03:44It's uh an agentic approach to validation which is essentially what we built at guitar um that we're now also shipping uh as part of sonar.
03:56And basically what we've got here is an agent that fully automates all the validation steps and as outcomes it delivers for you green PRs that are ready to merge or merge automatically.
04:08Now how does this work?
04:10Well first when you create your PR it reviews your PR and posts issues and inline comments.
04:14No surprise there.
04:16But our goal has been to really make this experience as noisefree as possible, both in the way we handle the UX as well as posting issues that are real and really matter.
04:28You can customize your review with your own rules and you can add your own custom checks and you can trigger your own custom workflow automations.
04:37And then you can also configure the system to block merging if there are any unresolved code review findings.
04:44It also analyzes CI failures and posts a summary of the root causes of those failures.
04:50And in particular, it's really good at catching flaky tests.
04:54And you can even set up rules that automatically retry flaky tests if you want.
04:59That's a very popular feature.
05:01You can also automatically fix any code review issues or any CI failures.
05:06Any kind of uh failures that happen in CI, it can fix it for you as well as address any of the issues that it raises.
05:13You can either ask it to specifically fix an issue or more interestingly you can set it up to automatically loop until all the fish issues have been fixed in your PR and you have a green PR ready to merge.
05:26And finally, you can automatically approve and merge PRs that are green.
05:32you can define rules and conditions that control when uh PRs get automatically approved by the agent and merged automatically.
05:45So we're seeing actually a very interesting trend in our users.
05:49There's a clear trajectory towards fully automated PRs.
05:53uh as users use the system and as they build trust in the accuracy of the AI what we see is they work towards full automation first they get reviews
06:04and they're happy with they build trust in the accuracy of the reviews they see that these reviews are not only accurate but they're raising really important issues so then they give teeth to the agent by allowing it to start blocking
06:17the PRs and then they start using the autofix capabilities to automatically turn PRs green and fix all the issues that were raised by code review or any CI failures that come up.
06:29And then where the real interesting leap is beginning to happen now is users setting up rules that define when the agent automatically approves and merges PRs.
06:41So over time as they build trust in the system they unlock more levels of automation and with the additional levels of automation and with the trust they're getting more value out of the system.
06:54Now because the system processes every single PR using AI it's also able to uncover new types of insights.
07:02For example, uh one of the insights that are is very useful to platform teams is a categorization of the top CI failures.
07:12Where are you seeing failures in your CI system where your developer is running into failures?
07:18Are these flakiness issues that you can that you need to address by improving the reliability of the tests or are there infrastructure issues that you need to address or are there other kinds of failures that they're running into?
07:30Very very useful for platform teams.
07:32We also have another example of what what else we have in terms of AI enabled insights are categorization of PRs.
07:41What exactly is happening across your engineering team in terms of the PRs that are being landed?
07:46This is extremely useful for leadership teams.
07:50For example, what amount of PRs are going to feature development versus fixing issues, chores, refactors, and so on.
07:59Now under the hood, if you look at the system, there's a number of components that constitute guitar.
08:05There are three interesting pieces here.
08:07One is the control plane that acts as this orchestration layer for handling PR workflows.
08:14And then there's an agent runtime or or harness.
08:17We built our own harness here that handles multi- aent execution, context management, memory management, um tool calling, all the integrations, etc. And having this having our own agent harness
08:31has really allowed us to optimize specifically for the validation use case and it's allowed us to optimize for token cost, precision and coverage of the agent itself.
08:43So that allows us to essentially guarantee outcomes at a fixed PR price while giving really good quality results.
08:52We also have an NLM proxy that handles the routing across different models as well as things like failover.
09:00Now that we're part of Sonar, we've been integrating guitar with sonar cube.
09:06There's a ton of opportunity here in combining an agentic approach to validation with program analysis.
09:13So combining agents with things like u taint analysis, control flow analysis, data flow analysis, software composition analysis.
09:24So all the traditional approaches combined with agents.
09:27We think there's a ton of opportunity here and that together you can get much better precision, much better coverage at a really great price point.
09:37So the overall results uh the value that we're seeing in in all of this is much better quality, much better security, especially with AI generated code, faster delivery times through all the automation
09:50going from PRs straight to approved and merged PRs, much higher productivity and sentiment by developers.
09:57So less time spent on actually reviewing code and worrying about what's happening in production with more confidence as well as sentiment because now you much better sentiment now because you have a lot of automation in place and less grunt work.
10:10And finally the to the features that we've built for platform teams gives them a lot more leverage to be able to fix issues and to add customizations and see insights
10:21in this critical part of the software development life cycle.
10:26To wrap up, PR validation is really critical, even more so in AI, but AI turns validation into a friction point.
10:37And guitar, what we've built, allows you to automate this step, fully automate PR validation, go from created PRs all the way to merge, completely eliminating this friction and helps you keep up with the rate at which you're generating code from AI.
10:53And finally, the combination of using agents uh doing a gentic validation plus an algorithmic program analysis will give you something that that I think is going to be much better than either alone.
11:07If you want to talk more and you want to see more, if you'd like to see a demo, come on and visit us at booth number P7 right down
11:14right down the hall there.
11:15Thank you.