Guide, Verify, Solve — Anirban Chatterjee, Sonar

AI Engineer · 22 min · 245 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13All right.
00:14[applause]
00:17Thank you.
00:18That's very helpful.
00:18My name is Anirban Chatterjee.
00:19I do product marketing at Sonar.
00:21I'm really excited to be talking to this group today.
00:23It's actually my first time here at this conference.
00:26Um and so I've been having a blast uh along with my team here uh meeting a whole bunch of AI engineers as well as leaders and you know, influencers
00:37and founders.
00:39Uh there's a lot going on in this space.
00:40I think this year there's really been a turning point from experimentation to engineering.
00:46And that makes me that warms my heart very deeply because I started my career many many many many many years ago as a software engineer uh writing code for for servers, if you can believe it.
00:57Um and I think there's a turning point that's happening right now where we're starting to add the capabilities that we need to add to these systems in order to make them repeatable, make them scalable, make them consistent
01:11much in the way we were doing with cloud computing not too long ago in order to expand the access that IT technology gave to small businesses and other innovators.
01:21I think AI is going to do the same thing for software development going forward.
01:24But in order to do that, in order to get there, we need to start adding safety and trust to these systems so that they can be used more widely
01:31across a wide variety of use cases so we can build new things and solve bigger problems.
01:36And how we get there is what we're going to talk about today.
01:39And uh for those of you who were in uh Tarik's keynote yesterday, uh he presented some of this data and I'm going to talk about it a little bit deeper today.
01:46So there was a study that Carnegie Mellon did um where they actually looked at projects that were posted on GitHub.
01:53And they were able to use the metadata to support to sort them into projects where uh there were just traditional tools that were being used and projects where an AI tool was used to write the code.
02:02In this case, it was Cursor, although it could have been any AI tool.
02:05And what they found was interesting.
02:07They found that there was, in fact, a temporary spike in productivity, um but it lasted about 3 months and then it went back down.
02:16And the reason for that, we think, is because there was also a persistent increase in static analysis warnings and code complexity.
02:22They're actually using SonarQube uh to actually collect the data on this and they saw that there was a persistent increase in these types of issues that went beyond the 3-month mark and persisted well into the future.
02:33And so, it's these types of issues that end up actually slowing developers down even more and and making And this is what makes it a challenge to deliver high-quality code
02:43using AI tools.
02:46The reason for this is that there's a differing need for quality depending on the criticality of the application, right?
02:51If you're experimenting, if you're playing around, if you're just one person building things to see what's possible, um if it's an internal non-critical application which has a few users, maybe it's just you, maybe it's a small team,
03:02maybe it's just a short-lived project that's not going to last very long, the gap between the quality that you're getting from the AI tool and the quality you need from the application is quite small, right?
03:11And so, you can And you can live with uh that gap.
03:14But as you move to higher levels of criticality, as you run into situations where you're supporting many, many users, it's a larger code base with many lines of code and many changes happening
03:24across that code base all the time.
03:27Uh you have many, many users.
03:28Some of them could be adversaries that are actively trying to break your software, right?
03:32And so, in those cases, the quality level you need is quite a bit higher than the quality level you're getting by default from these AI tools.
03:39And that's where this verification debt comes in.
03:42That's where you have to bring the humans in, bring your software engineers in to try to close that gap and make sure that the quality level is is up to an acceptable level before you ship that code
03:54into production.
03:55So, why is this happening?
03:56Why is this gap actually occurring?
03:57We know these models are excellent.
03:58They're getting better and better all the time.
04:00I'm really excited to start playing with Fable now that that's out to see what levels of of code we can get out of Fable going forward.
04:08But, we do know that because of the technology, because of the way that these models are built, they will still make mistakes.
04:13They will still have quality issues.
04:15They are still somewhat error-prone.
04:17And if you let these errors go into production code, you could have a catastrophic effect to your organization.
04:23They're also missing context, right?
04:24They only know what you tell it.
04:26They don't know the broader things that are happening elsewhere in the code base.
04:29They don't know what's happening with your business.
04:31They don't know what happened in the meeting you had with somebody else 2 weeks ago that's going to influence the code you're writing today.
04:36They don't have all of the context that you have as an engineer, right?
04:40And so, they don't always know your objectives the way you do.
04:42And that is going to also cause gaps between what you need from the software and the way it's built.
04:51We also know that models are diverse.
04:53No two models are the same and they have diverse quality issues.
04:55And we actually want to explore this.
04:57And so, we actually have a leaderboard that you can go to on our website right now.
05:00It's called the LLM leaderboard.
05:02And what we do is we take all of the major new models that come out and we evaluate them.
05:06We give them, you know, 4,000 or so coding tasks and we evaluate them using all of the metrics that SonarCube uses to evaluate code, right?
05:13We look at their correctness, complexity, the way the rate at which they're solving the tasks we assign them, and then our classic things maintainability, reliability, and security.
05:22And we're able to kind of graph all of these models across these different axes and show you where models perform well and where they have room to improve.
05:29And what you're looking at on the screen right now is actually Claude Opus 4.6 and Claude Sonnet 4.6. If you're a Claude customer, you might be toggling between these two models to control your token burn rates.
05:39And you'll see that Claude Sonnet is actually quite good from a correctness standpoint, from from solving tasks, and from a standpoint.
05:47But if you're requiring higher levels of maintainability or higher levels of security, if you're trying to get a lower complexity out of your code, you might benefit from switching to Opus for tasks like that.
05:56And so we run these kinds of analyses across a lot of different models, and you're always able to go to our website to get the latest analyses that we run.
06:03I think we're actually doing the latest Claude and and OpenAI models pretty soon.
06:07But this kind of data is helpful, right?
06:09Because it tells you what where models are good and where models are not good.
06:12It also serves to put some sunlight on the fact that you still need to be vigilant with these models, right?
06:17None of these models are ever going to be perfect.
06:19You're always going to have some kind of need for verification in the loop to make sure that the code that you're getting is the code you actually want to ship.
06:27Now, classically, that verification can be human verification.
06:30It can be you.
06:31Your own eyes reading the code, your own intellect reviewing the code to make sure that it's successful.
06:36But we know, based on experience and now based on research, that human review can also be compromised.
06:41This is a study that was done earlier this year by Wharton, and they actually gave quite a lot of human participants tasks to complete, and they gave those human participants
06:50the use of an AI tool to complete those tasks.
06:53But unbeknownst to those participants, the AI was told to confidently lie to these participants some of the time.
07:01And what they found in the data is that while participants did follow the AI advice 92.7% of the time when the AI was correct, they unfortunately also listened to the AI
07:10nearly 80% of the time when the AI was wrong.
07:13This is almost surely happening in code review as well, right?
07:17Especially when there's higher amounts of code being written, when there's multiple agents writing code simultaneously, when you now have to bring all those pieces together into a single software application,
07:26the load is just too great.
07:27There's there's only so many hours in the day, right?
07:29And you still have to ship something.
07:31And so there's a lot of rubber stamping that I'm sure is happening in all of your organizations.
07:35It's happening everywhere.
07:36And so we need to backstop that somehow with an automated verification tool.
07:41What can we do about it, right?
07:42As Tarek was talking about yesterday, you know, I think all of us sort of when we got involved with software, one of the things that we find most attractive about it is that code
07:53is is quite, you know, once you write code properly, it's going to run the same way every single time and there's a certain level of comfort in that, right?
07:59There's a certain level of comfort in knowing that if I write this function the right way, it is going to work this way every single time.
08:06And there's there's a clarity that comes to that and there's there's a certain confidence you get out of being able to build something that you know is going to work well for every user going forward, but
08:16we also know that code's written by humans, humans have requirements, and those requirements and externalities have impact in how this code functions.
08:24And as you add more and more and more code to the application, they can interact in unpredictable ways.
08:29As you now allow users to use those applications, those users can do all kinds of things you didn't expect, and so software is not provable in the same way that code is provable.
08:37Software can break in interesting and novel ways, and as you're using AI to write more and more software to solve bigger and bigger problems, um you're going to run into these limitations more and more often.
08:49And so having automatic verification as part of this process is an important part of the solution.
08:55It's going to help you control some of the risks that you're introducing by maybe releasing some of the control you have over the code that's actually being written.
09:04And so we believe that verification is going to be a key enabler and a and a key unblocker for all of the amazing things that we're going to be able to achieve with with the AI-driven software development going forward.
09:16And so when we say verification, what do we mean, right?
09:18We think there's two core elements to successful automated verification when it comes to AI coding.
09:23One is that needs to be zero trust.
09:26What do we mean by that?
09:27Zero trust in this context basically means that the code could really have come from anywhere.
09:31It could still be written by a human, it could be written by an AI as I just showed you a few slides ago.
09:35Different AIs will write code in different ways, and you're not going to want to be able You're not going to want to use that same AI to to validate the code because
09:44you're going to want a diversity of of tools being used to make sure that you're catching all the different issues that can happen.
09:50And so, no matter where the code is coming from, you want to have a a similar comprehensive regime to verify that code that works the same no matter how that code was written, right?
10:00Use a different methodology to review the code that was used to write the code.
10:04Uh it's completely auditable, completely explainable, so you can prove that verification was run the same way every single time, and it's algorithmic and repeatable and consistent no matter how you run it.
10:13It also needs to be multi-layered.
10:15You need to have multiple ways uh a multiple techniques being used, multiple approaches being used to review the code that is being generated, right?
10:23Um because you're never going to be able to find every single problem that can occur in software by just using one or two methods.
10:29You need to use computational review, you also need to use LLM driven review, and everything else in between, right?
10:38You heard a little bit about agentic I'm I'm really I've been hearing a lot about agentic loops this week, uh and Tarik uh talked yesterday about our framework for agentic loops.
10:46We call it ACDC, or agentic agent-centric development cycle.
10:49And there's three phases in in the ACDC that we that we like to talk about.
10:53The the easiest one By easiest, I mean the the fastest one to implement now, the one that many of you are probably already on a path to implementing, is the verification step, which is front which is top center, right?
11:03This is the most important piece that allows you to to write code in these agentic loops in a way that is going to be easily shippable.
11:11Uh it needs to be multi-layered, it needs to be reasoning based, and it needs to cut across quality issues, security issues, and compliance issues to make sure that you're shipping quality that you can stand behind, right?
11:22Um before the verification step, there's a guidance step.
11:25And what guide allows you to do is provide guardrails and context and constraints to make sure that the agent has everything it needs up front to write better code
11:34uh uh uh the first time, right?
11:36And then finally, after verification, you need to solve the issues that come up, right?
11:39And that's where the solve state come.
11:41That's where you can remediate any issues that are found in the code.
11:44You Hopefully, you're allowing the agent to have the agency to do so itself by providing access to the tools it needs to to find the issues and fix them itself and then just repeat the loop.
11:53And so these these agentic loops with verification at the core is how you can get to shipping quality software using AI agents.
12:01And there are prior there's different reasons why you would want to do this.
12:05And when we talk to customers, and we've talked to a lot of customers about this, uh the driving functions that are forcing them to adopt verification across all of their AI coding
12:14processes are are very similar, right?
12:17They want to make sure that AI code is verified consistently.
12:19They don't want to have different methods of verification applying to different projects or different teams.
12:24They want to have a standard rulebook that applies everywhere, no matter what tool is being used.
12:28They all also want to make sure they're using their AI tools effectively, right?
12:32Some of that a big part of this is token efficiency or just efficiency in general, but also just making sure that the tools are being used for the things they're being designed to do in ways that we know they're good at doing, right?
12:42So we talk a lot about token efficiency, and we talk a lot about making sure that the right models are used for the right projects, and so on.
12:48Finally, third, catching issues from a security standpoint as early as possible in development cycle.
12:54Shifting left on security issues has been very important for a number of years now, and now that AI is writing more and more code, catching security issues up front is extremely critical, especially now that we're in a world where CVEs
13:06are announced and then immediately exploited uh by bad actors almost often the same day.
13:12And so you need to make sure that your code is as hardened as possible from those types of issues creeping into production.
13:17And finally, maintaining compliance.
13:19Many of you, I'm sure, work in a regulated industry, uh and for those types of situations where you need to be able to prove that verification is run constantly
13:26and consistently across the board.
13:28Maintaining an audit trail that allows you to prove that is extremely important.
13:32We have a number of solutions that that help with that.
13:35SonarQube has been around for quite a while.
13:37There are probably quite a few of you that are already SonarQube users.
13:40It is a zero trust multi-layered verification platform that works across syntax issues, data flow issues, architectural issues, control flow issues.
13:46And it works across basically any language you would be using.
13:50We have a lot of deep hooks that I'm going to take you through in a moment that allow agents to have first party access to the SonarQube verification so they can more effectively write high quality code.
14:00And we also just recently, and by recently I mean just a few weeks ago, acquired a company called Gitarr, based right here in San Mateo.
14:06And they do AI code review.
14:09And more than that, they actually build a full they build and automate the full CI workflow so that you can not only find issues using an using our LM
14:20approach, but you can also automatically block if those issues cause a quality issue that you would want to push forward.
14:26It can write fixes and it can approve those fixes and merge those PRs completely automatically if you wanted to.
14:31Now, we'd want to earn that trust.
14:32It doesn't happen that way by default usually.
14:35By default they'll just find the issues and show them to you and enter a dialogue with you so you can have those issues fixed.
14:40But as you use it more and more and gain confidence, you can turn on more and more features and completely automate the PR review workflow if you like using Gitarr.
14:50So, the other big news from earlier this week, and Tarik alluded to this during his talk yesterday, is that we also announced a new agentic loop capability with a product called Sonar Vortex.
15:00And Sonar Vortex, I'm going to I'm going to show another flowchart that shows what it does, but basically it is providing your agents with tools in the inner loop, in the agentic loop, to not only get constraints and guardrails up front to write better code, but also run verification
15:15as it's writing code in real time so it can find and fix the issues that are being created.
15:19And then we also released the remediation agent.
15:21And the remediation agent allows you to tackle backlog issues and and take down your tech debt at a scale that you might not have the bandwidth to do now with human developers, right?
15:30You can basically take your your tech debt or your older issues, your legacy code, point them at remediation agent, and it can then improve your code base almost in the background
15:39while you focus on the on the innovation work at the front end that you're working on now.
15:44And those are both GA as of this week.
15:46Now, there was a very detailed chart that was shown during the keynote yesterday that I'm going to kind of break down for you and really explain what the different pieces of this chart mean.
15:56This is kind of how we see the ACDC applying not only to the inner agentic loops, but also the outer CICD loops that we're all working in to develop code.
16:06Before you start either of those loops though, it is really important to have a sense of the specifications of what you actually are going to want to accomplish with the software that's being written.
16:17This is This is your architectural constraints.
16:19This is the way your desired architecture for the software.
16:22This is your coding standards and your coding patterns that are acceptable.
16:25These are the list of dependencies that you are and are not allowed to use.
16:29These are your coding standards and and syntax standards that you obey in your organization.
16:33This could be your your logging practices or your observability and tracing practices.
16:38All of that goes into your specs and then you also need to define what your quality criteria are, right?
16:44And we have quality criteria that we ship with that you can use by default or you can adjust them as it makes sense for your organization, but this is like what are the levels of security, of quality, of maintainability that you're willing to accept
16:55in your code that you're pushing into production.
16:57You need to basically write that down and encode it.
16:59And now you're ready to start using LLM coding agents to write code.
17:02And whenever you're initiating a coding task with an with an with an agent, one of the first things that we can help is providing context and constraints so that the agent starts
17:14from the ground floor with an understanding of the code base and the guardrails that is relevant to it in that moment.
17:19You have to manage the context window of the agent you can't just throw your entire code base at the agent up front.
17:24It's going to spend a lot of time thrashing and exploring and burning tokens while it's doing it.
17:28We are able to efficiently provide just the context that it needs based on the work that it's being given so that it can get to work writing productive code very quickly.
17:37It then generates the source code as you can see and we also provide in loop verification to the agent as it's writing.
17:44It can call in to us and provide and get a list of issues that we are finding in real time in the code that's being written and the great thing about that is those issues can then be fixed immediately by the agent so they don't propagate
17:54into future agentic loops that are going to run in order to fully build out the software project that you're doing.
17:59This is being enabled by Sonar Vortex as of this week.
18:03Once all of the inner loops have run you get you reach a point later on when you have to start entering the formal review and shipping process for the code and this is the CICD process, right?
18:13And so there's a PR flow that gets initiated that I'm sure we're all familiar with.
18:17Guitar can live in that flow, SonarQube also lives in that flow in order to run a broad automated review of all of the code that is in the PR
18:25and an automated verification that actually returns issues across quality, security and maintainability.
18:31There's a superhuman review that is LLM driven by Guitar and there's a computational review that is run by SonarQube that actually assigns grades for all three of those things and won't allow the PR to go past into production unless it gets a passing grade across that criteria.
18:45So if there are issues that come up, you can actually use a fix agent to fix all the issues that are discovered there and then once you're actually able to pass that quality gate, that is when you're able to proceed in the test
18:55and build and deploy in the application.
18:58The verification needs to run in both the inner agentic loop and also in the outer loop for CICD.
19:05I'm running low on time so I'm going to hope this video completes.
19:08This is basically a video demo of the inner loop, the agentic loop.
19:11It's a it's a demo of Sonar Vortex, basically.
19:13And what you've just seen happen is we've given it a task and in starting that task, this is this is actually Cursor that's running right now.
19:20Cursor called into our Sonar Vortex context tool to get some context up front to to get give it an understanding of the code that's working and so it knows
19:29how to um how to write the code.
19:31It is now writing the code and once it's completed uh the initial write, it's going to call into our verification process to get a list of issues that it finds.
19:40And then if it and if issues get provided, it will actually fix those issues immediately in the inner loop.
19:46This is all happening automatically through an integration that we have directly with Cursor.
19:50We have similar integrations with uh with Claude Code, with Codex, with Any Gravity, with basically any major AI coding tool that you would have.
19:59And you can sort of get the gist.
20:00It It flagged an issue.
20:01It's got a plan.
20:02It's going to fix it immediately and it's going to run uh the analysis again and then it will not proceed until it actually is able to get a passing grade from us on the verification pass.
20:11I'm going to move past this cuz I'm out of time now.
20:16So, uh if you take one thing away from this presentation, it's that, you know, we believe very strongly, we're very convicted about this, that a governance and verification regime is a engine is extremely critical
20:27to unlock the next level of success that we need to be able to get from AI coding tools so we can solve bigger and bigger problems.
20:35And we know, based on our data, uh that Sonar customers and Sonar users are able to get higher levels of success from AI coding tools.
20:42I can I'm not going to go through all of these now cuz I'm out of time, but you can stop by the our big red booth downstairs and we'll be happy to talk to you about any of these.
20:49Um but, you know, the good news is I think many of you probably have access to this stuff already.
20:55Uh Sonar Cube is one of the most widely adopted verification tools in existence today.
20:58We have over 7 million developers around the world using us and we analyze close to 750 billion lines of code across our solutions every single day.
21:06And if there are a few of you in the room who care about Gartner at all, it's nice to know that we're a Gartner market uh magic quadrant leader as well.
21:12So, uh final slide, key takeaways, right?
21:15What are the things that we want to walk away from this?
21:17First, you know, establish some bounded autonomy guidelines for your AI agents.
21:22Give them the freedom to generate code, but also make sure that you're enforcing a centralized scheme of verification and constraints.
21:28Implement the ACDC, right?
21:30Give them context.
21:31Verify that they're doing the right work with independent metrics, and then use agents to solve their own mistakes.
21:36Empower them to do that.
21:38Make sure your developers are equipped with the orchestration tools that they need in order to design the context frameworks and the processes that are going to enable them to be successful and use AI most effectively.
21:48And finally, standardize on a single independent multi-layered verification platform that is used consistently across all projects, across all teams, across all developers, and all AI coding tools so that you can eliminate any blind spots that might be created by using silos of tools across the organization.
22:04Um that's all for me.
22:06Uh you can learn more by Oh, it's off the screen now.
22:09But uh you can come down to our big red booth downstairs, and we'd be happy to talk to you.
22:12Thank you very much.
22:28[music]