WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=VrpEyglYgeU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/VrpEyglYgeU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.400 --> 00:00:19.440
Please join me in welcoming the chief executive officer at Sonar, Tariq Sha.

s3
00:00:34.719 --> 00:00:36.559
Morning everyone.

s4
00:00:36.559 --> 00:00:37.920
Did you enjoy that last talk?

s5
00:00:37.920 --> 00:00:38.879
That was amazing.

s6
00:00:38.879 --> 00:00:42.480
Um, you particularly love the end the being unreasonable part.

s7
00:00:42.480 --> 00:00:44.079
I thought that was awesome.

s8
00:00:44.079 --> 00:00:51.520
Um, I also want to just I'm trying to calculate the odds of t following tar as the first two sessions in the morning.

s9
00:00:51.520 --> 00:00:56.399
Uh, I think the odds are pretty low on this one, but uh, thrilled to be here today.

s10
00:00:56.399 --> 00:00:59.680
Um, as as we just mentioned, I am with Sonar.

s11
00:00:59.680 --> 00:01:04.720
We are in the code verification space and I'm here today to talk about verification.

s12
00:01:04.720 --> 00:01:11.920
And I think we're all here uh in large part because we believe to some extent that AGI is here.

s13
00:01:11.920 --> 00:01:12.799
It's coming.

s14
00:01:12.799 --> 00:01:14.960
The models we just heard about Fable.

s15
00:01:14.960 --> 00:01:18.880
It's really incredible what is going on in the world today.

s16
00:01:18.880 --> 00:01:23.040
And yet we work almost exclusively with enterprises around the world.

s17
00:01:23.040 --> 00:01:27.439
And the conversation that we have more is the question mark version.

s18
00:01:27.439 --> 00:01:28.880
Is AGI here?

s19
00:01:28.880 --> 00:01:31.040
And why are they asking these questions?

s20
00:01:31.040 --> 00:01:35.200
It's because you can read the news every day.

s21
00:01:35.200 --> 00:01:49.200
And I'm not trying to name and shame here, but if you look at KPMG putting out reports that they have to uh retract because of hallucinations, uh EY doing the same thing, law firms

s22
00:01:49.200 --> 00:01:56.880
getting into lots and lots of trouble because of madeup citations, madeup case law, things like this.

s23
00:01:56.880 --> 00:02:01.600
I think we can really start to question how do we get value out of AI?

s24
00:02:01.600 --> 00:02:09.679
The models are amazing as we just heard, but the hard part as the other target just said is getting value out of it.

s25
00:02:09.679 --> 00:02:14.080
The struggle is that AI slop is everywhere.

s26
00:02:14.080 --> 00:02:16.480
I'm sure you all see this inside of your organizations.

s27
00:02:16.480 --> 00:02:20.000
I'm sure you see this in your everyday life.

s28
00:02:20.000 --> 00:02:21.440
That AI is amazing.

s29
00:02:21.440 --> 00:02:25.520
The models are incredible at generating very plausible output.

s30
00:02:25.520 --> 00:02:29.040
They're incredible at generating things that sound correct.

s31
00:02:29.040 --> 00:02:30.080
But are they correct?

s32
00:02:30.080 --> 00:02:33.280
And how do you know that they're correct is a big problem.

s33
00:02:33.280 --> 00:02:36.560
And it's a big problem in professional services as we saw.

s34
00:02:36.560 --> 00:02:38.160
It's a big problem in legal.

s35
00:02:38.160 --> 00:02:46.879
But really, I think if we're honest, it's it's a big problem in every sector, in every field, whether it's marketing or finance or you name it.

s36
00:02:46.879 --> 00:02:50.319
You have this question of how do you actually know if it's true?

s37
00:02:50.319 --> 00:02:53.599
How do you know if it's good or if it is slop?

s38
00:02:53.599 --> 00:02:59.920
And the question that we we deal in the coding space in particular, we deal with software development.

s39
00:02:59.920 --> 00:03:09.760
And the question we get as we talk to I'm sure many of the people here in the room and a lot of our customers is, isn't software development different?

s40
00:03:09.760 --> 00:03:15.120
And we can look at the data on this and uh the mythos models.

s41
00:03:15.120 --> 00:03:18.000
Um this is data from um meter.

s42
00:03:18.000 --> 00:03:20.000
Uh you may have seen this mer.

s43
00:03:20.000 --> 00:03:23.680
Um the coding agents are getting better uh very quickly.

s44
00:03:23.680 --> 00:03:25.360
They're getting a lot better very quickly.

s45
00:03:25.360 --> 00:03:28.800
And you can see uh the progression, the exponential curve here.

s46
00:03:28.800 --> 00:03:36.000
What this shows on this chart is how capable are the models at completing tasks that humans would take.

s47
00:03:36.000 --> 00:03:40.640
So can they complete a task that takes 1 hour, 2 hours, whatever it is.

s48
00:03:40.640 --> 00:03:49.840
the latest Mythos model, at least per the benchmarking which was done a month or so ago in the preview mode was you're getting to 16 to 18 hours.

s49
00:03:49.840 --> 00:03:59.120
So they're actually able the agents are able to complete longunning tasks and it really is starting to transform how work is happening.

s50
00:03:59.120 --> 00:04:05.120
But the critical caveat when you read the data is this is at a 50% success rate.

s51
00:04:05.120 --> 00:04:05.519
Okay.

s52
00:04:05.519 --> 00:04:13.200
So it is again able to complete tasks but is it able to complete tasks correctly is the question.

s53
00:04:13.200 --> 00:04:19.759
So if you start looking at let's dial up the accuracy rate you dial it up to 80%.

s54
00:04:19.759 --> 00:04:22.800
And there's still progress but it is much slower progress.

s55
00:04:22.800 --> 00:04:27.919
Instead of 18 hours you're at about 3 and a half hours or something along these lines.

s56
00:04:27.919 --> 00:04:31.600
And by the way this is still at 80% accuracy.

s57
00:04:31.600 --> 00:04:41.280
And as I was presenting this to the CTO of one of my uh large customers, his response was, "Butic, I would still put someone who gave me 80% accurate information

s58
00:04:41.280 --> 00:04:43.360
on a performance review probably, right?

s59
00:04:43.360 --> 00:04:46.880
This isn't necessarily enterprise grade.

s60
00:04:46.880 --> 00:04:59.120
The problem is that the models themselves, and full disclosure, we have not yet uh done this benchmarking on the Fable models obviously because they are just being released.

s61
00:04:59.120 --> 00:05:07.360
But as you look at the models, the models are getting smarter, but they still produce a lot of problem problematic code.

s62
00:05:07.360 --> 00:05:08.960
This is benchmarking that we do.

s63
00:05:08.960 --> 00:05:20.240
We give the models a series of over 4,000 problems and we basically ask it to generate the response to the problems and then we analyze both the functional correctness

s64
00:05:20.240 --> 00:05:25.759
which is critical and they all do extremely well on this notion of functional correctness, right?

s65
00:05:25.759 --> 00:05:32.320
Um, but then we look at how complex is the code, how buggy is the code, how secure is the code.

s66
00:05:32.320 --> 00:05:37.600
And what you see with even the state-of-the-art models is that complexity is still high.

s67
00:05:37.600 --> 00:05:40.000
It's actually quite variable as you can see here.

s68
00:05:40.000 --> 00:05:44.960
Um, GPT55 has done particularly well on the complexity side of things.

s69
00:05:44.960 --> 00:05:46.479
It still generates bugs.

s70
00:05:46.479 --> 00:05:53.520
It doesn't generate massive amounts of bugs, but it still generates bugs and it still generates security issues.

s71
00:05:53.520 --> 00:05:59.199
So this is the output of the models that are going into the agentic workflows.

s72
00:05:59.199 --> 00:06:02.880
And again, this is not, you know, I'm at the AI engineer conference.

s73
00:06:02.880 --> 00:06:17.280
This is not me saying AI is fake or or um incorrect, but it is um trying to address this question of how do you really get value in a production setting out of AI?

s74
00:06:17.280 --> 00:06:28.960
This is a study that was done in Carnegie Melon uh University and it looked at what is the actual productivity benefit that you see from the use of AI

s75
00:06:28.960 --> 00:06:30.080
coding agents.

s76
00:06:30.080 --> 00:06:44.000
And what you see I think really resonates with a lot of what I see firsthand in the market which is you have a initial just amazing boost of productivity of velocity in particular.

s77
00:06:44.000 --> 00:06:48.319
what you see is a three to 5x boost in productivity or in in velocity.

s78
00:06:48.319 --> 00:06:51.840
Um that dissipates in three months.

s79
00:06:51.840 --> 00:06:57.840
At the end of three months, it starts to come back to the the normal before you were using the agents.

s80
00:06:57.840 --> 00:07:08.400
And if you ask why, it is because of the two pieces in red here that you start to see there's an increase in velocity, but there's an increase in security issues,

s81
00:07:08.400 --> 00:07:15.520
there's an increase in maintainability issues, there's an increase in reliability issues, and there's an increase in complexity.

s82
00:07:15.520 --> 00:07:22.960
So essentially, you're building the technical debt as quickly as you are generating the code or maybe even more quickly.

s83
00:07:22.960 --> 00:07:25.199
And that creates a different set of work.

s84
00:07:25.199 --> 00:07:28.000
it creates a different bottleneck.

s85
00:07:28.000 --> 00:07:37.039
And so to us, this is now the critical question in AI, which is in a world in which code is provable.

s86
00:07:37.039 --> 00:07:49.280
And there's sessions that um I'm actually very much looking forward to attending about formal methods and proofs and things like this, code is provable, but when you start dealing with large code bases, software is not.

s87
00:07:49.280 --> 00:07:51.280
It's still very complex.

s88
00:07:51.280 --> 00:07:52.479
It is still very messy.

s89
00:07:52.479 --> 00:07:54.960
there's lots of um dependencies.

s90
00:07:54.960 --> 00:07:59.280
There's lots of uh technical debt already in most code bases.

s91
00:07:59.280 --> 00:08:03.520
And so this question of verification is actually key.

s92
00:08:03.520 --> 00:08:12.560
And what I'm going to be arguing is that you can treat verification as an afterthought or you can bake verification into the process.

s93
00:08:12.560 --> 00:08:24.000
And if you bake it into the process of generating code, of doing software development, you can actually start to get materially better outcomes from the coding agents than if you view it as an afterthought.

s94
00:08:24.000 --> 00:08:28.319
If you view it as just the old school code review.

s95
00:08:28.319 --> 00:08:36.959
So as we've been thinking through this, we basically have constructed a framework and there's lots of competing frameworks around this, but I'll just talk you through uh ours.

s96
00:08:36.959 --> 00:08:39.839
We call it the agent centric development cycle.

s97
00:08:39.839 --> 00:08:50.640
for shortand we call it AC/DC sometimes and the idea here is how do you get verification powered to Gentic loops at the center there's a lot of focus on

s98
00:08:50.640 --> 00:09:00.399
the code generation piece like how do you actually get the models and the agents to generate the code that you need to solve the problem and what we argue is that

s99
00:09:00.399 --> 00:09:12.080
you should surround this with the right disciplines the right tools the right processes to do three things to guide the agents and t is talking a lot about different aspects of this actually.

s100
00:09:12.080 --> 00:09:16.720
Guide the agents, verify the outcomes and then solve the problems.

s101
00:09:16.720 --> 00:09:27.519
And you have to make this part of the discipline, part of the process, part of the new software development life cycle if you want to be successful in the AI world.

s102
00:09:27.519 --> 00:09:31.279
So if I double click on some of these pieces, what do we mean by guide?

s103
00:09:31.279 --> 00:09:34.160
We've done a lot of experimenting around guide.

s104
00:09:34.160 --> 00:09:40.000
We've just launched a product um yesterday I think called sonar vortex that starts to get into this area.

s105
00:09:40.000 --> 00:09:53.440
What we find is critically important is to think about guide as context and constraints and we separate out context and constraints very deliberately because context is you have your code repositories.

s106
00:09:53.440 --> 00:10:00.480
How do we make it easier for the agents to understand for the models to understand what is in your codebase?

s107
00:10:00.480 --> 00:10:08.240
If you have a million lines of code, if you have a hundred million lines of code, you have a billion lines of code, the agents work better if they understand your codebase.

s108
00:10:08.240 --> 00:10:10.000
So, how do you give it architectural awareness?

s109
00:10:10.000 --> 00:10:25.440
How do you provide uh semantic navigation uh maps um and uh and help them understand the territory to borrow what Tar was just talking about and we find it equally valuable and I don't think this part is talked enough about to provide

s110
00:10:25.440 --> 00:10:27.200
the constraints as well.

s111
00:10:27.200 --> 00:10:31.440
You have guidelines that you want your code to follow.

s112
00:10:31.440 --> 00:10:35.120
You have dependencies you are okay using.

s113
00:10:35.120 --> 00:10:38.079
You have dependencies that you are not okay having.

s114
00:10:38.079 --> 00:10:39.200
You have coding standards.

s115
00:10:39.200 --> 00:10:40.000
You have guardrails.

s116
00:10:40.000 --> 00:10:41.440
You have intended architecture.

s117
00:10:41.440 --> 00:10:44.880
We spend a lot of time talking about existing architecture.

s118
00:10:44.880 --> 00:10:46.800
But what about where you want to go?

s119
00:10:46.800 --> 00:11:02.800
And so this idea of context and constraints uh we've found in our testing generates a massive improvement in agent effectiveness and a massive uh improvement in token consumption o over 30% reduction

s120
00:11:02.800 --> 00:11:06.000
in tokens being used to solve a given problem.

s121
00:11:06.000 --> 00:11:10.240
And and if you ask why it's because you're actually making the life of the agent easier.

s122
00:11:10.240 --> 00:11:13.279
You're helping it navigate better.

s123
00:11:13.279 --> 00:11:18.160
So then we get into the heart of this and we really think of guide as preemptive verification.

s124
00:11:18.160 --> 00:11:21.600
How do you make sure there's less to verify, less to fix, this sort of thing.

s125
00:11:21.600 --> 00:11:33.600
Then you get to the heart of verification and what we believe quite strongly and what we've seen work in practice is this idea of zero trust multi-layered verification.

s126
00:11:33.600 --> 00:11:34.800
Zero trust.

s127
00:11:34.800 --> 00:11:36.240
Every model has biases.

s128
00:11:36.240 --> 00:11:39.760
Every model produces has a character has a personality.

s129
00:11:39.760 --> 00:11:46.800
So, let's make sure we use different models and different techniques to make sure your code is safe, to make sure it's reliable, to make sure it's secure.

s130
00:11:46.800 --> 00:11:53.040
And multi-layered really speaks to the earlier point that software is complex.

s131
00:11:53.040 --> 00:11:54.800
Software is very messy.

s132
00:11:54.800 --> 00:11:59.279
Software has lots of of of intricacies involved with it.

s133
00:11:59.279 --> 00:12:13.279
And so what we believe and again have found to be quite um impactful here is that a combination of algorithmic verification looking at things like data flows, control flows, known patterns, secrets,

s134
00:12:13.279 --> 00:12:22.639
these areas combined with what is now possible with agentic verification looking at intent, business logic, the unknown unknowns.

s135
00:12:22.639 --> 00:12:36.800
Actually again to borrow from the last uh presentation the fusion of these things the the deliberate multi-layered fabric that you put in place can actually you can see the results of this in production.

s136
00:12:36.800 --> 00:12:51.040
So as we look at our partners and customers who use a multi-layered verification approach they are reporting AI derived production outages being 44% less frequent than the ones who do not.

s137
00:12:51.040 --> 00:12:58.240
So you can start seeing a material improvement in reliability, in security and in maintainability.

s138
00:12:58.240 --> 00:13:03.279
And then the last point I mentioned is technical debt does explode.

s139
00:13:03.279 --> 00:13:03.519
Right?

s140
00:13:03.519 --> 00:13:07.760
As you generate code, technical debt is also generated.

s141
00:13:07.760 --> 00:13:10.800
And again, this is not stop doing it.

s142
00:13:10.800 --> 00:13:14.320
This is be aware and let's start controlling it.

s143
00:13:14.320 --> 00:13:28.480
And so what we um have seen be super effective is to have an active process to have an active discipline again around code maintenance and thinking about how you do verified

s144
00:13:28.480 --> 00:13:29.760
code maintenance.

s145
00:13:29.760 --> 00:13:42.800
Um I won't walk through every step of this but a the agents whether that is a set of remediation agents whether it's a strong discipline around verification does keep your codebase clean

s146
00:13:42.800 --> 00:13:56.320
and a lot of people have asked me all right but do agents care about clean code human developers care about clean code do agents care about clean code and what we find again is they absolutely do because the agents have to

s147
00:13:56.320 --> 00:14:00.959
understand the codebase if they're going to operate on it so this is a oneshot view.

s148
00:14:00.959 --> 00:14:03.120
Um, we think this is something that compounds.

s149
00:14:03.120 --> 00:14:13.920
But if you just do the exact same agentic tasks on a typical codebase and then one that has been cleaned, you see a material reduction in the amount of tokens,

s150
00:14:13.920 --> 00:14:21.360
reasoning, energy, etc. needed for those cleaner uh code bases versus the typical code bases.

s151
00:14:21.360 --> 00:14:21.600
Right?

s152
00:14:21.600 --> 00:14:29.440
If you make the life of the of the agent easier, if you maintain your codebase, then you'll actually see compounding effects.

s153
00:14:29.440 --> 00:14:33.839
Now, the important thing in our mind is to construct the system.

s154
00:14:33.839 --> 00:14:43.839
This is how I started is saying, you know, I'm sure all of us do code reviews, you may use static analysis tools, you may use AI code uh review tools, a whole range of things.

s155
00:14:43.839 --> 00:14:46.560
And we believe that you have to put this in a system.

s156
00:14:46.560 --> 00:14:58.959
And again, uh, we're happy to in our booth downstairs talk through what this looks like, but we really believe that the construction of the software development life cycle in an AI world

s157
00:14:58.959 --> 00:15:03.839
um, needs to embed this notion of guide, verify, and solve inside of it.

s158
00:15:03.839 --> 00:15:05.519
And you need to do it in three loops.

s159
00:15:05.519 --> 00:15:07.440
And you need to think about these three loops.

s160
00:15:07.440 --> 00:15:11.120
There's the agentic loop, which I think is the key buzzword of the conference.

s161
00:15:11.120 --> 00:15:23.920
Um now but how do you provide the agents as it's generating the code as it's doing the work with the context and constraints with the inloop verification so that the agent is getting verification

s162
00:15:23.920 --> 00:15:32.320
as it's working and how do you fix problems that's that's the blue loop here what we what we talk about is the inner loop verification piece.

s163
00:15:32.320 --> 00:15:54.000
There's a second which is your continuous improvement process and how do you really combine the power of algorithmic and agentic to generate your your pull request review the code and by the way the velocity of this has to go up massively so to review the code using agents and to do this multi-layered verification

s164
00:15:54.000 --> 00:16:06.399
and then you have your evals and I think the opening speaker talked about how eval may be the buzzword of the um conference you have your EV valves and you have your quality gates to check are you actually passing.

s165
00:16:06.399 --> 00:16:20.399
So you have your your code maintenance loop, agentic loop, CI verification loop and deliberate design of these loops with verification at the center is a compounding system.

s166
00:16:20.399 --> 00:16:27.360
It's a system that reinforces itself and it reinforces itself in the positive and it reinforces itself in the negative.

s167
00:16:27.360 --> 00:16:41.519
And we've seen customers who uh have kind of neglected as they've rolled out AI coding tools, they've neglected verification, they've neglected this idea of code quality, of code um maintenance,

s168
00:16:41.519 --> 00:16:44.959
things like that, and you get into a downward spiral pretty quickly.

s169
00:16:44.959 --> 00:16:55.759
This is what the Carnegie Melon uh case study uh or study actually shows is that you actually have all the benefits start to dissipate or you can get into the self-reinforcing

s170
00:16:55.759 --> 00:16:56.399
loop.

s171
00:16:56.399 --> 00:17:05.600
And one of the tests we did with one of the large banks who are using some of the cutting edge the folks who are all around here today um cutting edge agentic

s172
00:17:05.600 --> 00:17:15.520
coding tools they can get a 92% reduction in issues if you actually take this guide verify solve approach inside of those agentic loops.

s173
00:17:15.520 --> 00:17:20.880
If again this compounds it's not that each loop is 92% better.

s174
00:17:20.880 --> 00:17:29.120
that as you go through solving the problem over minutes and hours that you actually see a compounding benefits.

s175
00:17:29.120 --> 00:17:34.320
So that is uh essentially how we see the benefit here.

s176
00:17:34.320 --> 00:17:49.360
The how we see the controlled um value creating use of AI in enterprise settings and when I say enterprises people with existing code bases people with with you know millions of lines of code already.

s177
00:17:49.360 --> 00:17:55.520
There's the agentic loop, there's a CI verification loop, there's the code maintenance loop.

s178
00:17:55.520 --> 00:17:59.679
I'm required by my marketing team to put up a version of this that has our products on here.

s179
00:17:59.679 --> 00:18:02.160
So these are our products and you can come and see us later.

s180
00:18:02.160 --> 00:18:10.480
But the most important thing is really to say our recommendation is this agent the AC/DC agentcentric development cycle.

s181
00:18:10.480 --> 00:18:15.039
The core part is deliberate verification built into the system.

s182
00:18:15.039 --> 00:18:17.760
So if you'd like to learn more um we have a booth.

s183
00:18:17.760 --> 00:18:19.280
It's the big red booth downstairs.

s184
00:18:19.280 --> 00:18:20.559
We'd love to talk more.

s185
00:18:20.559 --> 00:18:23.280
We have some doubleclick sessions coming up.

s186
00:18:23.280 --> 00:18:26.720
So, please do uh join those and uh have a great conference.

s187
00:18:26.720 --> 00:18:28.960
Thank you all.

s188
00:18:51.198 --> 00:18:53.198
[music]