WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=LQW_Wvqhp04

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/LQW_Wvqhp04.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.040 --> 00:00:14.680
Uh I My name is Tariq Shakir.

s3
00:00:14.680 --> 00:00:17.040
I am the CEO of Sona, as you just heard.

s4
00:00:17.040 --> 00:00:26.200
And we're here to talk today about how we have been um helping companies build software factories that actually work for enterprises.

s5
00:00:26.200 --> 00:00:33.840
And I thought it'd be an interesting way to start to start looking at actual factories cuz we know how to build actual factories.

s6
00:00:33.840 --> 00:00:40.960
And for those of you who have a little bit of the gray hair and all of this sort of thing, you may uh remember the 1980s.

s7
00:00:40.960 --> 00:00:45.040
And in the 1980s, you could find cars that were really good cars.

s8
00:00:45.040 --> 00:00:51.240
But if you look at the picture here and if you remember those times, the cars like didn't quite fit together perfectly.

s9
00:00:51.240 --> 00:00:51.520
Right?

s10
00:00:51.520 --> 00:00:55.280
You'd have different pieces that would leave big gaps.

s11
00:00:55.280 --> 00:00:59.920
It would be really good cars that just didn't kind of work the way they were supposed to.

s12
00:00:59.920 --> 00:01:04.600
And then uh Toyota and particularly Lexus came out with a commercial.

s13
00:01:04.600 --> 00:01:10.880
And it showed something very simple, which was a ball bearing running across the seams of this car.

s14
00:01:10.880 --> 00:01:17.280
And it was kind of revolutionary at the time because it showed that you could actually build a car that fit together perfectly.

s15
00:01:17.280 --> 00:01:30.000
And instead of building for volume and building and then fixing the old mindset, you could actually have a mindset focused on ship it right, on no rework, on customer happiness.

s16
00:01:30.000 --> 00:01:34.480
And there was not This was not a single change.

s17
00:01:34.480 --> 00:01:40.480
What happened was that people really started thinking about the factories differently.

s18
00:01:40.480 --> 00:01:46.480
They started engineering a building factories that were engineered to deliver quality.

s19
00:01:46.480 --> 00:01:50.880
And there's a number of different pieces that I think still resonate today.

s20
00:01:50.880 --> 00:01:54.880
The first is you prioritize quality instead of volume.

s21
00:01:54.880 --> 00:02:01.532
The second is that you fix the system, not just individual symptoms that you see.

s22
00:02:01.532 --> 00:02:12.452
The third is that you design for something that can be built as opposed to designing the wheels and designing the washer bottles and whatever it is and then hoping it fits together.

s23
00:02:12.452 --> 00:02:17.412
You solve problems as they occur as opposed to kicking the can down the road and fixing it at the end.

s24
00:02:17.412 --> 00:02:18.692
And you empower everybody.

s25
00:02:18.692 --> 00:02:23.092
Anyone could stop the line, not just the boss.

s26
00:02:23.092 --> 00:02:24.452
Now, why am I talking about this?

s27
00:02:24.452 --> 00:02:28.772
It's actually because modern software development actually embraced all of these principles.

s28
00:02:28.772 --> 00:02:41.252
If you think about how we build software today, what is considered best practice, a lot of the terms and the practices came from that idea of a Toyota production system.

s29
00:02:41.252 --> 00:02:52.772
Agile, Kanban, error budgets if you're in the SRE world, continuous integration, these ideas all kind of sprung from this notion of how do we build factories that work.

s30
00:02:52.772 --> 00:02:55.812
And it's worked really well for us in software development so far.

s31
00:02:55.812 --> 00:03:02.532
But agents, as we all know, you wouldn't be sitting here if you didn't understand that agents are changing the game.

s32
00:03:02.532 --> 00:03:07.852
And as agents are changing the game, weird things start to happen.

s33
00:03:07.852 --> 00:03:18.532
First of all, this is research that was published last month of the effect of implementing asynchronous agents in your development life cycle.

s34
00:03:18.532 --> 00:03:20.292
And I'm sure many of you have seen this.

s35
00:03:20.292 --> 00:03:25.092
You have an explosion of the number of lines of code written up 2,450%.

s36
00:03:25.612 --> 00:03:28.692
You have an increase in the number of commits.

s37
00:03:28.692 --> 00:03:34.132
So, the volume, code creation is now easy and it is now abundant.

s38
00:03:34.132 --> 00:03:40.652
And by the way, more and more people, I'm sure many people in this room are saying, "Okay, if it's easy and abundant, how do I automate it?

s39
00:03:40.652 --> 00:03:46.012
How do I build factories, agentic software factories that help automate and scale this?"

s40
00:03:46.012 --> 00:03:51.492
And today, we just finished our what we call state of code developer survey.

s41
00:03:51.492 --> 00:03:54.292
It'll be out next month, but this is a little preview.

s42
00:03:54.292 --> 00:04:05.494
Today, 5% of developers that we spoke to say that they're running agentic software factories, but 34% said in a year they expect to and 39% in financial services.

s43
00:04:05.494 --> 00:04:11.134
So, people are thinking about how do you actually build these next generation factories.

s44
00:04:11.134 --> 00:04:13.894
And there's a a couple of gotchas in here.

s45
00:04:13.894 --> 00:04:17.534
Security risk, the agents can work really well to help you build stuff.

s46
00:04:17.534 --> 00:04:28.814
It can also work really well to help you hack stuff, help you attack stuff, find vulnerabilities, find quality issues, etc. And so, as I was writing this um presentation

s47
00:04:28.814 --> 00:04:39.574
kind of over the weekend, a little just-in-time again from the Toyota production system, um Satya Nadella posted this article on X. If you haven't read it, I think it's a great article that I think everyone

s48
00:04:39.574 --> 00:04:40.694
uh should internalize.

s49
00:04:40.694 --> 00:04:44.774
But one of the things he was talking about insider risk of models inside of companies.

s50
00:04:44.774 --> 00:04:58.094
And one of the things that he talked about was the need for an engineering approach to containment and governance built around strong deterministic system design, human controls, and reliable operating procedures.

s51
00:04:58.094 --> 00:05:14.814
And this really [clears throat] resonated with me because what we are seeing is a paradigm shift in software factory design or software development, let's call it more broadly, that we need to engineer, we need to build systems that engineer confidence

s52
00:05:14.814 --> 00:05:19.094
and quality into agentic software factories.

s53
00:05:19.094 --> 00:05:25.054
So, to start doing that, I think it's important to understand the limitations that we're facing.

s54
00:05:25.054 --> 00:05:27.254
So, what's the hard thing about this?

s55
00:05:27.254 --> 00:05:28.694
And there's five that I'm going to call out.

s56
00:05:28.694 --> 00:05:32.814
The first is that human in the loop doesn't scale.

s57
00:05:32.814 --> 00:05:41.574
And this may be, I don't know, to this room it may not be all that controversial, but every virtually every company I talk to, I say, "What's your AI governance strategy?"

s58
00:05:41.574 --> 00:05:44.894
And they give me all the stuff and they say, "But don't worry, we have humans in the loop."

s59
00:05:44.894 --> 00:05:57.934
And the problem with humans in the loop, as I mentioned earlier, 2,450% increase in lines of code, 240% increase in commits, but the same study showed only a 30%

s60
00:05:57.934 --> 00:05:58.974
increase in releases.

s61
00:05:58.974 --> 00:06:04.974
We haven't figured out how to scale from producing stuff to shipping stuff.

s62
00:06:04.974 --> 00:06:15.735
There's automation needed, there's new tools needed, and really importantly, this is a concept that really I I see it every day now, and I'm sure many of you do as well,

s63
00:06:15.735 --> 00:06:17.855
uh called cognitive surrender.

s64
00:06:17.855 --> 00:06:24.935
Uh and this study was done just a couple months ago, and it looked at how often do humans agree with AI?

s65
00:06:24.935 --> 00:06:30.655
And it turns out they agree with AI when AI is right 93% of the time.

s66
00:06:30.655 --> 00:06:34.575
And four out of five times they agree with it when the AI is wrong.

s67
00:06:34.575 --> 00:06:34.975
All right?

s68
00:06:34.975 --> 00:06:39.815
So, it's not that human in the loop is Yeah, we just need to make things faster.

s69
00:06:39.815 --> 00:06:45.495
It's actually that it doesn't work in the agentic era.

s70
00:06:45.495 --> 00:06:46.295
So, that's problem one.

s71
00:06:46.295 --> 00:06:51.335
Problem two is that agents struggle with large complex code bases.

s72
00:06:51.335 --> 00:06:58.215
They're very very smart, but you need to understand what happens in a code base in order to work on it.

s73
00:06:58.215 --> 00:07:03.789
The left-hand side is a study that was done on code translation, and what it shows is that

s74
00:07:03.789 --> 00:07:03.815
[clears throat]

s75
00:07:03.815 --> 00:07:12.135
the agents were able to get it right 91% of the time when the code base was 10,000 lines of code.

s76
00:07:12.135 --> 00:07:20.775
And by the time it got to even 50,000, and I'm sure everyone in this room operates off of much bigger code bases than this, but when it was trying to translate 50,000,

s77
00:07:20.775 --> 00:07:23.415
the accuracy collapsed.

s78
00:07:23.415 --> 00:07:34.695
Similarly, another study looked at um how many tokens does it take on a large complex code base to complete the same task, and the variance was 30x, not 30%,

s79
00:07:34.695 --> 00:07:45.375
but 30x, and the reason is because it the agent spends a huge amount of their tokens trying to understand how to navigate the code base, the structure of the code base, things like this.

s80
00:07:45.375 --> 00:07:53.655
And it's kind of like if I'm trying to go from uh from LA to here, and if I pick a random starting point.

s81
00:07:53.655 --> 00:07:57.375
If I happen to pick White Plains, I will get to New York pretty quickly.

s82
00:07:57.375 --> 00:08:00.495
If I pick Albuquerque, it's going to take me a long time and a lot of cost.

s83
00:08:00.495 --> 00:08:04.095
Same thing happens with agents.

s84
00:08:04.175 --> 00:08:05.695
Errors compound.

s85
00:08:05.695 --> 00:08:08.055
This is another important concept.

s86
00:08:08.055 --> 00:08:14.897
This is a study out of meter and meter looks at and you can find this on their website.

s87
00:08:14.897 --> 00:08:21.017
They look at how effective are agents at completing tasks that humans can do.

s88
00:08:21.017 --> 00:08:25.457
And the way they look at this is can you complete a task that takes X number of hours.

s89
00:08:25.457 --> 00:08:34.737
So you can see here the Claude Mythos preview which is the latest version that they've published was good at completing tasks that take a human 16 hours.

s90
00:08:34.737 --> 00:08:41.777
Okay, so that's really good but there's an important caveat in the fine print here which is this is at 50% accuracy.

s91
00:08:41.777 --> 00:08:52.337
When you add it when you dial up the accuracy to 80% it the curve flattens substantially because as you do long running tasks errors compound.

s92
00:08:52.337 --> 00:09:01.057
If you have 95% accuracy on each task by the time you're at 20 steps, you have 30 36% accuracy.

s93
00:09:01.057 --> 00:09:01.497
Right?

s94
00:09:01.497 --> 00:09:06.657
And so that is another issue we have to deal with in software factories.

s95
00:09:06.657 --> 00:09:14.657
Probably the most aggressive headline I put in here is is this fourth point that LLMs are unreliable reward seeking judges.

s96
00:09:14.657 --> 00:09:19.257
Um as you know, so the question that we always get is well can't you just ask an LLM?

s97
00:09:19.257 --> 00:09:27.617
And so people have been studying this as well and this is all again the first on the left hand side is from PayPal AI's research lab.

s98
00:09:27.617 --> 00:09:40.577
They published a paper this month actually and found that between depending on the model and the situation between 7 and 17% of verdicts change across what they called identical replications.

s99
00:09:40.577 --> 00:09:45.777
Ask the same question, got a different answer between 7 and 17% of the time.

s100
00:09:45.777 --> 00:09:49.617
And if you start challenging, this is a study out of meta now.

s101
00:09:49.617 --> 00:09:52.577
If you start challenging the LLM, are you sure?

s102
00:09:52.577 --> 00:09:53.897
Very simple challenge.

s103
00:09:53.897 --> 00:09:58.497
It'll actually change its mind 7 to 34% again depending on the model.

s104
00:09:58.497 --> 00:10:03.897
So you ask the AI judge twice and you may get a different answer.

s105
00:10:03.897 --> 00:10:14.818
And then lastly and I won't dig into this one too much cuz I think it's probably self-evident to everyone as the time to exploit near zero, the ability for us to respond

s106
00:10:14.818 --> 00:10:16.698
is incredibly challenged.

s107
00:10:16.698 --> 00:10:22.738
And so, prevention is something that we believe the software factory has to bake in to the design.

s108
00:10:22.738 --> 00:10:25.738
Has to engineer into the factory.

s109
00:10:25.738 --> 00:10:38.218
So, as we've thought through this, we have um written down what we call the code confidence formula, which is basically how do you build How do you think about

s110
00:10:38.218 --> 00:10:45.098
a software factory that delivers what we all want, which is outcomes that we can feel confident in, right?

s111
00:10:45.098 --> 00:10:54.858
Not outcomes we can trust, not blind trust or anything like that, but outcomes that we can feel comfortable with or we can sit and tell our executives, our customers, our shareholders,

s112
00:10:54.858 --> 00:10:59.138
our CEOs, our boards that yes, I believe this factory is working well.

s113
00:10:59.138 --> 00:11:02.338
And we think there's four components to this.

s114
00:11:02.338 --> 00:11:07.578
The first is being crystal clear about the intent.

s115
00:11:07.578 --> 00:11:12.858
If you don't know what you're trying to build, you can't tell the agent what to build, and you will not get the outcome you want.

s116
00:11:12.858 --> 00:11:16.218
The second is guide rails.

s117
00:11:16.218 --> 00:11:19.218
And we very specifically talk about guide rails, not guard rails.

s118
00:11:19.218 --> 00:11:23.378
Guard rails prevent something from going off the road.

s119
00:11:23.378 --> 00:11:24.658
Guide rails help keep you on it.

s120
00:11:24.658 --> 00:11:30.978
And I think what agents need are the small nudges to guide and govern those agents.

s121
00:11:30.978 --> 00:11:39.938
The third part is verification that needs to be independent, thorough, and issues fixed as opposed to just cans kicked down the road.

s122
00:11:39.938 --> 00:11:42.058
And then lastly, evidence.

s123
00:11:42.058 --> 00:11:43.498
Trust, but verify.

s124
00:11:43.498 --> 00:11:47.418
Trust, but understand that it was done with proof.

s125
00:11:47.418 --> 00:11:49.538
Uh that is consistent, audible, and repeatable.

s126
00:11:49.538 --> 00:11:56.618
And we multiply these together because we've seen a number of situations where one of these elements is missing.

s127
00:11:56.618 --> 00:11:58.658
And if it's zero, the confidence goes to zero.

s128
00:11:58.658 --> 00:12:02.778
If you do all of this, but don't have evidence, you go to zero.

s129
00:12:02.778 --> 00:12:05.738
You don't have confidence in the outcome.

s130
00:12:05.738 --> 00:12:11.938
So, we really think about factory design is requiring these four elements.

s131
00:12:11.938 --> 00:12:19.579
And I'm going to And as we put together factors, as we help customers, I'm not going to walk through every box on this page, but as we help our customers

s132
00:12:19.579 --> 00:12:27.299
and partners build these software factories, we're very deliberate about how do you get the intent, bottom left-hand side in purple.

s133
00:12:27.299 --> 00:12:29.659
How do you put the guardrails in place?

s134
00:12:29.659 --> 00:12:33.659
How do you do verification in a way that's thorough and independent?

s135
00:12:33.659 --> 00:12:35.859
And how do you produce the evidence?

s136
00:12:35.859 --> 00:12:39.019
And by doing this, we see amazing results.

s137
00:12:39.019 --> 00:12:41.773
So, these are real results now from our from

s138
00:12:41.773 --> 00:12:41.899
[snorts]

s139
00:12:41.899 --> 00:12:42.579
our customers.

s140
00:12:42.579 --> 00:12:46.539
The first one is focusing on intent and guardrails.

s141
00:12:46.539 --> 00:12:55.699
And intent and guardrails, we have a product called Sonar Vortex, which gives intent guardrails and in the agentic loop starts to do verifications, again nudging the agents.

s142
00:12:55.699 --> 00:13:04.699
And what we see when we provide agents in long-running and large complex code bases with long-running tasks uh with

s143
00:13:04.699 --> 00:13:04.739
[snorts]

s144
00:13:04.739 --> 00:13:15.059
this information is a 36% reduction in token cost because you're putting it in White Plains, you're actually you're putting it close to the answer, you're helping it understand the map.

s145
00:13:15.059 --> 00:13:18.299
You don't have to burn tokens on things we already know.

s146
00:13:18.299 --> 00:13:22.259
And 92% reduction in issues in the commit.

s147
00:13:22.259 --> 00:13:22.459
Right?

s148
00:13:22.459 --> 00:13:28.219
So, it's cheaper and we can get a better outcome to go into that verification stage.

s149
00:13:28.219 --> 00:13:33.539
This is again all because we're nudging the agent to keep on track.

s150
00:13:33.539 --> 00:13:38.259
Then you get to verification, and we started life as a verification company.

s151
00:13:38.259 --> 00:13:48.259
This is how we see the world as through a verification lens, and we have a very strong point of view that independent verification has to cover the full range of issues

s152
00:13:48.259 --> 00:13:51.739
that is both precise and cost-effective.

s153
00:13:51.739 --> 00:13:52.019
Right?

s154
00:13:52.019 --> 00:13:56.979
Use CPUs when you can use CPUs, need GPUs GPUs when you need GPUs.

s155
00:13:56.979 --> 00:14:07.699
So, we look at deterministic mixture methods, and then you add in agentic mixture methods, and it comes up with what we think is the a gold standard of verification.

s156
00:14:07.699 --> 00:14:12.139
And what we see is when you apply this, our product is called SonarQube.

s157
00:14:12.139 --> 00:14:25.261
So, we talked to our customers, over 1,000 customers recently, and we said, "Are you How likely are you or how often have you seen these issues from a gentle coding?"

s158
00:14:25.261 --> 00:14:32.381
And the the statement to read here is SonarCube users are X% more likely than non-users to have seen a very positive impact.

s159
00:14:32.381 --> 00:14:45.101
So, 81% more likely to have seen to believe that AI coding has had a very positive impact on code security if you have that gold standard ver- verification.

s160
00:14:45.101 --> 00:14:54.221
34% more likely to have seen an impact on time to market, 76% on outage frequency.

s161
00:14:54.261 --> 00:15:04.541
So, again, as I come back to how do we have confidence to ship, what we're seeing is that if you put these pieces together and you build verification incredibly

s162
00:15:04.541 --> 00:15:21.181
strongly as a first-class citizen in the factory, you are actually building confidence and you're seeing results, better security, more confident in time to market, improved or reduction in number of outages, improved reliability.

s163
00:15:21.181 --> 00:15:25.181
And, of course, as I mentioned, you have to do this with evidence.

s164
00:15:25.181 --> 00:15:26.621
Make sure you pass the quality gate.

s165
00:15:26.621 --> 00:15:39.981
Make sure whatever There's lots of acronyms up here, whichever ones you need to pass, whether that is your IT general controls inside of your internal controls and operations, whether that's MISRA if you're building cars,

s166
00:15:39.981 --> 00:15:45.141
or any of the other ones that are on here, producing the evidence is what lets you sleep at night.

s167
00:15:45.141 --> 00:15:49.061
It's what gives you confidence that you can move forward.

s168
00:15:49.061 --> 00:16:03.181
So, that, in essence, is how we think about the challenge of the future, that code is abundant, that confidence is scarce, and really the goal that we have and that we think

s169
00:16:03.181 --> 00:16:09.781
that everyone should have as we think about the future of software development is how do we deliver confidence.

s170
00:16:09.781 --> 00:16:14.141
And the factories of the future need to be able to deliver confidence.

s171
00:16:14.141 --> 00:16:16.861
So, with that, I want to say thank you to everybody.

s172
00:16:16.861 --> 00:16:25.742
We have a big booth, it's a big red booth on the floor right over that way and would love to talk to you more about all of this.

s173
00:16:25.742 --> 00:16:27.751
Thank you very much, have a great morning.

s174
00:16:27.751 --> 00:16:29.751
[applause]