WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Se8jHLliLXE

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Se8jHLliLXE.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.719 --> 00:00:13.920
Uh I'm Justin Rio.

s2
00:00:13.920 --> 00:00:15.599
I'm the deputy CTO at DX.

s3
00:00:15.599 --> 00:00:22.000
I get to do a lot of our research around uh developer experience and its relation to developer productivity.

s4
00:00:22.000 --> 00:00:33.120
And certainly over the last year, a lot of that research has focused on AI's impact on the developer experience and basic aspects of productivity uh within an organization.

s5
00:00:33.120 --> 00:00:42.399
Uh so we produce these quarterly reports, our our state of AI and AI impact reports which looks at really a lot of raw data that we're able to get from our platform.

s6
00:00:42.399 --> 00:00:50.800
Our platform is a researchback platform uh built from the same folks who worked on Dora metrics and the space framework and the DevX framework.

s7
00:00:50.800 --> 00:00:59.600
Uh and really at the end of the day it's a data gathering platform that can look at a lot of different trends across an organization with you know how are we focusing on developer experience

s8
00:00:59.600 --> 00:01:08.560
are we seeing outcomes and improvements with developer experience and certainly are we seeing outcomes with the way that we're using AI in our workflows.

s9
00:01:08.560 --> 00:01:18.159
This session is going to be a little bit different than the 345 session I think that's up in the leadership room later today which just looks at the raw data from our newest report.

s10
00:01:18.159 --> 00:01:19.520
It's a little bit of a preview.

s11
00:01:19.520 --> 00:01:26.400
This is looking at more like organizational trends and how our workflow is changing and and things like that based on the data that we're seeing.

s12
00:01:26.400 --> 00:01:31.200
Uh so first things first, let's kind of look at what we're seeing in terms of the velocity impact.

s13
00:01:31.200 --> 00:01:40.079
Now if anybody's familiar with looking at various velocity metrics like PR throughput and deployment frequency like hopefully you understand the caveat that these metrics are not perfect.

s14
00:01:40.079 --> 00:01:44.799
They are proxy metrics for understanding the way that work flows through an organization.

s15
00:01:44.799 --> 00:01:47.600
They're not fully representative of the value generation.

s16
00:01:47.600 --> 00:01:50.399
We're going to look at some metrics that are closer to that as well.

s17
00:01:50.399 --> 00:01:57.200
But let's just try to understand what this technology is doing in terms of just the raw velocity metrics in the organization.

s18
00:01:57.200 --> 00:01:59.920
So here's our Dora metric for deployment frequency.

s19
00:01:59.920 --> 00:02:02.240
Who's familiar with Dora metrics?

s20
00:02:02.240 --> 00:02:03.200
All right, most of you.

s21
00:02:03.200 --> 00:02:03.600
Great.

s22
00:02:03.600 --> 00:02:12.400
Um, so this is one of the, you know, four key metrics that tries to understand directionally, you know, the speed of shipping work within an environment.

s23
00:02:12.400 --> 00:02:14.800
And we're seeing it steadily increase.

s24
00:02:14.800 --> 00:02:17.440
um it's tapering off a little bit.

s25
00:02:17.440 --> 00:02:19.360
I think there's reasons for that.

s26
00:02:19.360 --> 00:02:27.200
Uh some of the surge that we saw early on was due to people shipping, you know, PRs more frequently and and things like that.

s27
00:02:27.200 --> 00:02:30.879
Um so we're seeing increases in this.

s28
00:02:30.879 --> 00:02:33.760
This is one small part of the SDLC PDLC, right?

s29
00:02:33.760 --> 00:02:38.879
This is like the creation uh of the PR to actually trying to deploy this into production.

s30
00:02:38.879 --> 00:02:40.560
It doesn't tell us about revert rates.

s31
00:02:40.560 --> 00:02:42.560
It doesn't tell us about defect ratios.

s32
00:02:42.560 --> 00:02:44.720
It doesn't tell us about change failure rate.

s33
00:02:44.720 --> 00:02:49.040
It just tells us about how much stuff are we now kind of pushing out into production.

s34
00:02:49.040 --> 00:02:52.640
But you know the data is showing us that this number is steadily increasing.

s35
00:02:52.640 --> 00:02:53.519
Okay.

s36
00:02:53.519 --> 00:02:57.680
Um the trends vary a little bit across regions.

s37
00:02:57.680 --> 00:03:01.599
Um we are seeing North America just sort of like trending up.

s38
00:03:01.599 --> 00:03:06.159
We've seen Europe actually pull back a little bit just in the last quarter.

s39
00:03:06.159 --> 00:03:08.959
Um I think you know there's there's a number of reasons for that.

s40
00:03:08.959 --> 00:03:12.000
I mean obviously the way work is done in these regions are differently.

s41
00:03:12.000 --> 00:03:17.519
the way that we can spend money on this technology and think about token spend is different, the regulation is different.

s42
00:03:17.519 --> 00:03:28.560
Um, but if we just look at the through line in general, you know, we still see improvements uh in in this one metric of deploy frequency even across various regions.

s43
00:03:28.560 --> 00:03:32.222
We do just see some variance in those regions.

s44
00:03:32.222 --> 00:03:34.480
[clears throat] Here's one that's kind of interesting though.

s45
00:03:34.480 --> 00:03:37.599
Uh, the perceived rate of delivery.

s46
00:03:37.599 --> 00:03:42.560
How much faster do engineers feel in terms of what they're delivering?

s47
00:03:42.560 --> 00:03:44.480
It's gone up a little bit.

s48
00:03:44.480 --> 00:03:48.640
This number shows an increase of about four and a halfish percent.

s49
00:03:48.640 --> 00:04:00.000
Um, but given that that's a full year and given all the investment and cost and new spend on AI, it's interesting that this perceived rate has not stayed has not kind of crept up a little bit more.

s50
00:04:00.000 --> 00:04:16.479
especially when you look at some studies like obviously infamous flawed MER study if you're familiar with that study uh back in February they actually released a follow-up to that study saying that some of the signals they gathered were sort of imperfect and they needed to rethink some of that but what was interesting was that in these 16

s51
00:04:16.479 --> 00:04:26.800
engineers that were part of that study small study flawed study uh the productivity went down by about 19% but the perception went up by about 20% so there was like a 40%

s52
00:04:26.800 --> 00:04:30.720
spread in perception of productivity versus actual ual productivity.

s53
00:04:30.720 --> 00:04:37.199
But here when we look at this on more of an aggregate things interestingly stay uh sort of flat.

s54
00:04:37.199 --> 00:04:37.680
All right.

s55
00:04:37.680 --> 00:04:40.560
What about impact on quality and and delivery?

s56
00:04:40.560 --> 00:04:40.800
Right?

s57
00:04:40.800 --> 00:04:43.199
So those are just kind of our speed our velocity metrics.

s58
00:04:43.199 --> 00:04:46.400
But what is this actually doing to our software quality?

s59
00:04:46.400 --> 00:04:51.600
Um the first metric we'll look at here is the Dora change failure rate metric.

s60
00:04:51.600 --> 00:04:53.199
And uh let me prepare you.

s61
00:04:53.199 --> 00:04:54.479
Most of you are sitting down.

s62
00:04:54.479 --> 00:04:58.880
It's very volatile the impact that we've seen on quality over this period of time.

s63
00:04:58.880 --> 00:05:05.759
Each line on this graph represents a single company that was part of this particular part of this study.

s64
00:05:05.759 --> 00:05:11.600
And that line shows whether you've moved up in terms of your change failure rate or moved down in terms of your change failure rate.

s65
00:05:11.600 --> 00:05:12.960
You want to go down.

s66
00:05:12.960 --> 00:05:15.360
You want to be on the bottom side of this graph.

s67
00:05:15.360 --> 00:05:20.400
Uh but what's interesting is if you look at some of these top lines, you see people increasing as much as 2%.

s68
00:05:20.400 --> 00:05:24.800
Which doesn't sound like much until you realize that the industry benchmark is about 4%.

s69
00:05:24.800 --> 00:05:29.280
So that means shipping potentially 50% more defects than we were shipping before.

s70
00:05:29.280 --> 00:05:32.160
So you really just want to understand what side of this graph you're on.

s71
00:05:32.160 --> 00:05:34.320
You want to start measuring this stuff, right?

s72
00:05:34.320 --> 00:05:38.400
Um this pattern exists without AI, by the way.

s73
00:05:38.400 --> 00:05:41.039
This has a lot to do, it's not purely causal with AI.

s74
00:05:41.039 --> 00:05:47.120
A lot of this has to do with like your release pipeline, your automated testing, everything that's surrounding the way that you're releasing the code.

s75
00:05:47.120 --> 00:05:48.720
The pattern remains the same.

s76
00:05:48.720 --> 00:05:51.759
The amplitude has changed as a result of AI.

s77
00:05:51.759 --> 00:05:56.000
We always see this type of shift in volatility, but not usually to these extremes.

s78
00:05:56.000 --> 00:05:59.280
And that's the AI effect that we're seeing right there.

s79
00:05:59.280 --> 00:06:01.039
Here's a really interesting one.

s80
00:06:01.039 --> 00:06:07.759
These are two qualitative quant uh uh quality metrics uh that normally are sort of in sync with one another.

s81
00:06:07.759 --> 00:06:13.759
Code maintainability, how how maintainable dovel developers perceive the code to be in terms of making changes to it.

s82
00:06:13.759 --> 00:06:17.680
That number has gone up almost 4% according to our data.

s83
00:06:17.680 --> 00:06:21.840
By the way, we're looking at about 200,000 engineers in this study to gather this data.

s84
00:06:21.840 --> 00:06:30.000
The uh co the change confidence though, which traditionally we've seen closely associated with code maintainability.

s85
00:06:30.000 --> 00:06:31.280
Hey, this code is maintainable.

s86
00:06:31.280 --> 00:06:32.240
It's modular.

s87
00:06:32.240 --> 00:06:33.520
It's easy to modify.

s88
00:06:33.520 --> 00:06:36.400
I feel good about the changes that I'm pushing into production.

s89
00:06:36.400 --> 00:06:36.960
Nope.

s90
00:06:36.960 --> 00:06:40.240
Our change confidence number has gone down 6%.

s91
00:06:40.240 --> 00:06:49.919
This is really an interesting tension because it means like, okay, agents and assistants are making it easier for me to understand and modify the code that's in front of me, but I trust the outputs less.

s92
00:06:49.919 --> 00:06:54.080
I'm more afraid now of breaking things than I was a year ago.

s93
00:06:54.080 --> 00:06:57.039
So, it's an interesting psychological effect that we see here.

s94
00:06:57.039 --> 00:07:00.479
And a lot of this is because PR size is measurably increasing.

s95
00:07:00.479 --> 00:07:03.599
This is going to be one of the most important metrics that we look at this year.

s96
00:07:03.599 --> 00:07:07.199
Hasn't quite doubled, but look at that trajectory, right?

s97
00:07:07.199 --> 00:07:17.280
This is over the same period of study about a year and we've seen PRs go from uh around 44 lines on average per PR up to 72.

s98
00:07:17.280 --> 00:07:19.680
Now there's there's a number of reasons for this.

s99
00:07:19.680 --> 00:07:26.960
It's not just, you know, the likelihood that the way that these models work is that they're be being trained to output relatively mediocre code.

s100
00:07:26.960 --> 00:07:29.199
I've been writing code professionally since the late 90s.

s101
00:07:29.199 --> 00:07:34.479
I remember when it was a great mark of a developer to be able to implement the same use case with as little code as possible.

s102
00:07:34.479 --> 00:07:36.639
Now, it's like, well, no, let's get the thing out.

s103
00:07:36.639 --> 00:07:40.960
And, you know, the code that's being generated is going to be essentially mediocre because it's law of averages.

s104
00:07:40.960 --> 00:07:50.240
But on top of that, too, think about if you've got a build pipeline that takes 45 minutes or an hour to complete and now I've got AI generating functions for me instantly.

s105
00:07:50.240 --> 00:07:59.680
Am I going to push four different PRs with four different functions and wait 45 minutes or an hour or whatever that is per build or I'm just going to cram all that code into a single PR?

s106
00:07:59.680 --> 00:08:00.000
Right?

s107
00:08:00.000 --> 00:08:07.199
So there's multiple reasons for this, but it really bears mentioning that like every extra line of code is a potential bug, a potential vulnerability.

s108
00:08:07.199 --> 00:08:08.080
It's more to review.

s109
00:08:08.080 --> 00:08:09.599
It makes the code less portable.

s110
00:08:09.599 --> 00:08:16.080
So we need to pay attention to this metric as well because we're also seeing this associated with perception of incremental delivery.

s111
00:08:16.080 --> 00:08:24.800
I get to work on small incremental changes and this is actually one of the lowest qualitative developer experience drivers that we've seen impacted over the last year.

s112
00:08:24.800 --> 00:08:28.319
This number and this sentiment has gone down 10%.

s113
00:08:28.319 --> 00:08:37.760
And incremental delivery is awful is is as we've learned also very important for uh for the way that we try to deliver in a way that's easy to roll back

s114
00:08:37.760 --> 00:08:42.040
uh and where we can understand and review the changes that are being pushed through.

s115
00:08:42.040 --> 00:08:42.080
[snorts]

s116
00:08:42.080 --> 00:08:43.440
Okay.

s117
00:08:43.440 --> 00:08:45.519
Uh what about demographic differences?

s118
00:08:45.519 --> 00:08:50.480
Uh what is different about AI impacts across like you know junior devs and stuff like that?

s119
00:08:50.480 --> 00:08:53.519
Junior engineers are using AI the most.

s120
00:08:53.519 --> 00:08:55.440
This really should surprise nobody.

s121
00:08:55.440 --> 00:09:05.040
anytime we have sort of a new leap again writing code professionally since the late 90s I'm a bit of an OG I'm on this journey too but there's less to unlearn

s122
00:09:05.040 --> 00:09:14.640
when you're coming into the industry right in many many cases you've already been working with this kind of stuff like right out of school and so you're more likely to implement it

s123
00:09:14.640 --> 00:09:28.959
uh when you actually start working now we've seen some other trends there too uh we have interesting ways of looking per use case at efficiency uh which I'll get into in the second talk later today and one of those is looking at agent experience

s124
00:09:28.959 --> 00:09:33.200
like literally we're we're asking agents about their experience working with humans.

s125
00:09:33.200 --> 00:09:38.399
We're trying to figure out what use cases uh are they working on and how many tokens did they spend.

s126
00:09:38.399 --> 00:09:46.480
And definitely we also see a trend where the more junior developers for the same use case will spend more tokens than a more senior developer on the same use case.

s127
00:09:46.480 --> 00:09:47.839
And that's just a learning curve, right?

s128
00:09:47.839 --> 00:09:49.200
Again, that should surprise nobody.

s129
00:09:49.200 --> 00:09:53.760
But in terms of who's using it the most, we do see juniors using it the most.

s130
00:09:53.760 --> 00:10:01.200
However, despite that additional use, we see staff plus or more senior engineers saving about the same amount of time, right?

s131
00:10:01.200 --> 00:10:02.720
And and burning less tokens.

s132
00:10:02.720 --> 00:10:04.720
I want to make sure that that's kind of clear too.

s133
00:10:04.720 --> 00:10:14.959
So in terms of like raw time savings in the aggregate output of this, it's pretty flat between junior engineers who are using the tech more and staff engineers who have an easier time spotting hallucinations,

s134
00:10:14.959 --> 00:10:19.600
understanding the architecture around the changes they're making and things like that.

s135
00:10:19.600 --> 00:10:22.000
Smaller companies lead in time savings.

s136
00:10:22.000 --> 00:10:25.360
Sure, you know, the release pipelines are going to be less complicated.

s137
00:10:25.360 --> 00:10:28.959
You know, there's less overall complexity in the organization.

s138
00:10:28.959 --> 00:10:37.600
uh that's going to uh lead to more time savings for a company that already has less friction and has a little bit more agility to it than a larger

s139
00:10:37.600 --> 00:10:39.600
company.

s140
00:10:39.760 --> 00:10:41.040
What about measurement?

s141
00:10:41.040 --> 00:10:44.000
How should we be even be thinking about measuring this stuff?

s142
00:10:44.000 --> 00:10:56.000
I mean, it bears mentioning that that measuring developer productivity and measuring developer experience was a challenge before AI and we we never completed that conversation before kind of throwing accelerant on this.

s143
00:10:56.000 --> 00:10:56.320
Right?

s144
00:10:56.320 --> 00:11:07.120
So, this is still a difficult thing and I'd argue even more difficult to do now because it's confounded by several different aspects of of AI, but we're all going to have to answer this question this year, right?

s145
00:11:07.120 --> 00:11:11.839
Spent 10 million or way more if you come to my talk later today on tokens.

s146
00:11:11.839 --> 00:11:15.279
Uh where's our 10x productivity?

s147
00:11:15.279 --> 00:11:21.040
When we're thinking about this stuff, it's important to bear in mind that we don't throw away what we're measuring, right?

s148
00:11:21.040 --> 00:11:26.560
our foundational developer experience and developer productivity metrics are still what matter the most, right?

s149
00:11:26.560 --> 00:11:32.880
We want to understand how a AI is impacting these trusted metrics that we've already been looking at.

s150
00:11:32.880 --> 00:11:35.920
Like we want to maybe look at cohorts of users.

s151
00:11:35.920 --> 00:11:43.680
We maybe want to take data from the API telemetry and things that come out of the tools and use that to understand who's using what and where.

s152
00:11:43.680 --> 00:11:48.480
But we would only want to use those cohorts and look at them comparatively across our foundational metrics.

s153
00:11:48.480 --> 00:11:50.079
What is this actually doing to quality?

s154
00:11:50.079 --> 00:11:51.839
What is this actually doing to speed?

s155
00:11:51.839 --> 00:11:56.160
What is this doing to impact to the organization and value generation?

s156
00:11:56.160 --> 00:11:59.120
And so this is where our AI measurement framework comes from.

s157
00:11:59.120 --> 00:12:00.720
I won't spend too much time on this.

s158
00:12:00.720 --> 00:12:02.560
We have a big white paper available.

s159
00:12:02.560 --> 00:12:14.240
Uh my colleague Anton who's standing in the back of the room in the the white shirt will be able to hand out some hard copies of reports that we have after the talk that go into the way that we derive this methodology and how to use it and think about it.

s160
00:12:14.240 --> 00:12:17.360
Um it focuses on three key dimensions of measurement.

s161
00:12:17.360 --> 00:12:22.480
So looking at utilization, our daily active users, weekly active users, that sort of thing.

s162
00:12:22.480 --> 00:12:24.079
Uh it looks at impact.

s163
00:12:24.079 --> 00:12:28.480
So which metric should we be, you know, looking for moving the needle based on utilization?

s164
00:12:28.480 --> 00:12:33.600
Uh as our utilization goes up, what do we hope to see in terms of impact to the business?

s165
00:12:33.600 --> 00:12:34.959
And then cost.

s166
00:12:34.959 --> 00:12:41.839
It's a fair joke that we're 15 years after the last major hype cycle and we're still trying to figure out cloud cost, but the stuff is getting pretty expensive.

s167
00:12:41.839 --> 00:12:43.040
We're going to have to measure it.

s168
00:12:43.040 --> 00:12:45.680
Um so you can think about this as a bit of a maturity curve as well.

s169
00:12:45.680 --> 00:12:48.000
Most people start with utilization on the left.

s170
00:12:48.000 --> 00:12:53.680
Just figuring out what's happening in the organization with the tech, who's using it, what are they using, how often, and for what use cases.

s171
00:12:53.680 --> 00:13:04.720
And then how do we then cross reference those to the impact metrics, the value generation, what we really want to understand to know whether these investments are actually working, which is what we really want to figure out.

s172
00:13:04.720 --> 00:13:07.440
This is just an example of how we might correlate such a thing.

s173
00:13:07.440 --> 00:13:16.639
So you could look at two separate cohorts of users and you could compare them across metrics like PR cycle time or PR size or push back and review.

s174
00:13:16.639 --> 00:13:16.959
Right?

s175
00:13:16.959 --> 00:13:27.600
So lots of uh the way to think about this again is to right now separate cohorts and then look at how that plays out across these foundational productivity metrics that we've learned to trust.

s176
00:13:27.600 --> 00:13:31.760
We also need to able be able to understand our platform's readiness.

s177
00:13:31.760 --> 00:13:34.800
I like to say that in 2024 we gave everybody a coding assistant.

s178
00:13:34.800 --> 00:13:36.880
In 2025 we started building agents.

s179
00:13:36.880 --> 00:13:40.000
Now, we're realizing that our infrastructure wasn't ready for any of this.

s180
00:13:40.000 --> 00:13:44.720
A lot of the vendors that are here are are are selling solutions that help you with this problem.

s181
00:13:44.720 --> 00:13:53.519
Um, I think if you can measure your platform's AI readiness, you're going to have a much better handle on how efficient you're going to be spending tokens and providing context to these agents.

s182
00:13:53.519 --> 00:14:03.519
So, these are things like clear and accurate, well structured documentation, data structures with straightforward relations, manageable modular code, reliable local CI and non-flaky test suites.

s183
00:14:03.519 --> 00:14:09.600
If by the way any of this sounds familiar to you, it's because we used to just call this good developer experience, right?

s184
00:14:09.600 --> 00:14:12.720
But it turns out that what's good for humans is also good for agents.

s185
00:14:12.720 --> 00:14:18.800
So we may finally paradoxically be making those investments that we should have been making over the last couple of decades.

s186
00:14:18.800 --> 00:14:22.639
We also want to understand the effectiveness of the way that we work with AI.

s187
00:14:22.639 --> 00:14:24.000
And this is one way of doing that.

s188
00:14:24.000 --> 00:14:32.560
What you're actually seeing here is feedback, qualitative feedback coming back from agents telling us where they ran into issues with steering with the human.

s189
00:14:32.560 --> 00:14:37.040
the the context that was provided for them, the the feedback cycles that they had to go through.

s190
00:14:37.040 --> 00:14:42.560
So, we're actually getting that data from the agents to try to figure out how effective our teams are at working with AI.

s191
00:14:42.560 --> 00:14:46.800
And we can break this down into use cases, which is very important, too.

s192
00:14:46.800 --> 00:14:51.040
We want to understand which use cases are giving us the most efficient token spend.

s193
00:14:51.040 --> 00:14:56.079
And again, moving the needle for value creation and productivity within the business.

s194
00:14:56.079 --> 00:15:00.160
Finally, the fifth trend, integrating across the SDLC and PDLC.

s195
00:15:00.160 --> 00:15:01.199
Why would we want to do this?

s196
00:15:01.199 --> 00:15:04.720
Because code generation was never the bottleneck in the first place, right?

s197
00:15:04.720 --> 00:15:15.040
Even if engineers are getting like 100% accurate instant code coming from the models, which they are not, you would still only be attacking anywhere from maybe 14 to 16% of the overall

s198
00:15:15.040 --> 00:15:15.760
value stream.

s199
00:15:15.760 --> 00:15:17.360
So we need to think beyond that.

s200
00:15:17.360 --> 00:15:23.920
Uh especially because right now the productivity gains that we are actually seeing are more modest than expected.

s201
00:15:23.920 --> 00:15:35.120
Despite additional PR throughput uh increases, our median increase in the study that we did from November to to 2024 to February this year only found about a median 7.7%

s202
00:15:35.120 --> 00:15:40.959
increase in this velocity metric, a 13% average, but even our top performers were in the 70% range.

s203
00:15:40.959 --> 00:15:44.800
Nobody hit 2x, nobody hit 5x, nobody hit 10x.

s204
00:15:44.800 --> 00:15:53.360
And that's because our time savings that come out of AI are still being outweighed by other nonAI factors within our organization, right?

s205
00:15:53.360 --> 00:15:53.600
Right?

s206
00:15:53.600 --> 00:15:56.480
We might be saving a lot of time with AI and that's great.

s207
00:15:56.480 --> 00:16:09.519
But if we're still eclipsing those time savings because of meeting heavy days and context switching and other sources of interruption, the cumulative effects of the dev environment and friction in the way that we create software around the code, then we're not attacking the bottleneck.

s208
00:16:09.519 --> 00:16:18.560
And as Ellie Goldroat from the theory of constraints and the goal and uh inspire inspiration for the Phoenix project would tell us that an hour saved on something that isn't the bottleneck is worthless.

s209
00:16:18.560 --> 00:16:21.759
And so we need to find the bottleneck and fix the bottleneck.

s210
00:16:21.759 --> 00:16:24.320
And that's what a lot of organizations are doing.

s211
00:16:24.320 --> 00:16:33.839
Morgan Stanley has been really public about their creation of the DevGen AI agent, which is an agent for interpreting legacy code like uh mainframe natural and cobalt.

s212
00:16:33.839 --> 00:16:34.959
And I hate to admit Pearl.

s213
00:16:34.959 --> 00:16:37.519
I've written a lot of Pearl in my career, but it's legacy now.

s214
00:16:37.519 --> 00:16:43.440
The agent actually creates PRDS to hand to engineers to get rid of that reverse engineering step.

s215
00:16:43.440 --> 00:16:47.040
And it's saving Morgan Stanley about 300,000 hours a year right now.

s216
00:16:47.040 --> 00:16:49.759
Zapier is one of my favorite stories in AI right now.

s217
00:16:49.759 --> 00:16:57.600
They have a whole agent ecosystem that they're using to deal with a lot of administrative tasks and overhead like daily standups and things like that.

s218
00:16:57.600 --> 00:17:06.240
They've reduced their standups by using a agent summaries and things like that in this ecosystem that they've built to two times a week as opposed to five times a week.

s219
00:17:06.240 --> 00:17:10.880
They're successfully onboarding engineers within about a twoe period which is really fast.

s220
00:17:10.880 --> 00:17:14.079
Our industry benchmark for that is over a month usually.

s221
00:17:14.079 --> 00:17:20.400
But my favorite part about this story is that they're seeing about a 15% additional value creation per engineer.

s222
00:17:20.400 --> 00:17:28.079
So they are hiring more than they have in the history of their entire company because they realize the truth here that they're getting more capacity per single engineer.

s223
00:17:28.079 --> 00:17:34.720
They're getting a better return on investment per single engineer and hiring more engineers will improve their competitive edge.

s224
00:17:34.720 --> 00:17:36.080
This is the right attitude.

s225
00:17:36.080 --> 00:17:37.440
This is a throughput story.

s226
00:17:37.440 --> 00:17:39.760
This is an increased innovative capacity story.

s227
00:17:39.760 --> 00:17:42.320
This is not a headcount replacement story.

s228
00:17:42.320 --> 00:17:42.880
Fair.

s229
00:17:42.880 --> 00:17:50.160
uh they're automating about 3,000 of their code reviews a week triggered off of a pull request and um they look at superficial stuff.

s230
00:17:50.160 --> 00:17:57.280
They still need humans in the loop, but the initial superficial uh uh review that comes from the agent is all part of the system of record.

s231
00:17:57.280 --> 00:18:03.919
It stays in the PR comments and so the next engineer to actually perform the review will have an idea of what the agents already looked at.

s232
00:18:03.919 --> 00:18:05.679
So it can save some time.

s233
00:18:05.679 --> 00:18:11.360
Finally, Spotify, if anybody remembers the Spotify model from DevOps, they were sort of the northstar for DevOps.

s234
00:18:11.360 --> 00:18:24.160
uh they built an agent for S SRRES that will gather remediation steps from runbooks and information and context about the incident and put all that together into SR communication channels so that when an incident occurs

s235
00:18:24.160 --> 00:18:31.840
the S sur has uh immediate context and doesn't have to go through multiple minutes of discovery uh to figure out how to solve the issue.

s236
00:18:31.840 --> 00:18:36.480
So, if you want to dive deeper into this data, our Q2 report is coming out in just a few weeks.

s237
00:18:36.480 --> 00:18:40.080
Uh, but the Q1 report still has a lot of tidbits from this session in it.

s238
00:18:40.080 --> 00:18:46.160
And if you subscribe to our newsletter or you come back to the website, by midish end of July, we'll have our Q2 report.

s239
00:18:46.160 --> 00:18:50.720
And if you want to see a preview of that report, that's the talk that I'm giving later this afternoon.

s240
00:18:50.720 --> 00:18:51.760
Thanks everybody for your time.

s241
00:18:51.760 --> 00:18:54.240
I appreciate it.

s242
00:18:59.713 --> 00:19:01.713
[music]

s243
00:19:05.608 --> 00:19:07.608
[music]