The State of AI in Software Development: Data from 400+ Orgs — Justin Reock, DX

AI Engineer · 19 min · 243 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Uh I'm Justin Rio.
00:13I'm the deputy CTO at DX.
00:15I get to do a lot of our research around uh developer experience and its relation to developer productivity.
00:22And certainly over the last year, a lot of that research has focused on AI's impact on the developer experience and basic aspects of productivity uh within an organization.
00:33Uh so we produce these quarterly reports, our our state of AI and AI impact reports which looks at really a lot of raw data that we're able to get from our platform.
00:42Our platform is a researchback platform uh built from the same folks who worked on Dora metrics and the space framework and the DevX framework.
00:50Uh and really at the end of the day it's a data gathering platform that can look at a lot of different trends across an organization with you know how are we focusing on developer experience
00:59are we seeing outcomes and improvements with developer experience and certainly are we seeing outcomes with the way that we're using AI in our workflows.
01:08This session is going to be a little bit different than the 345 session I think that's up in the leadership room later today which just looks at the raw data from our newest report.
01:18It's a little bit of a preview.
01:19This is looking at more like organizational trends and how our workflow is changing and and things like that based on the data that we're seeing.
01:26Uh so first things first, let's kind of look at what we're seeing in terms of the velocity impact.
01:31Now if anybody's familiar with looking at various velocity metrics like PR throughput and deployment frequency like hopefully you understand the caveat that these metrics are not perfect.
01:40They are proxy metrics for understanding the way that work flows through an organization.
01:44They're not fully representative of the value generation.
01:47We're going to look at some metrics that are closer to that as well.
01:50But let's just try to understand what this technology is doing in terms of just the raw velocity metrics in the organization.
01:57So here's our Dora metric for deployment frequency.
01:59Who's familiar with Dora metrics?
02:02All right, most of you.
02:03Great.
02:03Um, so this is one of the, you know, four key metrics that tries to understand directionally, you know, the speed of shipping work within an environment.
02:12And we're seeing it steadily increase.
02:14um it's tapering off a little bit.
02:17I think there's reasons for that.
02:19Uh some of the surge that we saw early on was due to people shipping, you know, PRs more frequently and and things like that.
02:27Um so we're seeing increases in this.
02:30This is one small part of the SDLC PDLC, right?
02:33This is like the creation uh of the PR to actually trying to deploy this into production.
02:38It doesn't tell us about revert rates.
02:40It doesn't tell us about defect ratios.
02:42It doesn't tell us about change failure rate.
02:44It just tells us about how much stuff are we now kind of pushing out into production.
02:49But you know the data is showing us that this number is steadily increasing.
02:52Okay.
02:53Um the trends vary a little bit across regions.
02:57Um we are seeing North America just sort of like trending up.
03:01We've seen Europe actually pull back a little bit just in the last quarter.
03:06Um I think you know there's there's a number of reasons for that.
03:08I mean obviously the way work is done in these regions are differently.
03:12the way that we can spend money on this technology and think about token spend is different, the regulation is different.
03:17Um, but if we just look at the through line in general, you know, we still see improvements uh in in this one metric of deploy frequency even across various regions.
03:28We do just see some variance in those regions.
03:32[clears throat] Here's one that's kind of interesting though.
03:34Uh, the perceived rate of delivery.
03:37How much faster do engineers feel in terms of what they're delivering?
03:42It's gone up a little bit.
03:44This number shows an increase of about four and a halfish percent.
03:48Um, but given that that's a full year and given all the investment and cost and new spend on AI, it's interesting that this perceived rate has not stayed has not kind of crept up a little bit more.
04:00especially when you look at some studies like obviously infamous flawed MER study if you're familiar with that study uh back in February they actually released a follow-up to that study saying that some of the signals they gathered were sort of imperfect and they needed to rethink some of that but what was interesting was that in these 16
04:16engineers that were part of that study small study flawed study uh the productivity went down by about 19% but the perception went up by about 20% so there was like a 40%
04:26spread in perception of productivity versus actual ual productivity.
04:30But here when we look at this on more of an aggregate things interestingly stay uh sort of flat.
04:37All right.
04:37What about impact on quality and and delivery?
04:40Right?
04:40So those are just kind of our speed our velocity metrics.
04:43But what is this actually doing to our software quality?
04:46Um the first metric we'll look at here is the Dora change failure rate metric.
04:51And uh let me prepare you.
04:53Most of you are sitting down.
04:54It's very volatile the impact that we've seen on quality over this period of time.
04:58Each line on this graph represents a single company that was part of this particular part of this study.
05:05And that line shows whether you've moved up in terms of your change failure rate or moved down in terms of your change failure rate.
05:11You want to go down.
05:12You want to be on the bottom side of this graph.
05:15Uh but what's interesting is if you look at some of these top lines, you see people increasing as much as 2%.
05:20Which doesn't sound like much until you realize that the industry benchmark is about 4%.
05:24So that means shipping potentially 50% more defects than we were shipping before.
05:29So you really just want to understand what side of this graph you're on.
05:32You want to start measuring this stuff, right?
05:34Um this pattern exists without AI, by the way.
05:38This has a lot to do, it's not purely causal with AI.
05:41A lot of this has to do with like your release pipeline, your automated testing, everything that's surrounding the way that you're releasing the code.
05:47The pattern remains the same.
05:48The amplitude has changed as a result of AI.
05:51We always see this type of shift in volatility, but not usually to these extremes.
05:56And that's the AI effect that we're seeing right there.
05:59Here's a really interesting one.
06:01These are two qualitative quant uh uh quality metrics uh that normally are sort of in sync with one another.
06:07Code maintainability, how how maintainable dovel developers perceive the code to be in terms of making changes to it.
06:13That number has gone up almost 4% according to our data.
06:17By the way, we're looking at about 200,000 engineers in this study to gather this data.
06:21The uh co the change confidence though, which traditionally we've seen closely associated with code maintainability.
06:30Hey, this code is maintainable.
06:31It's modular.
06:32It's easy to modify.
06:33I feel good about the changes that I'm pushing into production.
06:36Nope.
06:36Our change confidence number has gone down 6%.
06:40This is really an interesting tension because it means like, okay, agents and assistants are making it easier for me to understand and modify the code that's in front of me, but I trust the outputs less.
06:49I'm more afraid now of breaking things than I was a year ago.
06:54So, it's an interesting psychological effect that we see here.
06:57And a lot of this is because PR size is measurably increasing.
07:00This is going to be one of the most important metrics that we look at this year.
07:03Hasn't quite doubled, but look at that trajectory, right?
07:07This is over the same period of study about a year and we've seen PRs go from uh around 44 lines on average per PR up to 72.
07:17Now there's there's a number of reasons for this.
07:19It's not just, you know, the likelihood that the way that these models work is that they're be being trained to output relatively mediocre code.
07:26I've been writing code professionally since the late 90s.
07:29I remember when it was a great mark of a developer to be able to implement the same use case with as little code as possible.
07:34Now, it's like, well, no, let's get the thing out.
07:36And, you know, the code that's being generated is going to be essentially mediocre because it's law of averages.
07:40But on top of that, too, think about if you've got a build pipeline that takes 45 minutes or an hour to complete and now I've got AI generating functions for me instantly.
07:50Am I going to push four different PRs with four different functions and wait 45 minutes or an hour or whatever that is per build or I'm just going to cram all that code into a single PR?
07:59Right?
08:00So there's multiple reasons for this, but it really bears mentioning that like every extra line of code is a potential bug, a potential vulnerability.
08:07It's more to review.
08:08It makes the code less portable.
08:09So we need to pay attention to this metric as well because we're also seeing this associated with perception of incremental delivery.
08:16I get to work on small incremental changes and this is actually one of the lowest qualitative developer experience drivers that we've seen impacted over the last year.
08:24This number and this sentiment has gone down 10%.
08:28And incremental delivery is awful is is as we've learned also very important for uh for the way that we try to deliver in a way that's easy to roll back
08:37uh and where we can understand and review the changes that are being pushed through.
08:42[snorts]
08:42Okay.
08:43Uh what about demographic differences?
08:45Uh what is different about AI impacts across like you know junior devs and stuff like that?
08:50Junior engineers are using AI the most.
08:53This really should surprise nobody.
08:55anytime we have sort of a new leap again writing code professionally since the late 90s I'm a bit of an OG I'm on this journey too but there's less to unlearn
09:05when you're coming into the industry right in many many cases you've already been working with this kind of stuff like right out of school and so you're more likely to implement it
09:14uh when you actually start working now we've seen some other trends there too uh we have interesting ways of looking per use case at efficiency uh which I'll get into in the second talk later today and one of those is looking at agent experience
09:28like literally we're we're asking agents about their experience working with humans.
09:33We're trying to figure out what use cases uh are they working on and how many tokens did they spend.
09:38And definitely we also see a trend where the more junior developers for the same use case will spend more tokens than a more senior developer on the same use case.
09:46And that's just a learning curve, right?
09:47Again, that should surprise nobody.
09:49But in terms of who's using it the most, we do see juniors using it the most.
09:53However, despite that additional use, we see staff plus or more senior engineers saving about the same amount of time, right?
10:01And and burning less tokens.
10:02I want to make sure that that's kind of clear too.
10:04So in terms of like raw time savings in the aggregate output of this, it's pretty flat between junior engineers who are using the tech more and staff engineers who have an easier time spotting hallucinations,
10:14understanding the architecture around the changes they're making and things like that.
10:19Smaller companies lead in time savings.
10:22Sure, you know, the release pipelines are going to be less complicated.
10:25You know, there's less overall complexity in the organization.
10:28uh that's going to uh lead to more time savings for a company that already has less friction and has a little bit more agility to it than a larger
10:37company.
10:39What about measurement?
10:41How should we be even be thinking about measuring this stuff?
10:44I mean, it bears mentioning that that measuring developer productivity and measuring developer experience was a challenge before AI and we we never completed that conversation before kind of throwing accelerant on this.
10:56Right?
10:56So, this is still a difficult thing and I'd argue even more difficult to do now because it's confounded by several different aspects of of AI, but we're all going to have to answer this question this year, right?
11:07Spent 10 million or way more if you come to my talk later today on tokens.
11:11Uh where's our 10x productivity?
11:15When we're thinking about this stuff, it's important to bear in mind that we don't throw away what we're measuring, right?
11:21our foundational developer experience and developer productivity metrics are still what matter the most, right?
11:26We want to understand how a AI is impacting these trusted metrics that we've already been looking at.
11:32Like we want to maybe look at cohorts of users.
11:35We maybe want to take data from the API telemetry and things that come out of the tools and use that to understand who's using what and where.
11:43But we would only want to use those cohorts and look at them comparatively across our foundational metrics.
11:48What is this actually doing to quality?
11:50What is this actually doing to speed?
11:51What is this doing to impact to the organization and value generation?
11:56And so this is where our AI measurement framework comes from.
11:59I won't spend too much time on this.
12:00We have a big white paper available.
12:02Uh my colleague Anton who's standing in the back of the room in the the white shirt will be able to hand out some hard copies of reports that we have after the talk that go into the way that we derive this methodology and how to use it and think about it.
12:14Um it focuses on three key dimensions of measurement.
12:17So looking at utilization, our daily active users, weekly active users, that sort of thing.
12:22Uh it looks at impact.
12:24So which metric should we be, you know, looking for moving the needle based on utilization?
12:28Uh as our utilization goes up, what do we hope to see in terms of impact to the business?
12:33And then cost.
12:34It's a fair joke that we're 15 years after the last major hype cycle and we're still trying to figure out cloud cost, but the stuff is getting pretty expensive.
12:41We're going to have to measure it.
12:43Um so you can think about this as a bit of a maturity curve as well.
12:45Most people start with utilization on the left.
12:48Just figuring out what's happening in the organization with the tech, who's using it, what are they using, how often, and for what use cases.
12:53And then how do we then cross reference those to the impact metrics, the value generation, what we really want to understand to know whether these investments are actually working, which is what we really want to figure out.
13:04This is just an example of how we might correlate such a thing.
13:07So you could look at two separate cohorts of users and you could compare them across metrics like PR cycle time or PR size or push back and review.
13:16Right?
13:16So lots of uh the way to think about this again is to right now separate cohorts and then look at how that plays out across these foundational productivity metrics that we've learned to trust.
13:27We also need to able be able to understand our platform's readiness.
13:31I like to say that in 2024 we gave everybody a coding assistant.
13:34In 2025 we started building agents.
13:36Now, we're realizing that our infrastructure wasn't ready for any of this.
13:40A lot of the vendors that are here are are are selling solutions that help you with this problem.
13:44Um, I think if you can measure your platform's AI readiness, you're going to have a much better handle on how efficient you're going to be spending tokens and providing context to these agents.
13:53So, these are things like clear and accurate, well structured documentation, data structures with straightforward relations, manageable modular code, reliable local CI and non-flaky test suites.
14:03If by the way any of this sounds familiar to you, it's because we used to just call this good developer experience, right?
14:09But it turns out that what's good for humans is also good for agents.
14:12So we may finally paradoxically be making those investments that we should have been making over the last couple of decades.
14:18We also want to understand the effectiveness of the way that we work with AI.
14:22And this is one way of doing that.
14:24What you're actually seeing here is feedback, qualitative feedback coming back from agents telling us where they ran into issues with steering with the human.
14:32the the context that was provided for them, the the feedback cycles that they had to go through.
14:37So, we're actually getting that data from the agents to try to figure out how effective our teams are at working with AI.
14:42And we can break this down into use cases, which is very important, too.
14:46We want to understand which use cases are giving us the most efficient token spend.
14:51And again, moving the needle for value creation and productivity within the business.
14:56Finally, the fifth trend, integrating across the SDLC and PDLC.
15:00Why would we want to do this?
15:01Because code generation was never the bottleneck in the first place, right?
15:04Even if engineers are getting like 100% accurate instant code coming from the models, which they are not, you would still only be attacking anywhere from maybe 14 to 16% of the overall
15:15value stream.
15:15So we need to think beyond that.
15:17Uh especially because right now the productivity gains that we are actually seeing are more modest than expected.
15:23Despite additional PR throughput uh increases, our median increase in the study that we did from November to to 2024 to February this year only found about a median 7.7%
15:35increase in this velocity metric, a 13% average, but even our top performers were in the 70% range.
15:40Nobody hit 2x, nobody hit 5x, nobody hit 10x.
15:44And that's because our time savings that come out of AI are still being outweighed by other nonAI factors within our organization, right?
15:53Right?
15:53We might be saving a lot of time with AI and that's great.
15:56But if we're still eclipsing those time savings because of meeting heavy days and context switching and other sources of interruption, the cumulative effects of the dev environment and friction in the way that we create software around the code, then we're not attacking the bottleneck.
16:09And as Ellie Goldroat from the theory of constraints and the goal and uh inspire inspiration for the Phoenix project would tell us that an hour saved on something that isn't the bottleneck is worthless.
16:18And so we need to find the bottleneck and fix the bottleneck.
16:21And that's what a lot of organizations are doing.
16:24Morgan Stanley has been really public about their creation of the DevGen AI agent, which is an agent for interpreting legacy code like uh mainframe natural and cobalt.
16:33And I hate to admit Pearl.
16:34I've written a lot of Pearl in my career, but it's legacy now.
16:37The agent actually creates PRDS to hand to engineers to get rid of that reverse engineering step.
16:43And it's saving Morgan Stanley about 300,000 hours a year right now.
16:47Zapier is one of my favorite stories in AI right now.
16:49They have a whole agent ecosystem that they're using to deal with a lot of administrative tasks and overhead like daily standups and things like that.
16:57They've reduced their standups by using a agent summaries and things like that in this ecosystem that they've built to two times a week as opposed to five times a week.
17:06They're successfully onboarding engineers within about a twoe period which is really fast.
17:10Our industry benchmark for that is over a month usually.
17:14But my favorite part about this story is that they're seeing about a 15% additional value creation per engineer.
17:20So they are hiring more than they have in the history of their entire company because they realize the truth here that they're getting more capacity per single engineer.
17:28They're getting a better return on investment per single engineer and hiring more engineers will improve their competitive edge.
17:34This is the right attitude.
17:36This is a throughput story.
17:37This is an increased innovative capacity story.
17:39This is not a headcount replacement story.
17:42Fair.
17:42uh they're automating about 3,000 of their code reviews a week triggered off of a pull request and um they look at superficial stuff.
17:50They still need humans in the loop, but the initial superficial uh uh review that comes from the agent is all part of the system of record.
17:57It stays in the PR comments and so the next engineer to actually perform the review will have an idea of what the agents already looked at.
18:03So it can save some time.
18:05Finally, Spotify, if anybody remembers the Spotify model from DevOps, they were sort of the northstar for DevOps.
18:11uh they built an agent for S SRRES that will gather remediation steps from runbooks and information and context about the incident and put all that together into SR communication channels so that when an incident occurs
18:24the S sur has uh immediate context and doesn't have to go through multiple minutes of discovery uh to figure out how to solve the issue.
18:31So, if you want to dive deeper into this data, our Q2 report is coming out in just a few weeks.
18:36Uh, but the Q1 report still has a lot of tidbits from this session in it.
18:40And if you subscribe to our newsletter or you come back to the website, by midish end of July, we'll have our Q2 report.
18:46And if you want to see a preview of that report, that's the talk that I'm giving later this afternoon.
18:50Thanks everybody for your time.
18:51I appreciate it.
18:59[music]
19:05[music]