WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=-X3yXTwGs9A NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/-X3yXTwGs9A.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.440 --> 00:00:18.200 Um welcome to AI's trust layer for Wall Street, SRE and platform engineering. s3 00:00:18.200 --> 00:00:19.560 My name is Andrew Norish. s4 00:00:19.560 --> 00:00:22.200 I'm the director of field engineering at Sourcegraph. s5 00:00:22.200 --> 00:00:33.600 Our field engineers work on the front lines with our customers, collaborating closely with the software engineers, the other technical team members at some of the largest companies in the world, and helping them build and manage their code bases s6 00:00:33.600 --> 00:00:36.920 better using the Sourcegraph platform. s7 00:00:37.240 --> 00:00:47.000 If you haven't heard of Sourcegraph before, we build code search and code graph products that help the owners of the largest code bases and the most complex code bases in the world deeply understand s8 00:00:47.000 --> 00:00:48.640 those code bases. s9 00:00:48.640 --> 00:00:58.520 For the last year or so, on top of the humans that we've been supporting for the past almost 13 years, uh we've also been starting to build support for agents, s10 00:00:58.520 --> 00:01:00.640 and they've been our primary customers. s11 00:01:00.640 --> 00:01:08.840 Uh for those agents, uh all this code base tooling, code search, code graph, code navigation has become a code base context engine. s12 00:01:08.840 --> 00:01:19.120 Uh today, I do want to talk a little bit less about how Sourcegraph directly supports the hands-on keyboard work of developers building those code bases, and focus more on how AI-driven s13 00:01:19.120 --> 00:01:27.320 changes in software development have impacted frontline support for those services, and where Sourcegraph can help. s14 00:01:28.240 --> 00:01:31.480 I wanted to start today with a bit of story time. s15 00:01:31.480 --> 00:01:40.240 Um this is a story that we have heard time and time again from so many of our customers, especially over the past couple years with the rise of agentic coding. s16 00:01:40.240 --> 00:01:45.080 Uh this story always starts the same, with an increase in AI-driven development. s17 00:01:45.080 --> 00:01:47.120 I'm sure this is something Obviously, we're all here. s18 00:01:47.120 --> 00:01:48.600 That's why we're here. s19 00:01:48.600 --> 00:01:53.440 Um engineering teams increasingly rely on AI and agentic tools to support their work. s20 00:01:53.440 --> 00:02:00.852 These tools inevitably lead to significant increases in developer velocity, and code bases inevitably start ballooning. s21 00:02:00.852 --> 00:02:10.452 Um organizations dedicate huge budgets to AI tooling and expect developers to use them heavily and just to justify those purchases that they're making. s22 00:02:10.452 --> 00:02:18.772 Uh and for the the past couple of years, the first question on basically any AI roadmap was how many heads can this save us, right? s23 00:02:18.772 --> 00:02:26.092 Um in Sourcegraph's most recent customer advisory board though, that question for maybe the first time ever just didn't come up. s24 00:02:26.092 --> 00:02:29.332 So, the mindset is clearly shifting. s25 00:02:29.332 --> 00:02:32.772 That brings in code review as a bottleneck. s26 00:02:32.772 --> 00:02:36.692 We have started to hearing this time and time again from so many of our customers. s27 00:02:36.692 --> 00:02:52.412 Between the increase in developer velocity, the pressure to use tools purchased by the business and the expectation that product releases should increase proportionally to the use of these AI tools, the developer priorities have never inevitably started shifting. s28 00:02:52.412 --> 00:03:07.812 Where the bottleneck used to be software development and in software development used to be developer bandwidth and how quickly a human can just type code, that bottleneck is now very clearly with how quickly humans can review the code that the AI is now writing. s29 00:03:08.052 --> 00:03:12.532 As part of that, comes the rise of AI-initiated incidents. s30 00:03:12.532 --> 00:03:16.532 AI isn't just writing code faster, it's writing so much more of it. s31 00:03:16.532 --> 00:03:20.252 PR counts have increased as PR sizes have ballooned. s32 00:03:20.252 --> 00:03:30.812 Uh and whether or not developers actually trust the code that they're pushing now matters a whole lot less than just getting ahead of an ever-growing list of PRs that they need to review. s33 00:03:30.972 --> 00:03:43.212 AI-written PRs often lacking the organizational and in larger code base context needed to actually be production-quality gets merged into production without a human ever seeing it more often than not now. s34 00:03:43.212 --> 00:03:48.532 It matters it's only a matter of time before breaking changes are missed and merged by code reviewers. s35 00:03:48.532 --> 00:03:54.092 And AI as you AI usage increases, so do AI-caused incidents. s36 00:03:54.092 --> 00:03:57.052 Enter the SREs becoming stretched. s37 00:03:57.052 --> 00:04:06.174 More and more SREs are required to resolve larger volumes of incidents without runbooks and code that maybe a human has never even looked at before. s38 00:04:06.174 --> 00:04:15.254 Uh the context were required to resolve these incidents is lacking again because a human has never seen it and it's even harder to track down than it was before. s39 00:04:15.254 --> 00:04:17.454 These incidents are becoming more frequent. s40 00:04:17.454 --> 00:04:18.934 They take longer to resolve. s41 00:04:18.934 --> 00:04:20.734 They pop up on a recurring basis. s42 00:04:20.734 --> 00:04:27.534 They're costlier and the burden just continues to fall to our front-line support to resolve them. s43 00:04:27.534 --> 00:04:35.654 Leadership has started looking to hiring more SREs as as a band-aid solution, but the role is increasingly in more and more high demand. s44 00:04:35.654 --> 00:04:43.174 It's getting more expensive and the rise of incidents is quickly starting to outpace the ability to hire people to resolve them. s45 00:04:43.174 --> 00:04:47.414 Again, this is something that we have started hearing time and time again from our customers. s46 00:04:47.414 --> 00:04:51.574 Uh raise raise your hand if if this is something you guys are feeling. s47 00:04:51.574 --> 00:04:53.574 Yeah. s48 00:04:54.054 --> 00:04:55.894 So, a couple of patterns that we've started to seeing. s49 00:04:55.894 --> 00:04:59.374 Obviously, I mentioned review as the new bottleneck. s50 00:04:59.374 --> 00:05:02.654 Um we've started to hearing this a lot from from a lot of our customers. s51 00:05:02.654 --> 00:05:11.934 Not only is AI responsible for writing more of the code, but because of a lack of trust in that AI written code, the developers are expected to spend more time reviewing it. s52 00:05:11.934 --> 00:05:23.494 Um organizations more and more expect developers to spend their time performing code review to verify that this untrusted code written by the AI meets their standards, the regulations, the requirements of their organization. s53 00:05:23.494 --> 00:05:26.574 That being said, I mean of course AI still hallucinates. s54 00:05:26.574 --> 00:05:27.774 It's getting a whole lot better. s55 00:05:27.774 --> 00:05:28.294 We all love it. s56 00:05:28.294 --> 00:05:31.374 That's why we're here, but it's not perfect. s57 00:05:31.374 --> 00:05:47.494 And um this is particularly true in really large, really complex organizations maybe with a whole lot of of legacy code where the AI just doesn't have all of the context needed to actually build the product in the way that your organization might need it to. s58 00:05:47.494 --> 00:05:53.534 If you can't clone the entire codebase locally, the AI just needs to make judgment calls and make guesses. s59 00:05:53.534 --> 00:06:01.814 Sometimes they're informed, but often it just means they uh only have a partial view of the environment that they're building in. s60 00:06:01.814 --> 00:06:07.975 As a result, incidents increasingly have started tracing back to unreviewed AI code. s61 00:06:08.095 --> 00:06:17.055 And that burden again falls to the responders uh who often at times have a lack of context in the code that they're responding to the incident for. s62 00:06:17.055 --> 00:06:21.135 Uh agents and engineers alike fail on things like cross-service problems. s63 00:06:21.135 --> 00:06:31.575 If it's difficult to track down the root cause of the these cross-service issues, organizations start falling back to slow and air-prone manual solutions to resolve them like commit scanning. s64 00:06:31.775 --> 00:06:36.055 And as a result of that, remediation just simply cannot keep pace anymore. s65 00:06:36.055 --> 00:06:40.935 A fix found in one repo often applies to hundreds if not thousands of others. s66 00:06:40.935 --> 00:06:51.935 It If disparate teams are relied upon to apply those fixes within the repos, the fix can't go out quick enough more often than not um and things start getting missed. s67 00:06:53.255 --> 00:06:57.495 All of this is even more true in the industries that you all are working in. s68 00:06:57.495 --> 00:07:06.775 Um when you're working in a financial institution, the stakes are just a whole lot higher and you have the responsibility for meeting all of those regulatory and compliance requirements. s69 00:07:06.775 --> 00:07:19.175 Uh I don't want to spend too much time on these next numbers that I have on the next few slides, but just to kind of show you how some of our our industry peers are thinking about this and some of the analytics that they've they've done. s70 00:07:19.175 --> 00:07:21.775 Um I'll pull up a few. s71 00:07:21.775 --> 00:07:23.775 Ooh. s72 00:07:24.215 --> 00:07:37.135 Well, this slide just deleted itself, but um yeah, this uh this slide was basically just demonstrating the expectations for how much new code is going to be written by AI come 20 s73 00:07:37.135 --> 00:07:37.695 27. s74 00:07:37.695 --> 00:07:39.215 I'm sure some of you have seen the number. s75 00:07:39.215 --> 00:07:44.095 It's about 2/3 of of code now um is the expectation. s76 00:07:44.095 --> 00:07:45.895 Oh, there it goes. s77 00:07:45.895 --> 00:07:58.135 Um as part of that, the code review challenge that I mentioned earlier, um so some organizations are estimating that almost twice the number of PRs are being merged now with agentic coding. s78 00:07:58.135 --> 00:08:09.297 The size of those PRs are is more than double the size of just a human-written PR um and that means developers are starting to spend almost twice as much time just doing code review. s79 00:08:09.297 --> 00:08:22.537 Um On top of that, the developers are feeling that more often than not AI-written code is actually more difficult to review uh because it wasn't written by or necessarily always for a human. s80 00:08:23.297 --> 00:08:26.257 Trust in AI code is still lagging tremendously. s81 00:08:26.257 --> 00:08:31.377 Almost every developer will say that they don't trust everything that AI is writing for them. s82 00:08:31.377 --> 00:08:38.217 And surprisingly at the same time, almost half of developers will say they don't always verify the code that AI is writing for them. s83 00:08:38.217 --> 00:08:41.537 Kind of scary for an enterprise, right? s84 00:08:42.097 --> 00:08:45.457 As a result, quality and security decline, like I mentioned. s85 00:08:45.457 --> 00:08:52.297 Um more incidents per PR, more incidents being traced back just to AI-generated PRs as well. s86 00:08:52.297 --> 00:08:55.217 And stability declines as those incidents rise. s87 00:08:55.217 --> 00:09:00.257 Um more deployment-related issues are being traced back to AI-generated code. s88 00:09:00.257 --> 00:09:06.697 And there's been an increase in things like rollbacks, hot fixes, customer-impacting incidents as a result of it. s89 00:09:06.697 --> 00:09:17.457 All of that at the same time, Anthropic has started reporting that because developers are becoming more and more reliant on AI to write the code for them, their comprehension of that code is also s90 00:09:17.457 --> 00:09:19.657 at a decline. s91 00:09:20.657 --> 00:09:25.577 Enter why I'm here today, the rise of SRE and platform team toil. s92 00:09:25.577 --> 00:09:35.457 More code being written by AI, lagging trust in that code, inability to keep pace with code review increases, and declining quality and security leads to a large increase in customer-facing incidents. s93 00:09:35.457 --> 00:09:46.097 That lands on the heads of all of our front-line support folks who are then asked to fix issues that, like I mentioned, a human may or may not have ever even looked at before. s94 00:09:46.337 --> 00:09:48.897 Um I don't want to be up here just doom and gloom. s95 00:09:48.897 --> 00:09:51.977 I I know I've spent a whole lot of time on on negatives here. s96 00:09:51.977 --> 00:10:03.737 Um but yeah, at least as of today, where AI is actually winning, it's not necessarily on this like end-all be-all broad automations of developing code, right? s97 00:10:03.737 --> 00:10:06.497 If it were, then we would all be out of jobs. s98 00:10:06.497 --> 00:10:16.818 Um AI is winning a whole lot more though when you can narrow its focus to well-bounded domains with all of the necessary context to meet your organizational and compliance requirements. s99 00:10:16.818 --> 00:10:31.338 Things like vulnerability remediation, security scanning, large-scale migrations, non-developers prototyping their their own tools, things like that with really bounded domains and bounded ideas um are where it starts to succeed. s100 00:10:31.338 --> 00:10:36.858 And that's where I think SREs and platform teams come in a whole lot more. s101 00:10:36.858 --> 00:10:46.298 Um SREs and platform teams have been working in these well-bounded, well-defined, semi-autonomous areas for a very, very long time at this point. s102 00:10:46.298 --> 00:10:49.658 Obviously, way, way before AI was even around. s103 00:10:49.658 --> 00:10:59.698 And because of that, they're now both kind of the barometers for how effective AI coding is and how trustworthy it is within your organization by how many incidents that AI code is producing. s104 00:10:59.698 --> 00:11:06.018 Um but they're also the teams that can see some of the highest gains uh from effective AI usage. s105 00:11:06.018 --> 00:11:14.978 Uh they already work with automation automated remediation tooling, semi-autonomous incident response runbooks, CVE monitoring and remediation. s106 00:11:14.978 --> 00:11:26.938 Um so, those teams build the processes and tooling to prevent AI-initiated incidents from becoming rampant, but ironically, they're also the teams that can leverage AI within their semi-autonomous existing processes s107 00:11:26.938 --> 00:11:30.498 to see real productivity and safety gains. s108 00:11:31.738 --> 00:11:35.658 Um want to go back to Sourcegraph, what my team and I do. s109 00:11:35.658 --> 00:11:41.498 Um as I mentioned, we're a code intelligence code understanding, code search, code navigation platform. s110 00:11:41.498 --> 00:11:51.698 Um and much of our tooling is built first and foremost to prevent some of those AI-introduced incidents from occurring in the first place, cuz that would be ideal. s111 00:11:51.698 --> 00:12:02.938 Uh tools for developers and agents to search through code efficiently and accurately, understand that code deeply, and use that knowledge to better build and more robust services. s112 00:12:02.938 --> 00:12:08.578 That same tooling is also really, really effective for issue identification and resolution as well. s113 00:12:08.578 --> 00:12:15.739 And in the context of SREs and in platform team, that's where I want to focus for the rest of this talk. s114 00:12:16.299 --> 00:12:22.299 Just to give you a high-level overview of kind of what Sourcegraph is, some of our features. s115 00:12:22.299 --> 00:12:32.779 Um at our core, we have a full code graph, a complete index of of your organization's entire codebase uh that really powers kind of all of our features and everything that we do. s116 00:12:32.779 --> 00:12:36.859 The next level up from there, as I've mentioned a few times, is code search and code intelligence. s117 00:12:36.859 --> 00:12:48.859 These tools that we've spent over a decade building and tuning to give human and agents the ability to navigate, especially really large and and complex codebases extremely quickly and efficiently. s118 00:12:48.859 --> 00:12:55.779 Uh the graph, search, and intelligence form kind of the context engine that I mentioned for agents specifically. s119 00:12:55.779 --> 00:13:09.539 Uh they're accessible via our LMT P, and they really power at our core all of our own AI and agentic-enabled features uh that ensure humans and agents have the information and the context they need to build code better. s120 00:13:11.579 --> 00:13:22.579 Um So, when I'm thinking about SRE specifically and how Sourcegraph can help SREs, uh I kind of broke it down into these three areas. s121 00:13:22.579 --> 00:13:26.659 And obviously, these are at the core of any SRE workflow. s122 00:13:26.659 --> 00:13:29.499 Investigation, prevention, and remediation. s123 00:13:29.499 --> 00:13:36.899 On the investigation front, with Sourcegraph, we have the tooling to find the root cause of cause of issues in minutes instead of days. s124 00:13:36.899 --> 00:13:43.899 Uh these issues that are being created by uh AI-written code uh are much easier to investigate, find. s125 00:13:43.899 --> 00:13:51.339 Uh our classic deterministic search allows you to search across every repo repo, every branch, every commit with just a single query. s126 00:13:51.339 --> 00:14:00.779 And then deep search, our identic search feature, allows you to easily ask what's broken, get cited answers, and be confident in the responses that that you're receiving. s127 00:14:01.179 --> 00:14:11.539 On the prevention front um it's obviously great to be able to find the root cause of a lot of these issues really quickly, but it's even better to prevent them from occurring to begin with. s128 00:14:11.539 --> 00:14:21.941 Um at Sourcegraph, we have built-in monitoring capabilities to alert when risky code is committed, when certain patterns are matched that may be non-compliant, things of that nature. s129 00:14:21.941 --> 00:14:30.501 And through these monitors, you can ensure that AI-written code follows all of those patterns and requirements that you and your organization have in production. s130 00:14:30.501 --> 00:14:32.701 And then lastly, remediation. s131 00:14:32.701 --> 00:14:41.141 Once you've even uh identified a break or a risky pattern, how do you make sure that that fix is applied quickly and to all impacted areas of your code? s132 00:14:41.141 --> 00:14:51.221 Uh Sourcegraph tooling allows you to find all those impacted areas like I've mentioned, but we also have the ability to make bulk changes across all of your repos within your entire organization s133 00:14:51.221 --> 00:14:58.701 to remediate them and make sure that they don't occur across other services with other teams, whatnot, very, very quickly. s134 00:14:59.101 --> 00:15:09.861 Uh I With the remaining time here, wanted to walk through just a couple of examples of how we at Sourcegraph and some of our customers have applied some of these principles. s135 00:15:09.861 --> 00:15:15.781 Uh this first example for security triage is actually something that we've implemented internally at Sourcegraph. s136 00:15:15.781 --> 00:15:22.381 We have tons of these little workflows set up to help us maintain our codebase and make it more autonomous. s137 00:15:22.381 --> 00:15:27.981 Um our security team built this sort of workflow organically really a while ago. s138 00:15:27.981 --> 00:15:33.221 Um we've been using HackerOne for a while now, even pre-ChatGPT. s139 00:15:33.221 --> 00:15:39.261 Uh but the volume of the reports that our security team has been receiving has been skyrocketing, frankly. s140 00:15:39.261 --> 00:15:48.661 Uh and they require real-time judgment to review, digest, and decide, like, is this something that the team actually needs to spend much time on? s141 00:15:48.661 --> 00:16:02.621 Uh so they set up uh from HackerOne webhooks that reach out to our agentic searching capabilities, and then make judgments on whether or not our team needs to dive deeper in it, maybe propose a change or remediation, s142 00:16:02.621 --> 00:16:09.141 if it just needs to be reviewed more deeply by our security team, or if it's something that we can ignore completely. s143 00:16:11.101 --> 00:16:13.581 Another example from one of our customers. s144 00:16:13.581 --> 00:16:17.861 Um this customer has over 5,000 developers within their enterprise. s145 00:16:17.861 --> 00:16:25.542 Uh and they wanted to leverage some of Sourcegraph's code understanding capabilities to automate some of their incident response. s146 00:16:25.542 --> 00:16:36.582 Um how they did that when there's a service record downtime or significant spike in their error rates, the internal root cause analysis agent that they've built within their organization automatically s147 00:16:36.582 --> 00:16:42.502 spins up, takes all the logs and traces, and connects those back to the source of the fault. s148 00:16:42.502 --> 00:16:48.862 First, you have to identify what branch or tag was deployed in production because it's rarely main, of course. s149 00:16:48.862 --> 00:16:55.742 And then using a agentic search over the Git history to find the specific commit and author that triggered the incident. s150 00:16:55.982 --> 00:17:05.222 The agent then has the ability to revert and kick off a redeployment or simply triage and share the investigation with the SRE team SRE team. s151 00:17:06.102 --> 00:17:11.942 Last example I have, um this is another customer that had a really great internal use case. s152 00:17:11.942 --> 00:17:14.902 Um they maintain their own internal code review agent. s153 00:17:14.902 --> 00:17:18.022 It's used by thousands of their own developers, as well. s154 00:17:18.022 --> 00:17:24.622 Um and they custom built it to be unique and applicable to their specific code base and specifications. s155 00:17:24.622 --> 00:17:31.982 Uh in this case, a massive mono repo that more often than not, pre-Sourcegraph, they were having to just grep every time. s156 00:17:31.982 --> 00:17:36.502 Um their eval suite is made up of hundreds of historical branches of that code base. s157 00:17:36.502 --> 00:17:49.222 And so their agents needed the ability to search across all of those branches um instantly, affordably, at scale without relying on the non-determinism of some of the the other agentic tools available to them. s158 00:17:49.222 --> 00:17:52.102 Um so they used our read-only code finder agent to make that possible. s159 00:17:52.102 --> 00:17:58.862 And when the evals pass, they push and publish processes are are are kicked off. s160 00:18:01.822 --> 00:18:12.822 Yeah, so all in all, um if you or your SRE team or if you your SRE teams, your platform teams are really starting to see a lot of the burdens s161 00:18:12.822 --> 00:18:24.823 of incidents and the rise of incidents caused by AI-generated code, um if your teams don't really have a clear way and effective way for investigating uh what caused those incidents s162 00:18:24.823 --> 00:18:34.903 and then coming up with a very quick remediation plan to address the increasing volume of those incidents within your code base or really if your SREs, platform teams, software engineers s163 00:18:34.903 --> 00:18:47.303 uh are looking for better improved uh context engine code understanding platform that they use or they would like to use with all of their agentic tooling please feel free to come s164 00:18:47.303 --> 00:18:49.903 talk to us at our booth. s165 00:18:49.943 --> 00:18:52.143 Thanks, y'all.