WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=JJGbw4ggaFs NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/JJGbw4ggaFs.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.160 --> 00:00:02.480 Hi everyone. s2 00:00:02.480 --> 00:00:05.240 I don't know if you're familiar with what I'm about to show. s3 00:00:05.240 --> 00:00:12.400 But remember when someone from the product is like saying that some page is too slow and maybe we can optimize it. s4 00:00:12.400 --> 00:00:17.800 And then someone from engineering would say, "Yeah, probably what we'd have to dig in to find out." s5 00:00:17.800 --> 00:00:21.880 So, we just kind of leave it as is. s6 00:00:21.880 --> 00:00:26.760 And then a few weeks later, Jenny would say, "No, no, no, but it's actually way too slow now. s7 00:00:26.760 --> 00:00:28.400 We have to prioritize it. s8 00:00:28.400 --> 00:00:30.280 How long is it going to take?" s9 00:00:30.280 --> 00:00:34.840 And we would say something like, "Well, somewhere between an hour and a week. s10 00:00:34.840 --> 00:00:38.560 We'd have to look at it to find out." s11 00:00:38.560 --> 00:00:42.480 And then when she asked who can even take it, the answer is only Dave. s12 00:00:42.480 --> 00:00:50.320 He's the only one who kind of knows that code and everyone else who wrote it left a decade ago. s13 00:00:50.400 --> 00:01:04.280 And amazingly, this still happens in teams all the time because we would know how much time it takes to do a specific optimization, but we never know how long it's going to take to investigate it and to find it. s14 00:01:04.280 --> 00:01:16.280 And miraculously, every time we look, we we actually find things that can be done around performance and stability, but we never stop to proactively look for them because it doesn't make sense. s15 00:01:16.280 --> 00:01:17.000 So, hi. s16 00:01:17.000 --> 00:01:19.480 I'm Mike, co-founder and CTO at Thundra. s17 00:01:19.480 --> 00:01:31.040 We're building a runtime intelligence layer for coding agents that captures functional level context and deep forensic context on things that matter so that coding agents can help you fix what's going on with production. s18 00:01:31.040 --> 00:01:39.680 And as a part of that, we built an agentic workflow that helps with continuously optimizing performance of production applications. s19 00:01:39.680 --> 00:01:50.000 And I'm going to share a bit more about what we built, what were the challenges along the way, and hopefully you can take something out of it and apply it in your day-to-day. s20 00:01:50.000 --> 00:01:57.600 So, we're surrounded by a lot of agentic PRs that all look kind of good to us. s21 00:01:57.600 --> 00:02:04.440 And one agent wrote them and the other one says, "Yeah, I I went over it and it looks fine." s22 00:02:04.440 --> 00:02:12.200 And we still feel that urge to to verify before we just merge it to production. s23 00:02:12.200 --> 00:02:18.440 And then when it doesn't work, we end up asking ourselves, "How the hell did we get to this place?" s24 00:02:18.440 --> 00:02:21.600 And if you feel like that, I want you to know that you're not alone. s25 00:02:21.600 --> 00:02:34.920 So, Google just published their Dora metrics for 2026 and we can see that the biggest impact of AI adoption on engineering is individual effectiveness or that feeling of, "Oh my god, I'm so fast. s26 00:02:34.920 --> 00:02:36.959 I can do everything in the world." s27 00:02:36.959 --> 00:02:41.959 And for me personally, lack of sleep is another symptom of that. s28 00:02:41.959 --> 00:02:45.160 But the second one is software delivery instability. s29 00:02:45.160 --> 00:02:49.240 And the throughput is actually not impacted as much as we expected. s30 00:02:49.240 --> 00:02:59.640 So, we basically feel more effective, we're more effective individually, but as a team, our throughput is kind of the same and our software breaks more often, which is not exactly s31 00:02:59.640 --> 00:03:04.320 what we were hoping for with this AI revolution just yet. s32 00:03:04.320 --> 00:03:14.239 But maybe there's an agent for that and if we can build faster with AI and we can fix faster with AI, then we can get those gains that we were talking about. s33 00:03:14.239 --> 00:03:17.480 So, I'm going to start with why we even wanted to do this. s34 00:03:17.480 --> 00:03:22.640 Then we're going to go over the tech and the process, which I think is also really important. s35 00:03:22.640 --> 00:03:26.680 And then share some gotchas and takeaways along the way. s36 00:03:26.680 --> 00:03:30.560 So, that leaks kind of faster than we can bail. s37 00:03:30.560 --> 00:03:41.720 We have these issues that we ignore because they're not important enough then they degrade to the point where they are important enough and then we reach this crisis mode where we are all hands-on-deck. s38 00:03:41.720 --> 00:03:49.760 We fix it in emergency mode and then we go straight back to ignoring, which is a leaky bucket by definition. s39 00:03:49.760 --> 00:03:55.360 And that mostly happens because the research phase is a black box. s40 00:03:55.360 --> 00:04:05.480 It could take an hour or weeks and we have to pay that debt and to make sure that we spend time and engine engineering time on it in order to even know s41 00:04:05.480 --> 00:04:07.200 what can be done about it. s42 00:04:07.200 --> 00:04:07.840 And that's hard. s43 00:04:07.840 --> 00:04:13.880 It's legitimately hard to prioritize something where you're not sure exactly what you're going to get out of it. s44 00:04:13.880 --> 00:04:16.760 But what if we can automate that investigation? s45 00:04:16.760 --> 00:04:27.360 So, we can basically run on a weekly basis with real production context, analyze the sweet spot in the and flag the high ROI opportunities in a way that's scored. s46 00:04:27.360 --> 00:04:31.800 So, it runs automatically without us having to stop and do something about it. s47 00:04:31.800 --> 00:04:41.160 It has the production context in mind and it can give a high ROI scored performance opportunities of the things that are easy and impactful. s48 00:04:41.160 --> 00:04:46.720 Kind of like that performance sprint that you run every few months, just automate it. s49 00:04:46.720 --> 00:04:54.520 Now, let's talk about how we can actually do it cuz the dream is very nice, but but the devil's in the details. s50 00:04:54.520 --> 00:05:05.240 So, first of all, we wanted an infrastructure for the agentic workflows that would be vendor neutral in terms of compute and where it runs, in terms of the harness and in also in terms of the model. s51 00:05:05.240 --> 00:05:06.760 Things are changing all the time. s52 00:05:06.760 --> 00:05:12.480 We wouldn't want to constrain ourselves to a specific vendor, specific model or anything like that. s53 00:05:12.480 --> 00:05:19.520 And especially like with hot, we want our customers to [clears throat] be able to use whatever agents they wish. s54 00:05:19.520 --> 00:05:24.120 We also wanted it to be secure in terms of the tool calls, permissions, and authentication. s55 00:05:24.120 --> 00:05:29.919 We want some trigger system, whether those are scheduled runs like the weekly run or a set of webhooks. s56 00:05:29.919 --> 00:05:34.160 For example, if we see some SLO breach, we would want to investigate it. s57 00:05:34.160 --> 00:05:38.840 And we really wanted it to be easy to maintain and update over time. s58 00:05:38.840 --> 00:05:47.880 I think one of the biggest learnings we've had with agentic workflows is that even if they work out of the box or we get to a point where we're happy about them. s59 00:05:47.880 --> 00:05:50.960 In time, we evolve and our expectations go up. s60 00:05:50.960 --> 00:06:01.680 So, just being able to maintain and update those logics and build that feedback loop was very important for us so that it's reliable and the people actually trust the outcome. s61 00:06:01.680 --> 00:06:15.800 Specifically, we chose to work with GitHub agent workflows for that and we can choose whatever agent we want to work with and build the workflows on top of that, but there are many other great tools that could be used for that. s62 00:06:15.800 --> 00:06:18.000 So, this is basically how it looks like. s63 00:06:18.000 --> 00:06:31.480 You can see that there's a description of the job and the goal and the analysis and then we basically go over the GitHub repository and generate that weekly deep insight report analyzing production data s64 00:06:31.480 --> 00:06:36.040 and finding those low-hanging high-ROI opportunities. s65 00:06:36.040 --> 00:06:57.280 So, for this specific setup, GitHub actions runs weekly and it uses Cloud Code, that was our specific choice, and then captures the runtime intelligence overhead via MCP so that we can basically look at the different endpoints, connect to the function level of overhead is there, and send a report to Slack. s66 00:06:57.280 --> 00:07:01.000 Of course, that could have been Teams or an email or anything like that. s67 00:07:01.000 --> 00:07:03.280 It could have been Cursor or Copilot. s68 00:07:03.280 --> 00:07:16.800 It's just the setup that we started with um to make sure that we have something that runs without us in the loop and sends that report to somewhere we actually live in, which is Slack. s69 00:07:16.800 --> 00:07:31.800 So, we want to take the production context, the traces, the queries, the latencies, and analyze them with the agent, score and flag which opportunities matter because if you can optimize something, but it runs every 3 weeks or you can s70 00:07:31.800 --> 00:07:36.120 like reduce 20 milliseconds, then it doesn't matter. s71 00:07:36.120 --> 00:07:40.360 And then the most important part here is the different the evidence. s72 00:07:40.360 --> 00:07:50.440 So, the agent actually fixes it reruns the tests and sees the impact that it had on that specific flow that was optimized. s73 00:07:50.440 --> 00:07:54.400 So that the human gets something after we already detected it. s74 00:07:54.400 --> 00:07:56.280 We understood the root cause. s75 00:07:56.280 --> 00:08:04.120 We understood why it matters to the business and we verified that the fix actually impacted that time. s76 00:08:04.120 --> 00:08:06.600 And then a human reviews that. s77 00:08:06.600 --> 00:08:08.000 And the loop is actually closed. s78 00:08:08.000 --> 00:08:10.600 So it's not hey, I have this idea of something that you can do. s79 00:08:10.600 --> 00:08:13.480 It's here's something that works. s80 00:08:13.480 --> 00:08:21.400 And we believe would make an impact on these specific business flows in production that are running 7,000 times a week. s81 00:08:21.800 --> 00:08:25.919 And then the human gets in the loop as a review gate. s82 00:08:25.919 --> 00:08:29.480 And of course it didn't work out of the box if you were wondering. s83 00:08:29.480 --> 00:08:35.919 So the first hurdle we had along the way is what we call the plausible unverified. s84 00:08:35.919 --> 00:08:37.680 So the agent would suggest a fix. s85 00:08:37.680 --> 00:08:39.200 It sounds right. s86 00:08:39.200 --> 00:08:46.400 It would look fairly real and then after we verified it it it just didn't work. s87 00:08:46.400 --> 00:08:52.000 And it's it's right that the agent suggested something that could theoretically cause that flow down. s88 00:08:52.000 --> 00:08:57.240 But what we wanted is to ground it on what's actually happening in production. s89 00:08:57.240 --> 00:08:59.400 Second part was complex queries. s90 00:08:59.400 --> 00:09:02.280 Um we specifically use ClickHouse. s91 00:09:02.280 --> 00:09:09.320 It's just an amazing columnar database but it's also slightly different than the classic SQL patterns. s92 00:09:09.320 --> 00:09:11.040 Now I'll talk about that in a bit. s93 00:09:11.040 --> 00:09:13.320 And the third one is the lazy fix. s94 00:09:13.320 --> 00:09:26.480 I'm sure you guys also experienced that when something throws an exception and then the agent says, "Well, maybe we can just catch that exception and then everything will be fine." s95 00:09:26.480 --> 00:09:36.440 But what we really want is to understand why that exception was even thrown in the first place or why are the results lagging. s96 00:09:36.440 --> 00:09:43.680 So those were things that we found that methodology could be very, very impactful with. s97 00:09:43.680 --> 00:09:48.920 It's not just the understanding of the data and how to connect it. s98 00:09:48.920 --> 00:09:59.520 It's also being quite thorough on what we want to do in that process and sort of building the playbook of how a senior engineer would do that. s99 00:09:59.560 --> 00:10:03.080 And of course, we need the context to be right. s100 00:10:03.080 --> 00:10:08.920 So, the problem with context there are also There are only two problems with context. s101 00:10:08.920 --> 00:10:12.840 You either have too much of it or you have too little of it. s102 00:10:12.840 --> 00:10:24.760 And what we found around runtime context, especially from production, is that more often than not, you actually have these two problems together because on one hand, you have a lot of low signal data s103 00:10:24.760 --> 00:10:26.960 that is hard for the agent to reason over. s104 00:10:26.960 --> 00:10:38.440 And on the other, you might not have all the logs and traces and metrics that you need in order to investigate that issue, which would still leave some room for assumptions and theories s105 00:10:38.440 --> 00:10:44.600 of that are not necessarily what our users are experiencing in production. s106 00:10:44.600 --> 00:10:54.680 And also, when we talk about metrics like service level, CPU and memory, or endpoints in the P90s, they are not connected to the function level. s107 00:10:54.680 --> 00:11:04.520 So, our coding agents reason over code and they look at these metrics and there are some relations between them, but they don't exactly speak the same language. s108 00:11:04.520 --> 00:11:15.839 So, when we ask questions about taking time and what can I do about it, we're often finding that there are some gaps there and again, accuracy could drop from that. s109 00:11:15.839 --> 00:11:28.600 So, what we did is what we call prod-to-code, which is to be able to explain what's going on in production in the same level that agents reason over because the agent's context s110 00:11:28.600 --> 00:11:34.200 lies on a function and file level, not on an service and endpoint level. s111 00:11:34.200 --> 00:11:44.440 So, basically, our context is running on a function level, and it is also connected to the endpoint or event consumer or cron job that ended up starting this task. s112 00:11:44.440 --> 00:11:52.880 So, you can ask a question like, "Hey, this endpoint that sometimes takes 7 seconds, where is the time spent, and what can be done about it?" s113 00:11:52.880 --> 00:11:58.880 Whether that's a function or an outbound call to a database, an LLM, or another microservice. s114 00:11:59.120 --> 00:12:10.000 And with that, you basically have the complete function level context for every single function, sort of like this service map, but on a function level of the different invocations, s115 00:12:10.000 --> 00:12:22.240 where they come from, how often they run, and the difference the context only when it's needed, only when we see requests that are taking longer than the P99 or some threshold that we can define. s116 00:12:22.240 --> 00:12:29.800 Then, we will capture those forensic evidence so that we can say, "Hey, here is an example of a request that took longer. s117 00:12:29.800 --> 00:12:31.960 Let's find out why." s118 00:12:31.960 --> 00:12:42.440 And we also have the ability to see that on top of the code, which is the HUD, the heads-up display, but but I guess it's kind of a way to explain s119 00:12:42.440 --> 00:12:51.560 how that data set is actually structured in a way that is much more comprehensive for a coding engineer to read and over. s120 00:12:51.560 --> 00:12:56.720 And then, we talked about the complex queries and what to do with them. s121 00:12:56.720 --> 00:13:07.280 So, the basic layer gave us the HUD query language, which are basically ClickHouse queries over that structure of functions and endpoints and forensics. s122 00:13:07.280 --> 00:13:10.320 On top of that, we also added a set of skills. s123 00:13:10.320 --> 00:13:23.040 We found that sometimes just querying the data is enough, but being able to get to the right query and to ask it again and again really created a lot of s124 00:13:23.040 --> 00:13:30.320 variance in our eval, and the skills actually help work with that data so that agents can use it. s125 00:13:30.320 --> 00:13:38.040 So for example, if we're talking about a 500, an HTTP 500, we would want to understand where that error came from. s126 00:13:38.040 --> 00:13:50.320 If we're talking about a memory spike, we want to understand what was running on those specific pods at that specific time where memory was higher and compare it to a baseline so that we see the diff. s127 00:13:50.320 --> 00:14:00.960 And all of these things were extremely helpful to be able to be a bit more methodological around how that works and to create more consistent results. s128 00:14:00.960 --> 00:14:03.560 And on top of that, there are a set of automations. s129 00:14:03.560 --> 00:14:15.520 So if we have the data, the query language, a set of skills, we can build automations on top of them like auto fixing issues as they arise or detecting s130 00:14:15.520 --> 00:14:31.160 dead code that isn't even running for the last 60 days and eliminating it or this automated performance improve- improvements automation that are we are talking about right now. s131 00:14:31.160 --> 00:14:43.960 So for performance, one example of that is to be looking for artificial delays like timeouts and sleeps and plus one queries, missing indexes, sequential assings and so on so forth. s132 00:14:43.960 --> 00:14:49.480 These are specific things that are much easier to find when you actually look for them. s133 00:14:49.480 --> 00:14:59.120 And in a code base that's 20 years old and has hundreds of contributors, it makes sense that you'll find quite a lot of those. s134 00:14:59.120 --> 00:15:05.040 And removing them is fairly easy and impactful. s135 00:15:05.040 --> 00:15:09.640 And then when you're asking something like, "Why are my endpoints taking long?" s136 00:15:09.640 --> 00:15:21.080 you can actually find the specific reasons and not just guess a bunch of static code analysis which could get you some result. s137 00:15:21.080 --> 00:15:23.040 I'm not saying it's never going to work. s138 00:15:23.040 --> 00:15:31.000 But, when you're talking about an automation, we have to think about how to build something that is robust enough for us to trust over time. s139 00:15:31.000 --> 00:15:40.480 And then we said, "Okay, now that our evas are looking good, we run weekly, we find real slow endpoints from production that are invoked. s140 00:15:40.480 --> 00:15:43.880 We score them, we test them, we verify them. s141 00:15:43.880 --> 00:15:49.760 Maybe we can just open pull requests and everyone will fix everything and the world will be amazing." s142 00:15:50.240 --> 00:16:03.080 Well, that's not exactly how that worked because people are still people and no one wants to wake up for, you know, a rain of 80 pull requests, as small as small as they can be. s143 00:16:03.080 --> 00:16:06.880 That's just not how people operate. s144 00:16:06.880 --> 00:16:08.120 And no one has time for that. s145 00:16:08.120 --> 00:16:11.320 We're too busy building other things and it's fine. s146 00:16:11.320 --> 00:16:20.600 So, what we actually do is we use that priority to make sure that we only flag the ones that matter and we actually started with one at a time s147 00:16:20.600 --> 00:16:23.240 to create that appetite and that habit. s148 00:16:23.240 --> 00:16:29.680 So, we look at whether this is a hot path in terms of how often it runs and how critical is it for the business. s149 00:16:29.680 --> 00:16:38.080 The business impact as in, you know, if this is something that has to do with payments or sign-up flows, obviously, we're more sensitive to that. s150 00:16:38.080 --> 00:16:43.640 Kind of like asking ourselves, would we be able to convince the product manager to prioritize it? s151 00:16:43.640 --> 00:16:46.920 And we also looked at the risk. s152 00:16:46.920 --> 00:16:56.440 If it's a risky change that requires a migration or anything like that, then obviously, it would require more time from the human that's reviewing it. s153 00:16:56.440 --> 00:17:04.920 And in that case, we're not necessarily looking for the highest impact ones, but for the highest ROI ones. s154 00:17:04.920 --> 00:17:18.560 And because we automate the investigation, we can look at the impact and the risk together and only surface and and require attention on the ones that we believe are the right ones that are worth s155 00:17:18.560 --> 00:17:21.240 the engineering time. s156 00:17:21.439 --> 00:17:29.640 So, instead of opening ADPRs, we built this human-friendly report that basically tries to convince you that it's worth your while. s157 00:17:29.640 --> 00:17:44.240 Something like, "Hey, this endpoint, it's usually taking around 200 ms, but every once in a in a while it takes like 45 seconds, and it happens because you're using distinct and not the search index of Mongo." s158 00:17:44.240 --> 00:17:49.040 And there's a very short explanation of what's happening and what's the fix. s159 00:17:49.040 --> 00:17:55.440 And then you can either create a ticket and fix it on your own, and create a PR, or just look at it. s160 00:17:55.440 --> 00:18:09.560 And we found that building these small gists that are humanly readable and it's easy to understand what's the thing, made a huge, huge difference. s161 00:18:09.560 --> 00:18:19.440 Because we're still living in this hybrid world where humans are in the loop, and we want to respect our place in people's lives and to make sure that we flag things s162 00:18:19.440 --> 00:18:29.080 that really matter, and that we have some confirmation, not only that the fix is good, but also that this issue is worth fixing. s163 00:18:29.080 --> 00:18:38.280 And then, when you look at that endpoint and you deploy that change, and all of a sudden it's flat again, it is pretty satisfying. s164 00:18:38.280 --> 00:18:46.360 And then next time you'll get that report, maybe you'll have a look, and it will be easier to convince you. s165 00:18:46.360 --> 00:18:50.320 So, four things that I learned that could be relevant for you. s166 00:18:50.320 --> 00:18:56.280 One is we need to define what matters, and the scoring and the guardrails are what makes this reliable. s167 00:18:56.280 --> 00:19:07.240 We can automate a bunch of things, and it becomes easier to just create some slop, but when we start with what are the things that are worth it, even though it's much cheaper s168 00:19:07.240 --> 00:19:22.480 to fix these things than it was a few years ago, it's still not free and therefore we need to understand that it's worth the impact the human reviewer and the risk that it entails if it does. s169 00:19:22.480 --> 00:19:34.520 Second part is that a lot of what we're talking about today is accelerating developers and what they do and I think what's interesting about this experience is that we automated something s170 00:19:34.520 --> 00:19:37.160 that it's not just doing it faster. s171 00:19:37.160 --> 00:19:43.280 We're automating a phase that just did not happen in the day-to-day lives of engineers. s172 00:19:43.280 --> 00:19:49.920 No one actually stopped every week and had a look on whether there are low-hanging fruits that could be relevant. s173 00:19:49.920 --> 00:20:00.360 But now we can use the agents to not only do the things that we're doing faster, but also to help us surface opportunities that we would probably never do s174 00:20:00.360 --> 00:20:02.040 without them. s175 00:20:02.040 --> 00:20:10.200 This is It sounds simple, but it's hard to actually apply it in the day-to-day, but context over cleverness works almost every time. s176 00:20:10.200 --> 00:20:27.040 If they have the right context, if they have the right skills, if they know exactly what these agents get much much more useful and every time a new model comes out, things get better and yet I do believe that at least for most of the cases, the models are good enough already s177 00:20:27.040 --> 00:20:29.640 to be able to to automate that. s178 00:20:29.640 --> 00:20:37.680 But it is up to us to help steer and guide to the right directions to make sure that we get the results that we want. s179 00:20:37.680 --> 00:20:50.600 They're not necessarily the the absolute right thing because it doesn't necessarily exist and you know, when we know our domain and our business, we understand which issues matter more. s180 00:20:50.600 --> 00:20:58.800 What's worth fixing, what latencies actually impact our customers' experience in the most significant way. s181 00:20:58.800 --> 00:21:02.600 And that helps us make sure that we focus on the right things. s182 00:21:02.600 --> 00:21:08.160 And in that aspect, agentic engineering is not like coding with an agent. s183 00:21:08.160 --> 00:21:19.920 If something works 80% of the time and you're using it with your cursor in your IDE, that's fine because you're there, you're in context, and you can help fix and steer. s184 00:21:19.920 --> 00:21:36.160 If we're talking about an automation that runs autonomously, we have to have a very high confidence that we're doing the right thing and that it's not going to just stray off and hand us a bunch of things that we would s185 00:21:36.160 --> 00:21:42.440 either waste time on reviewing or just not be able to trust over time. s186 00:21:42.440 --> 00:21:54.440 And I think like the hardest part of this automation was to get to a point where we feel confident enough that the issue is worth fixing and that the fix is verified in runtime s187 00:21:54.440 --> 00:22:03.800 so that it is handed to a person when we have a fairly good confidence that it actually works. s188 00:22:03.800 --> 00:22:08.440 And yet, they still review it, but we know it's worth their time. s189 00:22:08.440 --> 00:22:26.520 And getting to that agentic engineering automation level requires crossing towards the 80-90% trust, and it's something that is dramatically different than using an agent directly as an engineer. s190 00:22:26.840 --> 00:22:29.040 So, I hope that was helpful. s191 00:22:29.040 --> 00:22:42.880 And if there are any questions or anything like that, I'm available and always love to geek out on the AI and the LDC, and uh looking forward to hear about cool automations that you build on your own. s192 00:22:42.880 --> 00:22:45.160 Thank you.