From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization - May Walter, Hud

AI Engineer · 22 min · 192 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01Hi everyone.
00:02I don't know if you're familiar with what I'm about to show.
00:05But remember when someone from the product is like saying that some page is too slow and maybe we can optimize it.
00:12And then someone from engineering would say, "Yeah, probably what we'd have to dig in to find out."
00:17So, we just kind of leave it as is.
00:21And then a few weeks later, Jenny would say, "No, no, no, but it's actually way too slow now.
00:26We have to prioritize it.
00:28How long is it going to take?"
00:30And we would say something like, "Well, somewhere between an hour and a week.
00:34We'd have to look at it to find out."
00:38And then when she asked who can even take it, the answer is only Dave.
00:42He's the only one who kind of knows that code and everyone else who wrote it left a decade ago.
00:50And amazingly, this still happens in teams all the time because we would know how much time it takes to do a specific optimization, but we never know how long it's going to take to investigate it and to find it.
01:04And miraculously, every time we look, we we actually find things that can be done around performance and stability, but we never stop to proactively look for them because it doesn't make sense.
01:16So, hi.
01:17I'm Mike, co-founder and CTO at Thundra.
01:19We're building a runtime intelligence layer for coding agents that captures functional level context and deep forensic context on things that matter so that coding agents can help you fix what's going on with production.
01:31And as a part of that, we built an agentic workflow that helps with continuously optimizing performance of production applications.
01:39And I'm going to share a bit more about what we built, what were the challenges along the way, and hopefully you can take something out of it and apply it in your day-to-day.
01:50So, we're surrounded by a lot of agentic PRs that all look kind of good to us.
01:57And one agent wrote them and the other one says, "Yeah, I I went over it and it looks fine."
02:04And we still feel that urge to to verify before we just merge it to production.
02:12And then when it doesn't work, we end up asking ourselves, "How the hell did we get to this place?"
02:18And if you feel like that, I want you to know that you're not alone.
02:21So, Google just published their Dora metrics for 2026 and we can see that the biggest impact of AI adoption on engineering is individual effectiveness or that feeling of, "Oh my god, I'm so fast.
02:34I can do everything in the world."
02:36And for me personally, lack of sleep is another symptom of that.
02:41But the second one is software delivery instability.
02:45And the throughput is actually not impacted as much as we expected.
02:49So, we basically feel more effective, we're more effective individually, but as a team, our throughput is kind of the same and our software breaks more often, which is not exactly
02:59what we were hoping for with this AI revolution just yet.
03:04But maybe there's an agent for that and if we can build faster with AI and we can fix faster with AI, then we can get those gains that we were talking about.
03:14So, I'm going to start with why we even wanted to do this.
03:17Then we're going to go over the tech and the process, which I think is also really important.
03:22And then share some gotchas and takeaways along the way.
03:26So, that leaks kind of faster than we can bail.
03:30We have these issues that we ignore because they're not important enough then they degrade to the point where they are important enough and then we reach this crisis mode where we are all hands-on-deck.
03:41We fix it in emergency mode and then we go straight back to ignoring, which is a leaky bucket by definition.
03:49And that mostly happens because the research phase is a black box.
03:55It could take an hour or weeks and we have to pay that debt and to make sure that we spend time and engine engineering time on it in order to even know
04:05what can be done about it.
04:07And that's hard.
04:07It's legitimately hard to prioritize something where you're not sure exactly what you're going to get out of it.
04:13But what if we can automate that investigation?
04:16So, we can basically run on a weekly basis with real production context, analyze the sweet spot in the and flag the high ROI opportunities in a way that's scored.
04:27So, it runs automatically without us having to stop and do something about it.
04:31It has the production context in mind and it can give a high ROI scored performance opportunities of the things that are easy and impactful.
04:41Kind of like that performance sprint that you run every few months, just automate it.
04:46Now, let's talk about how we can actually do it cuz the dream is very nice, but but the devil's in the details.
04:54So, first of all, we wanted an infrastructure for the agentic workflows that would be vendor neutral in terms of compute and where it runs, in terms of the harness and in also in terms of the model.
05:05Things are changing all the time.
05:06We wouldn't want to constrain ourselves to a specific vendor, specific model or anything like that.
05:12And especially like with hot, we want our customers to [clears throat] be able to use whatever agents they wish.
05:19We also wanted it to be secure in terms of the tool calls, permissions, and authentication.
05:24We want some trigger system, whether those are scheduled runs like the weekly run or a set of webhooks.
05:29For example, if we see some SLO breach, we would want to investigate it.
05:34And we really wanted it to be easy to maintain and update over time.
05:38I think one of the biggest learnings we've had with agentic workflows is that even if they work out of the box or we get to a point where we're happy about them.
05:47In time, we evolve and our expectations go up.
05:50So, just being able to maintain and update those logics and build that feedback loop was very important for us so that it's reliable and the people actually trust the outcome.
06:01Specifically, we chose to work with GitHub agent workflows for that and we can choose whatever agent we want to work with and build the workflows on top of that, but there are many other great tools that could be used for that.
06:15So, this is basically how it looks like.
06:18You can see that there's a description of the job and the goal and the analysis and then we basically go over the GitHub repository and generate that weekly deep insight report analyzing production data
06:31and finding those low-hanging high-ROI opportunities.
06:36So, for this specific setup, GitHub actions runs weekly and it uses Cloud Code, that was our specific choice, and then captures the runtime intelligence overhead via MCP so that we can basically look at the different endpoints, connect to the function level of overhead is there, and send a report to Slack.
06:57Of course, that could have been Teams or an email or anything like that.
07:01It could have been Cursor or Copilot.
07:03It's just the setup that we started with um to make sure that we have something that runs without us in the loop and sends that report to somewhere we actually live in, which is Slack.
07:16So, we want to take the production context, the traces, the queries, the latencies, and analyze them with the agent, score and flag which opportunities matter because if you can optimize something, but it runs every 3 weeks or you can
07:31like reduce 20 milliseconds, then it doesn't matter.
07:36And then the most important part here is the different the evidence.
07:40So, the agent actually fixes it reruns the tests and sees the impact that it had on that specific flow that was optimized.
07:50So that the human gets something after we already detected it.
07:54We understood the root cause.
07:56We understood why it matters to the business and we verified that the fix actually impacted that time.
08:04And then a human reviews that.
08:06And the loop is actually closed.
08:08So it's not hey, I have this idea of something that you can do.
08:10It's here's something that works.
08:13And we believe would make an impact on these specific business flows in production that are running 7,000 times a week.
08:21And then the human gets in the loop as a review gate.
08:25And of course it didn't work out of the box if you were wondering.
08:29So the first hurdle we had along the way is what we call the plausible unverified.
08:35So the agent would suggest a fix.
08:37It sounds right.
08:39It would look fairly real and then after we verified it it it just didn't work.
08:46And it's it's right that the agent suggested something that could theoretically cause that flow down.
08:52But what we wanted is to ground it on what's actually happening in production.
08:57Second part was complex queries.
08:59Um we specifically use ClickHouse.
09:02It's just an amazing columnar database but it's also slightly different than the classic SQL patterns.
09:09Now I'll talk about that in a bit.
09:11And the third one is the lazy fix.
09:13I'm sure you guys also experienced that when something throws an exception and then the agent says, "Well, maybe we can just catch that exception and then everything will be fine."
09:26But what we really want is to understand why that exception was even thrown in the first place or why are the results lagging.
09:36So those were things that we found that methodology could be very, very impactful with.
09:43It's not just the understanding of the data and how to connect it.
09:48It's also being quite thorough on what we want to do in that process and sort of building the playbook of how a senior engineer would do that.
09:59And of course, we need the context to be right.
10:03So, the problem with context there are also There are only two problems with context.
10:08You either have too much of it or you have too little of it.
10:12And what we found around runtime context, especially from production, is that more often than not, you actually have these two problems together because on one hand, you have a lot of low signal data
10:24that is hard for the agent to reason over.
10:26And on the other, you might not have all the logs and traces and metrics that you need in order to investigate that issue, which would still leave some room for assumptions and theories
10:38of that are not necessarily what our users are experiencing in production.
10:44And also, when we talk about metrics like service level, CPU and memory, or endpoints in the P90s, they are not connected to the function level.
10:54So, our coding agents reason over code and they look at these metrics and there are some relations between them, but they don't exactly speak the same language.
11:04So, when we ask questions about taking time and what can I do about it, we're often finding that there are some gaps there and again, accuracy could drop from that.
11:15So, what we did is what we call prod-to-code, which is to be able to explain what's going on in production in the same level that agents reason over because the agent's context
11:28lies on a function and file level, not on an service and endpoint level.
11:34So, basically, our context is running on a function level, and it is also connected to the endpoint or event consumer or cron job that ended up starting this task.
11:44So, you can ask a question like, "Hey, this endpoint that sometimes takes 7 seconds, where is the time spent, and what can be done about it?"
11:52Whether that's a function or an outbound call to a database, an LLM, or another microservice.
11:59And with that, you basically have the complete function level context for every single function, sort of like this service map, but on a function level of the different invocations,
12:10where they come from, how often they run, and the difference the context only when it's needed, only when we see requests that are taking longer than the P99 or some threshold that we can define.
12:22Then, we will capture those forensic evidence so that we can say, "Hey, here is an example of a request that took longer.
12:29Let's find out why."
12:31And we also have the ability to see that on top of the code, which is the HUD, the heads-up display, but but I guess it's kind of a way to explain
12:42how that data set is actually structured in a way that is much more comprehensive for a coding engineer to read and over.
12:51And then, we talked about the complex queries and what to do with them.
12:56So, the basic layer gave us the HUD query language, which are basically ClickHouse queries over that structure of functions and endpoints and forensics.
13:07On top of that, we also added a set of skills.
13:10We found that sometimes just querying the data is enough, but being able to get to the right query and to ask it again and again really created a lot of
13:23variance in our eval, and the skills actually help work with that data so that agents can use it.
13:30So for example, if we're talking about a 500, an HTTP 500, we would want to understand where that error came from.
13:38If we're talking about a memory spike, we want to understand what was running on those specific pods at that specific time where memory was higher and compare it to a baseline so that we see the diff.
13:50And all of these things were extremely helpful to be able to be a bit more methodological around how that works and to create more consistent results.
14:00And on top of that, there are a set of automations.
14:03So if we have the data, the query language, a set of skills, we can build automations on top of them like auto fixing issues as they arise or detecting
14:15dead code that isn't even running for the last 60 days and eliminating it or this automated performance improve- improvements automation that are we are talking about right now.
14:31So for performance, one example of that is to be looking for artificial delays like timeouts and sleeps and plus one queries, missing indexes, sequential assings and so on so forth.
14:43These are specific things that are much easier to find when you actually look for them.
14:49And in a code base that's 20 years old and has hundreds of contributors, it makes sense that you'll find quite a lot of those.
14:59And removing them is fairly easy and impactful.
15:05And then when you're asking something like, "Why are my endpoints taking long?"
15:09you can actually find the specific reasons and not just guess a bunch of static code analysis which could get you some result.
15:21I'm not saying it's never going to work.
15:23But, when you're talking about an automation, we have to think about how to build something that is robust enough for us to trust over time.
15:31And then we said, "Okay, now that our evas are looking good, we run weekly, we find real slow endpoints from production that are invoked.
15:40We score them, we test them, we verify them.
15:43Maybe we can just open pull requests and everyone will fix everything and the world will be amazing."
15:50Well, that's not exactly how that worked because people are still people and no one wants to wake up for, you know, a rain of 80 pull requests, as small as small as they can be.
16:03That's just not how people operate.
16:06And no one has time for that.
16:08We're too busy building other things and it's fine.
16:11So, what we actually do is we use that priority to make sure that we only flag the ones that matter and we actually started with one at a time
16:20to create that appetite and that habit.
16:23So, we look at whether this is a hot path in terms of how often it runs and how critical is it for the business.
16:29The business impact as in, you know, if this is something that has to do with payments or sign-up flows, obviously, we're more sensitive to that.
16:38Kind of like asking ourselves, would we be able to convince the product manager to prioritize it?
16:43And we also looked at the risk.
16:46If it's a risky change that requires a migration or anything like that, then obviously, it would require more time from the human that's reviewing it.
16:56And in that case, we're not necessarily looking for the highest impact ones, but for the highest ROI ones.
17:04And because we automate the investigation, we can look at the impact and the risk together and only surface and and require attention on the ones that we believe are the right ones that are worth
17:18the engineering time.
17:21So, instead of opening ADPRs, we built this human-friendly report that basically tries to convince you that it's worth your while.
17:29Something like, "Hey, this endpoint, it's usually taking around 200 ms, but every once in a in a while it takes like 45 seconds, and it happens because you're using distinct and not the search index of Mongo."
17:44And there's a very short explanation of what's happening and what's the fix.
17:49And then you can either create a ticket and fix it on your own, and create a PR, or just look at it.
17:55And we found that building these small gists that are humanly readable and it's easy to understand what's the thing, made a huge, huge difference.
18:09Because we're still living in this hybrid world where humans are in the loop, and we want to respect our place in people's lives and to make sure that we flag things
18:19that really matter, and that we have some confirmation, not only that the fix is good, but also that this issue is worth fixing.
18:29And then, when you look at that endpoint and you deploy that change, and all of a sudden it's flat again, it is pretty satisfying.
18:38And then next time you'll get that report, maybe you'll have a look, and it will be easier to convince you.
18:46So, four things that I learned that could be relevant for you.
18:50One is we need to define what matters, and the scoring and the guardrails are what makes this reliable.
18:56We can automate a bunch of things, and it becomes easier to just create some slop, but when we start with what are the things that are worth it, even though it's much cheaper
19:07to fix these things than it was a few years ago, it's still not free and therefore we need to understand that it's worth the impact the human reviewer and the risk that it entails if it does.
19:22Second part is that a lot of what we're talking about today is accelerating developers and what they do and I think what's interesting about this experience is that we automated something
19:34that it's not just doing it faster.
19:37We're automating a phase that just did not happen in the day-to-day lives of engineers.
19:43No one actually stopped every week and had a look on whether there are low-hanging fruits that could be relevant.
19:49But now we can use the agents to not only do the things that we're doing faster, but also to help us surface opportunities that we would probably never do
20:00without them.
20:02This is It sounds simple, but it's hard to actually apply it in the day-to-day, but context over cleverness works almost every time.
20:10If they have the right context, if they have the right skills, if they know exactly what these agents get much much more useful and every time a new model comes out, things get better and yet I do believe that at least for most of the cases, the models are good enough already
20:27to be able to to automate that.
20:29But it is up to us to help steer and guide to the right directions to make sure that we get the results that we want.
20:37They're not necessarily the the absolute right thing because it doesn't necessarily exist and you know, when we know our domain and our business, we understand which issues matter more.
20:50What's worth fixing, what latencies actually impact our customers' experience in the most significant way.
20:58And that helps us make sure that we focus on the right things.
21:02And in that aspect, agentic engineering is not like coding with an agent.
21:08If something works 80% of the time and you're using it with your cursor in your IDE, that's fine because you're there, you're in context, and you can help fix and steer.
21:19If we're talking about an automation that runs autonomously, we have to have a very high confidence that we're doing the right thing and that it's not going to just stray off and hand us a bunch of things that we would
21:36either waste time on reviewing or just not be able to trust over time.
21:42And I think like the hardest part of this automation was to get to a point where we feel confident enough that the issue is worth fixing and that the fix is verified in runtime
21:54so that it is handed to a person when we have a fairly good confidence that it actually works.
22:03And yet, they still review it, but we know it's worth their time.
22:08And getting to that agentic engineering automation level requires crossing towards the 80-90% trust, and it's something that is dramatically different than using an agent directly as an engineer.
22:26So, I hope that was helpful.
22:29And if there are any questions or anything like that, I'm available and always love to geek out on the AI and the LDC, and uh looking forward to hear about cool automations that you build on your own.
22:42Thank you.