WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=CgsWxRUY5Eo NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/CgsWxRUY5Eo.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.880 --> 00:00:02.000 Hi there. s2 00:00:02.000 --> 00:00:05.240 Welcome to AI Engineer World's Fair 2026 event. s3 00:00:05.240 --> 00:00:06.560 I am Rajat Shah. s4 00:00:06.560 --> 00:00:15.440 I'm a staff software engineer at Netflix where I work in the AI platform organization building large-scale distributed systems for machine learning model hosting. s5 00:00:15.440 --> 00:00:25.520 In this talk, I'm here to um share how we did uh improve our performance engineering uh throughput by introducing AI agents into the mix. s6 00:00:25.520 --> 00:00:37.640 And this is more of a uh playbook or a practitioner's guide uh to help you also replicate similar learnings in your own organizations uh to improve uh the infrastructure s7 00:00:37.640 --> 00:00:41.640 cost and uh ship faster. s8 00:00:42.680 --> 00:00:44.040 Let's first talk about the problem. s9 00:00:44.040 --> 00:00:50.880 Why does performance engineering doesn't scale and uh what does it cost to actually do it right? s10 00:00:51.080 --> 00:00:56.800 The problem is arising arising from the fact is that you are authoring code now at a 10x faster speed. s11 00:00:56.800 --> 00:01:02.240 The uh coding agents are getting better and better uh at uh solving problems. s12 00:01:02.240 --> 00:01:11.280 And um as more and more uh engineers uh adopt it, it it gets very easy to produce code in your system. s13 00:01:11.280 --> 00:01:23.880 And this is slight exaggeration, but the compute cost also uh is increasing at a similar pace uh because uh it doesn't always uh write the fastest code. s14 00:01:23.880 --> 00:01:29.280 Uh so, this is where the problem arises because of that new wide coding era. s15 00:01:29.280 --> 00:01:31.360 Uh the AI agent ships code. s16 00:01:31.360 --> 00:01:35.000 Uh it is pretty much tuned to ship code fast. s17 00:01:35.000 --> 00:01:42.600 Um and of course, you could say that uh as the uh the coding agents are evolving, newer models are coming into play. s18 00:01:42.600 --> 00:01:47.120 Uh they get better and better at uh simply writing performant code. s19 00:01:47.120 --> 00:01:48.760 But, that's not always true. s20 00:01:48.760 --> 00:01:57.120 Your um agent doesn't know specific details about your platforms and your frameworks uh and your internal code base patterns. s21 00:01:57.120 --> 00:02:10.479 So, it it tends to just produce code based on what it might have already seen other code bases using or inventing new patterns in your code bases that you did not anticipate s22 00:02:10.479 --> 00:02:16.120 an engineer to use as a pattern to use your framework. s23 00:02:16.760 --> 00:02:21.680 So, let's look at what a performance engineer typically does. s24 00:02:21.680 --> 00:02:30.080 I'm calling this as a human performance engineer, which is responsible for identifying bottlenecks in a service and fixing them. s25 00:02:30.720 --> 00:02:38.680 Typically, a a human would trigger profiling on a single production instance of a fleet of production instances. s26 00:02:38.680 --> 00:02:43.959 You would go and download it, potentially open it in a visualizer. s27 00:02:43.959 --> 00:02:47.640 The raw data that you download typically is not great to look at. s28 00:02:47.640 --> 00:02:55.040 It It could be, for example, a JSON structure data of the call stack and where the CPU is spent. s29 00:02:55.040 --> 00:03:07.160 So, using a visualizer helps you at least see and visualize the call stack and CPU time of various um method in your services better. s30 00:03:07.160 --> 00:03:18.320 Once you have that visualizer open, you pretty much end up spending a lot of time in uh like just looking at and and finding in this in this treasure s31 00:03:18.320 --> 00:03:27.440 hunt on on the path on the potential places where you could improve the code to to make it more performant. s32 00:03:27.440 --> 00:03:31.560 This takes a lot of time in order to even learn how to look at it. s33 00:03:31.560 --> 00:03:35.959 Um And there's a learning curve to curve curve to it. s34 00:03:35.959 --> 00:03:38.840 And this is where the real bottleneck ends up being. s35 00:03:38.840 --> 00:03:45.959 You end up you end up having to spend straight many, many minutes to uh to identify the bottlenecks. s36 00:03:45.959 --> 00:03:56.920 Once you've identified some code paths and some packages that that are spending significant CPU cycles, you you would end up searching it in your code bases in your code repos s37 00:03:56.920 --> 00:04:01.360 and see if it has a potential to improvement. s38 00:04:01.360 --> 00:04:10.400 Hopefully, you you hit you have luckier and you find a root cause and you produce a code review out. s39 00:04:10.400 --> 00:04:15.040 You merge it and you get some performance wins. s40 00:04:15.040 --> 00:04:17.600 And then you repeat all of this again. s41 00:04:17.600 --> 00:04:18.600 You see the problem, right? s42 00:04:18.600 --> 00:04:31.680 This is a very manual effort and very tedious effort to get right and this ends up being a bottleneck if you were to do it across many of your code bases and code paths and that's why this is done very rarely. s43 00:04:31.680 --> 00:04:43.520 People typically end up looking at profiling data only when something is going wrong at 2:00 a.m. and somebody needs to fix a problem because your CPU is unbearable. s44 00:04:44.160 --> 00:04:45.760 So, we asked this question internally. s45 00:04:45.760 --> 00:04:49.400 Can we Can an LLM read this profiling data? s46 00:04:49.400 --> 00:05:01.120 The 20 minutes that I mentioned an engineer spends in identifying hot paths, can an LLM agent which is fed that data also do it much faster? s47 00:05:01.120 --> 00:05:05.880 And we tried to answer this question through some live services. s48 00:05:05.880 --> 00:05:11.320 So, the next couple of slides will be about this experiment and how we do it, right? s49 00:05:11.360 --> 00:05:20.040 Before we get to it, the the foundation of of our assumption is that every profiler essentially speaks the same language. s50 00:05:20.040 --> 00:05:31.560 Even though you would profile and you could have your services written in Java, Python, Go, etc. The profiler's example that I mentioned here do the same same thing. s51 00:05:31.560 --> 00:05:46.080 They would run it in your they would run against your production service on a single instance and they would essentially try to capture the call stack, the self CPU and inclusive CPU that is spent on s52 00:05:46.080 --> 00:05:52.120 on each of the methods in your call stack, Uh sample data very high frequency. s53 00:05:52.120 --> 00:06:06.200 So, irrespective of which language, which runtime you're using in production, the output of a profiling data is actually very similar and very well-structured for an LLM agent to use. s54 00:06:06.200 --> 00:06:10.080 This is one assumption that we had going into the experiment. s55 00:06:10.080 --> 00:06:23.600 The second assumption we had going into the experiment is that there are many common patterns that the coding agents have learned through their training sites across all the publicly available code sources that they might have had. s56 00:06:23.600 --> 00:06:33.360 So, things like O of N squared loops, loop invariants, things that you're computing every time in your loop which could be computed just once outside it. s57 00:06:33.360 --> 00:06:41.240 If you have certain places where you are doing the object allocation very repeatedly and if you could move it out as well. s58 00:06:41.240 --> 00:06:50.919 Um Or there are times when you would be like having some contention and places where you could optimize through better batching. s59 00:06:50.919 --> 00:06:54.280 All of these are very common patterns that an agent already knows. s60 00:06:54.280 --> 00:07:00.640 Uh it uh I mean, assuming it is trained on very good quality software code. s61 00:07:00.640 --> 00:07:06.840 The better the quality, the better agent knows which patterns are not great. s62 00:07:06.840 --> 00:07:23.280 Um and this is a very good point for an LLM agent because it essentially means that it can technically look at the the code base and it can identify those patterns very easily. s63 00:07:23.280 --> 00:07:32.600 So, this is more about like pattern recognition, like finding out these code patterns to find some code smells that that are worth fixing. s64 00:07:32.600 --> 00:07:41.000 So, when you have that profiling output that I mentioned and you have coding agent that could understand it, you feed it into it. s65 00:07:41.000 --> 00:08:00.040 Um The profiling data that it needs to read is actually very well structured in a way that it could see that the in this case uh the um the profiling data is uh in in the case of row one and row three, they are the same call path and the immutable s66 00:08:00.040 --> 00:08:09.040 map copy of that I have in my row three is actually uh used inside uh a tensor merge method that we have. s67 00:08:09.040 --> 00:08:18.440 Uh so um don't think too much about the method names here, but the key point that I wanted to call out is that it was able to use these s68 00:08:18.440 --> 00:08:20.160 functions. s69 00:08:20.160 --> 00:08:29.320 Uh it knows the meaning of these functions and when it sees it is being used in a a poor way, it can actually identify that this is a quadratic s70 00:08:29.320 --> 00:08:33.039 uh um algorithm and not a linear algorithm. s71 00:08:33.039 --> 00:08:47.400 So, this is where uh our first aha moment was that okay, if you feed a profiling data, it if it knows the pattern, it assuming the coding agent was trained on a very good quality data, it can actually identify and up and find out that s72 00:08:47.400 --> 00:08:52.520 uh find in your code base where that bad uh patterns are actually uh running. s73 00:08:52.520 --> 00:08:54.480 And this is not by looking at the code base. s74 00:08:54.480 --> 00:09:01.560 This is purely by looking at the call stack that the profiling uh uh data produced. s75 00:09:02.160 --> 00:09:10.880 And once uh a prof a an AI agent gets this, what what are the steps that it needs to do in order to get to the potential fix? s76 00:09:10.880 --> 00:09:19.560 So, we we we have mentioned that it can identify uh a pattern and say that okay, this is a potential problem to be fixed. s77 00:09:19.560 --> 00:09:21.320 What what happens next? s78 00:09:21.320 --> 00:09:34.960 Um it could knowing where that method lives, it could do a code search and first of all, before uh I mean, yeah, it could do a code search and find out the code repo where this method is defined s79 00:09:34.960 --> 00:09:37.520 uh or this code is defined. s80 00:09:37.520 --> 00:09:42.880 And it it needs to extract out the exact commit that is currently running in production, which is typically easy to get. s81 00:09:42.880 --> 00:09:50.360 You know which build is running in your production, and you could point that to the LLM to uh check out the Git repo at that same commit. s82 00:09:50.360 --> 00:10:07.080 And then uh once it has uh find found out uh and has the Git repo cloned, uh it can look for uh that exact code path, uh skip any internal library details, and find out uh the code patterns. s83 00:10:07.080 --> 00:10:22.360 And now that it now once it has the uh full uh uh code repo cloned out, the um the methods uh definition and implementation identified, it can trace the entire call call path of that method. s84 00:10:22.360 --> 00:10:36.280 And this uh structured data once it uh has this information, becomes uh like very powerful, and we'll we are going to talk about how uh it leverages this. s85 00:10:36.320 --> 00:10:40.640 So, these were our understanding going into the experiment. s86 00:10:40.640 --> 00:10:45.640 We uh we knew it could find uh the patterns. s87 00:10:45.640 --> 00:10:54.560 We tested it with with a couple of uh of our services to see if it can actually not just find the pattern, but also take it one step further and s88 00:10:54.560 --> 00:11:01.280 uh produce a real fix and provide a real fix that uh an engineer could approve and merge. s89 00:11:01.280 --> 00:11:14.160 So, first finding uh the the O of N squared um uh example that I just mentioned, it uh once it knew uh that this is a problem, and looking at the profile s90 00:11:14.160 --> 00:11:22.160 profiling data, it knows that this is consuming 8.8% of the CPU time during that uh uh period of profiling. s91 00:11:22.160 --> 00:11:37.240 So, for us, uh the coding agent not just ended up finding the problem, given enough instructions as a form of let's say skill or prompt, uh you could actually have it uh do those four steps that I just mentioned, which is checking out the Git repo, s92 00:11:37.240 --> 00:11:41.960 uh finding where that code is implemented, and actually uh sending a code review out. s93 00:11:41.960 --> 00:11:48.400 All of it could be done uh in a very large code base with powerful enough code agents in less than 5 minutes. s94 00:11:48.400 --> 00:12:00.760 Um and in this case, uh we also, which I'll talk about a little bit in the later slides, uh were able to identify what is the savings uh if you were to uh reimplement it with a s95 00:12:00.760 --> 00:12:05.960 uh with an optimized implementation in terms of both CPU as well as latency savings. s96 00:12:05.960 --> 00:12:14.680 Uh I I've shown the numbers here of our real production uh savings that we were able to observe uh through this uh exercise. s97 00:12:14.680 --> 00:12:26.280 So, this was like a first uh proof for us that you could actually indeed introduce an an AI agent into the mix and get uh your productivity wins of not just identifying, s98 00:12:26.280 --> 00:12:29.880 but also going all the way to a code review. s99 00:12:29.880 --> 00:12:41.760 Uh the second uh great benefit is that once you have this well set up, um the uh other example that we are able to take it forward to it is, s100 00:12:41.760 --> 00:12:57.920 let's say it identifies the uh pattern in one of the services that we profile, um and the learning that we had is that it could actually uh look at all your potential places where that same bad pattern s101 00:12:57.920 --> 00:12:59.640 uh is being reused. s102 00:12:59.640 --> 00:13:03.800 So, in the previous example, it it just identified one bad implementation. s103 00:13:03.800 --> 00:13:05.560 You fix it, you get all the benefits. s104 00:13:05.560 --> 00:13:19.280 In this case, it identified not just a bad code, but because that pattern, in this case, which is a counter object uh for our uh spectator metrics, uh that are uh being s105 00:13:19.280 --> 00:13:32.160 created on every single uh uh iteration of a hot path in our service stack, um it could essentially see that this is uh a real problem and it could s106 00:13:32.160 --> 00:13:42.600 double down and search multiple services where that same pattern is being repeated and uh scale up the effort to fix it in multiple services. s107 00:13:42.600 --> 00:13:51.080 In this case, we found that same bad pattern were actually implemented in seven different services uh through cross repo code searches. s108 00:13:51.080 --> 00:14:00.840 And if fixed, it could actually uh if fixed across all of those different code repos, it could have savings between 0.5 to 4.6% of CPU cycles. s109 00:14:00.840 --> 00:14:15.040 Um so we have discussed the problem, our hypothesis, and the experiment which proved out that a uh LLM coding agent could actually help improve uh the uh the finding and fixing s110 00:14:15.040 --> 00:14:18.160 of your um suboptimal code. s111 00:14:18.160 --> 00:14:22.040 Let's see how we can actually build this into the development cycle. s112 00:14:22.040 --> 00:14:32.160 Uh it doesn't have to be a very retrospective uh problem where once a production service is uh having issues, you end up doing all of those exercises. s113 00:14:32.160 --> 00:14:44.800 I'll talk about how you can actually leverage in your end-to-end software development cycle so that you get the benefits uh much early and potentially avoid having that suboptimal code s114 00:14:44.800 --> 00:14:47.560 reach production altogether. s115 00:14:47.960 --> 00:15:01.800 Uh the first uh real problem is that LLMs if you invoke them, they do have some memory these days, but it's very compact memory and it will uh it it won't have all the s116 00:15:01.800 --> 00:15:08.800 uh information that uh you are uh uh that you potentially as a performance engineer know when you're trying to debug. s117 00:15:08.800 --> 00:15:23.680 So, you you when you you if you think of a coding agent and if you want to build it at par uh uh as sufficient as as efficient as a human engineer would typically be, you want to introduce a long-term memory that it could reuse. s118 00:15:23.680 --> 00:15:29.280 So that's our first real problem to solve for and I'll talk about how you solve that. s119 00:15:29.280 --> 00:15:35.520 The simplest way is to first think of you want to build a catalog. s120 00:15:35.520 --> 00:15:37.800 Patterns, anti-patterns. s121 00:15:37.800 --> 00:15:47.920 Once you have identified enough of them, you could put it in a catalog that that users uh that is used by an LLM agent which can be stateless of its own. s122 00:15:47.920 --> 00:15:59.360 And together a stateful catalog and a state stateless LLM can become a full fleet-wide memory um for for a coding agent to use. s123 00:15:59.360 --> 00:16:07.880 The foundation for this and and to foundation for incorporating in this development cycle is practically just this. s124 00:16:07.880 --> 00:16:16.560 You want to keep this very central so that it's not very team-specific, not very product-specific, but rather a central ever-growing catalog. s125 00:16:16.560 --> 00:16:23.560 As the catalog patterns get written, if there are certain things that could be generalized. s126 00:16:23.560 --> 00:16:30.960 In that example, the first example that I the first finding that I provided around O of N squared implementation is actually very generalizable. s127 00:16:30.960 --> 00:16:37.680 All coding languages will potentially have the same problem if it if the code is written and implemented in that way. s128 00:16:37.680 --> 00:16:47.960 So you you can take it a step further in that your catalog can become even more ubiquitous and usable across languages and frameworks as well. s129 00:16:47.960 --> 00:16:52.560 And you think of it as a blueprint. s130 00:16:52.560 --> 00:17:06.079 As as those as that catalog starts growing, it becomes your building block for future coding agents to actually use for identifying the problem in your development cycle. s131 00:17:06.079 --> 00:17:08.360 And that's the next thing I'm going to talk about. s132 00:17:08.360 --> 00:17:22.400 The solution that I mentioned so far of a memory is not very fancy vector search or or a vector database that needs to store all the catalog of patterns and anti-patterns. s133 00:17:22.400 --> 00:17:30.320 Rather, you can start with just a markdown files in a in a centralized Git repo. s134 00:17:30.320 --> 00:17:40.240 Um, as the production findings appear, have the coding agent put more and more patterns and anti-patterns into this. s135 00:17:40.240 --> 00:17:52.240 And the benefit is that even if one service finds this, even if one uh, profiling, even if one service did the profiling to find this pattern, multiple services that in the future s136 00:17:52.240 --> 00:18:03.920 are going to run profiling, the agent that is doing the profiling of that other service can use the pattern catalog from the first service to find the patterns more optimally and not having to s137 00:18:03.920 --> 00:18:10.560 redo all the exercise that the first agent did in order to identify that anti-pattern. s138 00:18:12.040 --> 00:18:15.560 Uh, you might think of like how how do I get started with this? s139 00:18:15.560 --> 00:18:19.120 Well, you you don't always have to start fresh, but you could start fresh. s140 00:18:19.120 --> 00:18:32.280 There is no harm in having a catalog that is just empty in the beginning and as more and more profiling happens, uh, more uh, powerful the coding agents that are referring it could become. s141 00:18:32.280 --> 00:18:40.480 But there are few, many in fact, sources of how you could public sources of how you could improve your coding agents. s142 00:18:40.480 --> 00:18:53.560 If you're into C++ optimizations, Jeff Dean had this wonderful blog post around how to look for certain optimization opportunities in your C++ code. s143 00:18:53.560 --> 00:19:04.160 PyTorch has a torch fix code repo where it catalogs several anti-patterns that could help optimize kernels and the model graph of your PyTorch model. s144 00:19:04.160 --> 00:19:08.080 And then you might have your own performance playbooks as well. s145 00:19:08.080 --> 00:19:15.640 Typically, as your organization grows and your enterprise software evolves, you end up noting down certain patterns and anti-patterns. s146 00:19:15.640 --> 00:19:31.760 Those all could become the bootstrapping point for that uh catalog Git repo that I mentioned that can act as a centralized um place for all uh catalog all patterns and anti-patterns to be noted. s147 00:19:32.040 --> 00:19:34.640 And what would an entry in that catalog look like? s148 00:19:34.640 --> 00:19:36.920 Uh the example is right here. s149 00:19:36.920 --> 00:19:47.680 Uh as the left-hand side it mentions the uh small hints that an LLM uh coding agent in the future could use to easily query that uh catalog. s150 00:19:47.680 --> 00:19:51.840 It can mention a list of symbols, uh the services where it was confirmed. s151 00:19:51.840 --> 00:20:03.520 Some confidence level is great here as well because as more and more services confirm this, you want to keep updating this so that uh the future agents get more and more confident that, "Okay, this is worth a human s152 00:20:03.520 --> 00:20:08.960 uh uh human review, and I would want to more confidently send out a code review to a human." s153 00:20:08.960 --> 00:20:16.080 Uh I'm still keeping the confidence uh bar to just send a code review and not directly uh push it to production. s154 00:20:16.080 --> 00:20:17.240 That's by intent. s155 00:20:17.240 --> 00:20:28.160 I uh I'll talk a little bit on why, but I do feel like there is still need for a human approval because you're modifying an existing code that is running just fine in in production s156 00:20:28.160 --> 00:20:30.400 in order to optimize it, which is which is very risky. s157 00:20:30.400 --> 00:20:34.800 If you don't know the business context, if you don't have enough test coverage, uh you might end up breaking. s158 00:20:34.800 --> 00:20:41.320 So, there is a need for human uh to actually be the one responsible for approving that code review. s159 00:20:41.320 --> 00:20:46.920 Um and on the right here, I mentioned what the entry in that pattern-anti-pattern catalog could look like. s160 00:20:46.920 --> 00:20:52.840 Uh a anti-pattern and a good pattern so that both the things are well noted here. s161 00:20:52.960 --> 00:21:00.880 Um so, I mentioned a code review and a human uh involvement in order to approve and merge that code review is still needed. s162 00:21:00.880 --> 00:21:04.000 So, how do you optimize for noise there? s163 00:21:04.000 --> 00:21:14.600 There could be enough signals that the uh AI agent uh could could think of uh improving and it could like arbitrarily send code reviews for them without knowing enough about s164 00:21:14.600 --> 00:21:16.120 your system. s165 00:21:16.120 --> 00:21:28.760 There are a few tips that I mentioned here could be very powerful in in making sure that the agent first has enough verifications done before looking for an human's attention. s166 00:21:29.600 --> 00:21:44.920 The integration test, unit test, the most basic functional test that your system needs to have should be run ahead of time by the coding agent itself so that it knows that as it is making code changes to optimize it, it is also not s167 00:21:44.920 --> 00:21:48.360 causing any production business logic to fail. s168 00:21:48.360 --> 00:21:59.360 So, as as you have good coverage and unit test, it gets more confidence that it gets more confidence that your that the change it is going to propose is actually right. s169 00:21:59.360 --> 00:22:06.560 Once it has made that functional correctness check, you still the our code coverage isn't typically ideal. s170 00:22:06.560 --> 00:22:12.440 There are still surprises in production and canary deployments could be very powerful in this case. s171 00:22:12.440 --> 00:22:25.600 If you provide enough automation around this, the canary deployment could be a prerequisite before the code review requires human attention. s172 00:22:25.600 --> 00:22:26.800 What is a canary? s173 00:22:26.800 --> 00:22:38.200 You typically have you typically have two machines, one containing your old code, another containing your new code, which could be the performance fixed code. s174 00:22:38.200 --> 00:22:51.400 And you send the same traffic to both of them over a period of let's say 10 minutes and try to compare the CPU usage across between them. s175 00:22:51.400 --> 00:23:01.960 The comparison report typically is is what you would want the AI agent to make decision on and judgment on whether it should open a code review for it. s176 00:23:01.960 --> 00:23:12.400 In this case, the observability report could contain standard infra things like how how much CPU reduction, how much latency reduced, is there an increase in error rate? s177 00:23:12.400 --> 00:23:23.000 If if that's the case, it should see that as a red signal to not proceed because it might have gotten the business logic incorrect in an attempt to optimize the code. s178 00:23:23.320 --> 00:23:26.480 And what I mentioned here is that this is not an AI problem. s179 00:23:26.480 --> 00:23:32.280 The observability, canary, verify logic, these are all standard checks that you need need to have in your system. s180 00:23:32.280 --> 00:23:42.640 And then final guardrail that you want to have is an engineer decision, like I mentioned, still very important, critical uh in the mix. s181 00:23:42.880 --> 00:23:44.920 Um Here's a mental model. s182 00:23:44.920 --> 00:23:47.960 Profiler gives the estimate, canary gives ground truth. s183 00:23:47.960 --> 00:23:58.240 Uh canary is the way you verify and evaluate that your uh suggested fixes actually uh going to improve and have some uh positive impact and no negative impact. s184 00:23:58.240 --> 00:24:01.400 An engineer makes the eventual decision. s185 00:24:01.560 --> 00:24:18.760 So, the uh path so far has been about how do you uh improve the efficiency of getting from a profiled uh profiling data to producing a code review fix. s186 00:24:18.760 --> 00:24:27.880 Uh that's the reactive path that I just covered so far, uh which means you have the code already running in production, and now you're trying to improve it. s187 00:24:27.880 --> 00:24:32.360 That's a very that's typically too late in the game. s188 00:24:32.360 --> 00:24:39.600 Now, anytime you try to change a thing that is running in production in order to improve the performance of it, you have to be very, very careful. s189 00:24:39.600 --> 00:24:43.080 So, you want that to shift left. s190 00:24:43.080 --> 00:24:48.280 You want that reactive path to be your initial guide to build that initial catalog. s191 00:24:48.280 --> 00:25:04.360 And as your pattern catalog grows, you want to move closer and closer to the proactive path where uh as a human or a coding agent authors new code uh the the reviewer agent could actually look up that catalog s192 00:25:04.360 --> 00:25:20.160 and provide an inline uh comment uh by understanding the code change and ensure and providing a suggestion that hey, based on the pattern catalog and the observed uh profiling data that I have, I feel confident that this uh is an anti-pattern to introduce. s193 00:25:20.160 --> 00:25:22.520 Can you uh rewrite this code? s194 00:25:22.520 --> 00:25:29.040 That could be a a review comment that a reviewer agent could be uh providing. s195 00:25:29.360 --> 00:25:34.320 And if you shift left all the way, like why even wait until a code review? s196 00:25:34.320 --> 00:25:37.600 Uh that that's where you get the most power. s197 00:25:37.600 --> 00:25:50.040 Because you know, the anti-patterns are already in a catalog and if you are using a coding agent to write your code, you could actually ask the coding agent and hook that catalog s198 00:25:50.040 --> 00:26:02.400 in it so that when it is uh producing newer tokens and producing newer code, it could actually, before writing out the inefficient code, directly reference the catalog up front and s199 00:26:02.400 --> 00:26:08.360 uh and write your uh code in an optimal way to start with. s200 00:26:08.360 --> 00:26:17.480 Uh this could sometimes slow down the uh speed of uh newer code uh being written and might end up consuming more tokens. s201 00:26:17.480 --> 00:26:32.400 And that's why the pattern catalog needs to be very well uh structured and indexed so that the the way parses and reads the catalog doesn't fill up the agent context too much, but rather it can navigate it in a very hierarchical format s202 00:26:32.400 --> 00:26:41.000 and only uh and be able to find out the exact uh places it needs to look for a uh a given pattern or an anti-pattern. s203 00:26:41.000 --> 00:26:56.840 This means that if you can catch this in the code authoring phase itself, you skip all of the unnecessary uh overhead of uh somebody reviewing your code and providing a uh suggestion or even having that code change reach production. s204 00:26:56.840 --> 00:27:09.440 So, the sooner you catch it or even sooner you uh uh introduce that pattern catalog into your software development end-to-end life cycle, the easier it gets for you to keep the s205 00:27:09.440 --> 00:27:13.000 uh code and service running optimally. s206 00:27:13.400 --> 00:27:23.080 And uh I do want to highlight this that uh it's easier to imagine putting AI agent into every single step of your software development life cycle for performance improvements, s207 00:27:23.080 --> 00:27:30.400 but that's not the intent of this uh discussion or the uh the thing uh or or the playbook that I'm mentioning. s208 00:27:30.400 --> 00:27:32.920 You still want these foundations very, very right. s209 00:27:32.920 --> 00:27:35.480 Your test coverage needs to be rock solid. s210 00:27:35.480 --> 00:27:39.560 Uh business logic all needs to be very well encoded in your test. s211 00:27:39.560 --> 00:27:42.160 Your canary automation needs to be amazing. s212 00:27:42.160 --> 00:27:44.000 Uh and this is where you don't need AI. s213 00:27:44.000 --> 00:27:54.320 You pretty much just want AI agent to know how to invoke canary, where to get the downloaded report from canary canary, so that it can directly read the numbers from the report and s214 00:27:54.320 --> 00:27:56.040 uh make a judgment call. s215 00:27:56.040 --> 00:28:02.600 And that pattern catalog that I mentioned is also a foundational piece which needs to be gotten uh which which you need to get right. s216 00:28:02.600 --> 00:28:20.360 It It is by design a Git repo, so that it's both a human and a coding agent readable and uh authorable, and you could actually have humans also authoring patterns and anti-pattern as they're let's say building new frameworks and new platform components that uh they have enough context on at the time of building. s217 00:28:21.400 --> 00:28:26.600 The takeaways uh that I do want to mention, first build your code foundations. s218 00:28:26.600 --> 00:28:44.520 Uh think of uh all of the uh automation that you could do, uh but before introducing an LLM, try to see if there are good uh uh integration points that an agent could uh use such as how can it automatically trigger profiling on an instance on a s219 00:28:44.520 --> 00:28:45.960 uh CPU new that you have? s220 00:28:45.960 --> 00:28:54.520 Can it download the profiling data effectively, uh then feed it into uh an LLM uh uh model. s221 00:28:54.520 --> 00:29:05.560 And can it, once it has identified the fixes by cloning the Git repo, etc., can it actually validate that through a real production canary through shadow traffic or real traffic? s222 00:29:05.560 --> 00:29:16.280 All of those integration and foundation pieces needs to be really good in order for this entire life to life and this entire uh life cycle improvement that I'm mentioning. s223 00:29:16.280 --> 00:29:19.400 Otherwise, it will just cause more friction and more bugs in the production. s224 00:29:19.400 --> 00:29:25.160 So, you have to make these uh foundation pieces investment a lot uh more important. s225 00:29:25.160 --> 00:29:25.920 And start from here. s226 00:29:25.920 --> 00:29:29.760 Don't think of uh the reactive path as a bad approach. s227 00:29:29.760 --> 00:29:37.035 Uh reactive path is where you want to start with and then gradually move towards uh towards the left and uh into the proactive path. s228 00:29:37.035 --> 00:29:38.120 [clears throat] s229 00:29:38.120 --> 00:29:51.160 And the other takeaway, like I mentioned, uh is important is that as you build this uh memory of or essentially a catalog, the um the sooner you catch it, the s230 00:29:51.160 --> 00:29:59.160 better it gets for your uh end-to-end software development life cycle to have optimus optimal code running in production. s231 00:29:59.160 --> 00:30:10.360 Um This is a I'll uh this is a great mental model for how you step-by-step proceed in automating um or improving your uh efficiency of software development life cycle. s232 00:30:10.360 --> 00:30:15.440 The current state, uh which is typically the norm, uh you spend hours finding the problems. s233 00:30:15.440 --> 00:30:18.040 It's all very manual, no LLM in the mix. s234 00:30:18.040 --> 00:30:27.760 Start with introducing an LLM into the mix just for the um the identification of um uh of a potential fix. s235 00:30:27.760 --> 00:30:36.560 It might be okay if you are still having to trigger a profiler uh and produce output and feed it into an AI agent and then having to manually run canary, s236 00:30:36.560 --> 00:30:46.320 but it still gives you a lot of uh leeway to spend time on the other parts of the workflow as opposed to just staring at those flame graphs to get the um s237 00:30:46.320 --> 00:30:48.760 hot parts identified. s238 00:30:48.760 --> 00:31:00.520 And then this talk mainly covered the uh level two orchestration, which is um our level two automation in an agentic spectrum, which is more about the tools, the integrations, s239 00:31:00.520 --> 00:31:17.400 the the key harnesses, and the hooks that our AI agent needs to have in order to uh have full power in order to have like good uh uh uh capability to identify a problem and give you in various development life cycle areas of uh opportunities s240 00:31:17.400 --> 00:31:20.400 to uh improve your code base. s241 00:31:20.400 --> 00:31:27.840 And uh once you have this in place, once you have this standard step-by-step workflow, which could be very static, right? s242 00:31:27.840 --> 00:31:32.960 In this case, the workflow that I mentioned is is very uh well-defined, predefined. s243 00:31:32.960 --> 00:31:39.320 There is no uh uh LLM to reason and plan and uh do the further level of automations. s244 00:31:39.320 --> 00:31:50.520 All it has to do is uh you provide it a uh fixed workflow, which is download the profile uh from certain place, um or rather trigger the profile, download the profile, s245 00:31:50.520 --> 00:31:55.960 um and then uh analyze it, uh run the canary, and provide a suggested fix. s246 00:31:55.960 --> 00:32:07.440 Uh you could once you have that in place, you could essentially have a scheduled run on your services every week, so that it can keep continuously identifying newer problems s247 00:32:07.440 --> 00:32:20.840 uh that might have not been uh uh uh or that might have been introduced in the last one week, for example, and keep uh the the identification of the problem closer to when you wrote that code. s248 00:32:20.840 --> 00:32:35.680 And if you were to think of uh more automation uh in this place, you could uh um imagine a a a level where the workflow is not very fixed, and rather the AI agent has capabilities to even s249 00:32:35.680 --> 00:32:41.800 uh plan, reason, act, uh but you have to invest a lot more in evaluation sandboxing. s250 00:32:41.800 --> 00:32:58.120 You can't simply run AI agents without the right level of security guardrails so that prompt injection and other security attacks that a typical agent infrastructure currently cannot always solve for. s251 00:32:58.120 --> 00:33:03.160 Those things need to be heavily invested in if you were to go with level three autonomy. s252 00:33:03.160 --> 00:33:05.320 So, start with level one. s253 00:33:05.320 --> 00:33:08.440 Try to move to level two and you get maximum benefits. s254 00:33:08.440 --> 00:33:14.600 If you need more automation, you could think of the next step as needed. s255 00:33:16.560 --> 00:33:18.000 Thank you so much. s256 00:33:18.000 --> 00:33:23.520 I hope you got some learnings out of this small talk that I had. s257 00:33:23.520 --> 00:33:32.800 If if you want to connect more and connect further, I might end up putting a more detailed blog post on this on my shaharrajat.com personal website. s258 00:33:32.800 --> 00:33:36.640 I'll be happy to connect with you and provide more information. s259 00:33:36.640 --> 00:33:37.120 Take care. s260 00:33:37.120 --> 00:33:39.360 Thank you.