WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=E3KbFLAGD6A NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/E3KbFLAGD6A.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.000 --> 00:00:01.309 All s2 00:00:01.309 --> 00:00:03.309 [music] s3 00:00:13.200 --> 00:00:13.519 right. s4 00:00:13.519 --> 00:00:15.679 Hello everyone. s5 00:00:15.679 --> 00:00:21.680 I'm going to try to speak as loud as I can so it's reasonably good with all the all the commotion. s6 00:00:21.680 --> 00:00:23.039 Uh my name is Greg. s7 00:00:23.039 --> 00:00:25.279 I'm an AI engineer at Sentry. s8 00:00:25.279 --> 00:00:37.280 I work on the infrastructure that powers seir centuries uh debugging agent and I have a little rage baity title of the presentation that calls to stop prompting but I do not mean loop engineering. s9 00:00:37.280 --> 00:00:40.000 I'm going to get uh a little bit deeper here. s10 00:00:40.000 --> 00:00:43.680 Uh who still reads code that AI generates? s11 00:00:43.680 --> 00:00:45.360 Anyone? s12 00:00:45.360 --> 00:00:48.239 It's not a trick question and I'm not trying to shame you. s13 00:00:48.239 --> 00:00:49.920 I I think it's contextual. s14 00:00:49.920 --> 00:00:59.600 I read code when it's sentry code and I need to make sure that we are not breaking the actual production for sentry but at the same time if I am s15 00:00:59.600 --> 00:01:12.960 working on a side project it doesn't really matter uh to read every line of code and I can I can be a little bit more lenient with it uh but then every now and then I go back to the actual code I want to make improvements and s16 00:01:12.960 --> 00:01:20.240 um it's bad it's it's overly defensive code it's tests are crap they are not testing anything. s17 00:01:20.240 --> 00:01:24.479 Um, I would not want to maintain or work in that codebase anymore. s18 00:01:24.479 --> 00:01:28.000 So then what you do is you you talk to the agent. s19 00:01:28.000 --> 00:01:30.560 You say, "Hey, you've done a mistake. s20 00:01:30.560 --> 00:01:31.759 This is overly complicated. s21 00:01:31.759 --> 00:01:33.439 The code's too defensive. s22 00:01:33.439 --> 00:01:38.880 Um, you don't have to create endpoints for everything you're doing, otherwise I have to maintain it forever." s23 00:01:38.880 --> 00:01:45.360 And I had to remove a lot of swearing from this slide because in my actual transcripts, there is going to be a lot of a lot of rage. s24 00:01:45.360 --> 00:01:54.560 Usually those are very elementary mistakes that the agents are making if you don't put any guard rails in place and um and that just outrages me. s25 00:01:54.560 --> 00:02:00.640 So then you do that, the agent says, "You're absolutely right. s26 00:02:00.640 --> 00:02:02.240 Let me fix this." s27 00:02:02.240 --> 00:02:06.159 And it's going to run and try to fix the the mistakes. s28 00:02:06.159 --> 00:02:10.640 And five minutes pass and compaction of context happens. s29 00:02:10.640 --> 00:02:14.319 And we're back to back to square one. s30 00:02:14.319 --> 00:02:21.360 And you would have to now repeat yourself, reprompt your fixes, reprompt the agent on the right track. s31 00:02:21.360 --> 00:02:28.560 And my entire thesis for this talk, my entire claim here is you should stop prompting. s32 00:02:28.560 --> 00:02:33.040 You should start codifying the actual rules that the agent should be following. s33 00:02:33.040 --> 00:02:45.840 um rules and policies that you want to establish inside your codebase that are going to capture both convention and the quality of code that you are expecting and would make you proud. s34 00:02:45.840 --> 00:02:53.280 And so when I say you would want to add deterministic checks or rules, what are you thinking of? s35 00:02:54.480 --> 00:02:56.480 Yes. s36 00:02:57.120 --> 00:02:58.160 What's that? s37 00:02:58.160 --> 00:02:58.800 Evals. s38 00:02:58.800 --> 00:02:59.200 Yes. s39 00:02:59.200 --> 00:02:59.840 Hooks. s40 00:02:59.840 --> 00:03:00.400 Hooks. s41 00:03:00.400 --> 00:03:01.280 Yes. s42 00:03:01.280 --> 00:03:07.120 Um the three that I'm thinking of before I even get to hooks and evils, hooks and evils are important. s43 00:03:07.120 --> 00:03:11.840 Tests are absolutely the class of that hooks are how you run those things at the right time. s44 00:03:11.840 --> 00:03:21.599 Um to me the three things that I want to optimize on the deterministic level are making sure that I have tests, making sure that there is strict typing, making sure that there are there are splinters. s45 00:03:21.599 --> 00:03:27.599 Those are not total statements and they are not enough to actually get your uh agent to write good code. s46 00:03:27.599 --> 00:03:34.480 as an example, if you just take from this that you have to put strict typing in place, you will still be in a bad place. s47 00:03:34.480 --> 00:03:55.200 Um, if you don't discourage or ban um a way of typing that removes edge cases you don't really want to see, then the behaviors that or the states that are representable in the app are going to lead the agent to introduce them. s48 00:03:55.200 --> 00:03:59.040 But the thing that I want to predominantly focus on in this talk is llinters. s49 00:03:59.040 --> 00:04:09.200 Lintters are our way to codify um the rules in a deterministic manner that we want to um remove from the from the codebase. s50 00:04:09.200 --> 00:04:16.799 Historically like before AI when you think about llinters um the way that this worked is the trade-off wasn't clear. s51 00:04:16.799 --> 00:04:19.759 Writing an actual llinter took a lot of work. s52 00:04:19.759 --> 00:04:29.360 um it was a maintenance cost that you had to sort of sign yourself up for and you had to consider how often the issue that you try to lint away s53 00:04:29.360 --> 00:04:39.360 comes up is it even worth it and if it's if it's pre pre- aents and those are humans that are working in my codebase and the only thing I'm doing is s54 00:04:39.360 --> 00:04:51.680 once every free week telling people to use the right component from the design system then I don't necessarily need a llinter for that right and people have memory so if I tell them once or twice they will usually remember that. s55 00:04:51.680 --> 00:05:04.320 But then this shifts entirely into this new world where the things that are writing code do not have any memory and are going to make the same repeated mistakes over and over again. s56 00:05:04.320 --> 00:05:10.160 And on the on the at the same time to write a new lter is very cheap and very easy. s57 00:05:10.160 --> 00:05:16.639 You can just have the agent do that and it's going to be much less frustrated with the actual lint rules that you're trying to to write. s58 00:05:16.639 --> 00:05:26.639 So my claim for this is simple and if you take one thing from this entire talk that I'm going to yap about a little bit more is you can after this talk take your laptop s59 00:05:26.639 --> 00:05:40.880 go to your agent and ask hey look at all my past transcripts look at my GitHub reviews the feedback from the bots the feedback from my peers and try to codify which ones of those are actual s60 00:05:40.880 --> 00:05:47.759 high quality lint rules that are deterministic and will be able to capture issues that we otherwise reprompt all the time. s61 00:05:47.759 --> 00:05:57.919 Um, that's something that I do now and I happen to do that even more often uh sort of on the schedule to try to see whether there are new things that I haven't captured. s62 00:05:57.919 --> 00:06:03.440 So, here are some examples of what I mean when I say custom lint rules. s63 00:06:03.440 --> 00:06:08.400 Those are uh lint rules that try to capture conventions that you want in your codebase. s64 00:06:08.400 --> 00:06:10.160 There is like a whole simple class of them. s65 00:06:10.160 --> 00:06:13.360 Those are the ones that you know and have written for a long time. s66 00:06:13.360 --> 00:06:25.919 And those are things like I want to lint out console log statements from a particular project or I want to make sure that if I have a design system or react practices that I want to follow or discourage s67 00:06:25.919 --> 00:06:29.360 those are codified and are limited away. s68 00:06:29.360 --> 00:06:35.280 Those are the simple things but I think with agents we are getting into more interesting complexity. s69 00:06:35.280 --> 00:06:37.600 We can push this a little bit further. s70 00:06:37.600 --> 00:06:44.160 So one example here, this is a little bit of a tricky example, but um I'll go through it. s71 00:06:44.160 --> 00:06:45.600 Sentry is a pretty old codebase. s72 00:06:45.600 --> 00:06:47.520 We are on Django, we are on DRF. s73 00:06:47.520 --> 00:06:49.680 We don't have strict typing everywhere. s74 00:06:49.680 --> 00:06:55.759 This is a very common case for any mature codebase where you're not going to be at the top leading edge of the technology. s75 00:06:55.759 --> 00:06:59.520 Um but we still want to improve the state of the world for for the agents. s76 00:06:59.520 --> 00:07:13.360 So concretely here um there is there was a lot of drift of what the open API schema for sentry said it's going to do and what it did and so we started introducing s77 00:07:13.360 --> 00:07:14.560 lint rules for that. s78 00:07:14.560 --> 00:07:26.479 First lint rule is if something is a end point endpoint a function for an endpoint it needs to return a response type and within that response type there cannot be any types so that we can infer the actual information of the type. s79 00:07:26.479 --> 00:07:34.240 Second, if it's a endpoint, you get that little extent schema decorator so we can ensure that it's documented. s80 00:07:34.240 --> 00:07:44.400 And then third, enforce that the actual type being produced here agrees with what the open API schema at the end of the day um declares. s81 00:07:44.400 --> 00:07:48.240 And that helps us because now APIs are a little bit more in sync. s82 00:07:48.240 --> 00:07:53.919 And that gets us to even more insane examples of um of llinters. s83 00:07:53.919 --> 00:08:01.919 So sir agent is a debugging agent and it has to talk to Sentry a lot. s84 00:08:01.919 --> 00:08:17.199 It uses Sentry telemetry to debug and RCA issues and provide root cause analysis solutions all of that because it has to talk to open to um Sentry API we oftent times have to guide it and the way we do it is through skills. s85 00:08:17.199 --> 00:08:26.080 oftentimes those are old endpoints that we are not able to um easily fix and they have some complexities that are hard to describe on the level of schema itself. s86 00:08:26.080 --> 00:08:31.039 So instead we guide that through skills which is a pretty common pattern in the agentic world. s87 00:08:31.039 --> 00:08:37.120 But here is an example of a skill that tells how a metric should be created at sentry. s88 00:08:37.120 --> 00:08:45.440 We generated a lot of those skills and examples with those code blocks as the as the examples of code that can then be run in the sandbox by agent. s89 00:08:45.440 --> 00:08:54.399 And in the original attempt to do that, we lost the accuracy of the agent significantly because lots of those examples were hallucinated. s90 00:08:54.399 --> 00:09:11.680 So the first lintter we created was a llinter that says if there is a code block inside the skill, you got to extract those code blocks, you have to make sure the type check, they can run in our sandbox and whatever they declare agrees with open API schema from Sentry s91 00:09:11.680 --> 00:09:13.200 and that helped. s92 00:09:13.200 --> 00:09:22.399 But what what the f the first thing that the agent did when we tried to rerun the skill generation with that rule is oh there are lint rule errors s93 00:09:22.399 --> 00:09:27.519 let me remove the code blocks and that wasn't very desired by us. s94 00:09:27.519 --> 00:09:30.240 So we said no no no examples have to be put in code blocks. s95 00:09:30.240 --> 00:09:36.720 So it put the examples back but it removed the bactics because only the bactics are linted. s96 00:09:36.720 --> 00:09:39.600 So let's let's fix the problem right. s97 00:09:39.600 --> 00:09:48.399 So we're like, "No, no, no. If it looks like a code, smells like a code, if it has underscores and parenthesis, push those into codelocks." s98 00:09:48.399 --> 00:09:52.800 So I was like, "Okay, so you want me to put code into codelocks. s99 00:09:52.800 --> 00:09:55.200 I'm going to describe the APIs." s100 00:09:55.200 --> 00:10:04.080 So now in pros, it started describing what the APIs should do, but it did one mistake, which is it referred to the URLs uh of the APIs. s101 00:10:04.080 --> 00:10:17.440 So then we linted away the URLs and we played this little whack-a-ole for a little while, but at the end of the day, we ended up with skills that are synced to our open API schema and that wasn't that expensive to do. s102 00:10:17.440 --> 00:10:28.000 And we have I'm not going to say unbreakable model, but we have a model where when API changes, we actually don't have to follow up on the agent skills all the time. s103 00:10:28.000 --> 00:10:32.399 it it's it's kept in sync through the through the llinters. s104 00:10:32.959 --> 00:10:44.320 And when it comes to the actual movements or the actual how do we do this you can use any sort of tool that you're using already es oxlin flake 8 clippy s105 00:10:44.320 --> 00:10:58.160 as grabb as grabb is great uh because it's language agnostic so it kind of you know unified tooling warm fuzzy feelings about that um there is a bunch of new tools that are popping up that are trying to approach it from a different uh perspective s106 00:10:58.160 --> 00:11:03.519 one example is taskless where is instead of writing the actual lter you write what you want the llinter to do. s107 00:11:03.519 --> 00:11:09.200 You write your intent in English language and then the actual llinter is um derived from it. s108 00:11:09.200 --> 00:11:17.760 Um and then the last thing that I uh that I will repeat is try asking the agent for the actual lers. s109 00:11:17.760 --> 00:11:21.200 You will be surprised uh what it can what it can generate. s110 00:11:21.200 --> 00:11:23.120 But that's only half the picture. s111 00:11:23.120 --> 00:11:31.680 What if there are rules that are deterministic cannot express um cannot be expressed through llinters? s112 00:11:31.760 --> 00:11:33.120 There is plenty of those, right? s113 00:11:33.120 --> 00:11:45.839 Those are all types of qualitative measures like you have a state machine with just three states and I created a small PR to add small functionality and ballooned up the number of states into 10. s114 00:11:45.839 --> 00:11:48.320 I don't know how to lint against that. s115 00:11:48.320 --> 00:11:50.560 I don't think I reliably can link against that. s116 00:11:50.560 --> 00:11:57.920 I can like create small tactical things like no more than four states, but it's all It's all made up. s117 00:11:57.920 --> 00:12:10.880 This is a this is the situation where we're talking about qualitative measures and um this is where you are engaging your engineering experience or if tweet as as Twitter would say your taste s118 00:12:10.880 --> 00:12:21.680 um and you just have to decline it straight up or you can have an LLM agent that is going to do a pass and for simpler thing help you with that s119 00:12:22.399 --> 00:12:37.920 concretely here I've uh I've started to create policies for the repository and those policies are such as I don't want too many tests, I want stronger tests or I do not want s120 00:12:37.920 --> 00:12:40.000 a code that's too defensive. s121 00:12:40.000 --> 00:12:49.600 I'm sure you've seen those where every single possible case is try code by the agent and it's it's just sloppy long long functions. s122 00:12:49.600 --> 00:12:57.519 I would rather have code that in its type system does not allow to express states that are undesired. s123 00:12:57.680 --> 00:13:02.880 Um so I created those policies and then I created a skill that I called grumpy engineer. s124 00:13:02.880 --> 00:13:13.760 We call it Garfield internally and the way it works is it takes all the policies and run sub agents for each of the policies and the sub agent can either give feedback s125 00:13:13.760 --> 00:13:21.200 or stand down and we loop that and this is the closest you will hear me doing loops by the way. s126 00:13:21.200 --> 00:13:30.160 uh we will loop that until the agent stands stands down and says I I'm okay with that I validate that this is fine and only then I read the code s127 00:13:30.160 --> 00:13:42.399 and sort of my metric of success here is how much I'm casting up the agent how much when I'm reviewing the code I'm like oh yeah this is not ideal but I can I can steer it here and I don't have to do s128 00:13:42.399 --> 00:13:56.000 very elementary feedback all the time it is still qualitative but the goal here isn't to remove the code review quite yet you So this things will change as models get better but more so to raise the floor and make it s129 00:13:56.000 --> 00:14:01.680 easier to um to work with the agents uh as you as you're going forward. s130 00:14:01.680 --> 00:14:10.000 There is also one thing that I want to mention that is um beware of a snake oil. s131 00:14:10.000 --> 00:14:20.959 We are talking about the tricky part being qualitative measures and I've done experiments where I tried to capture a qualitative measure in a quantitative metric. s132 00:14:20.959 --> 00:14:27.760 So two concrete examples one of them is um cyclomatic complexity and the other one is test coverage. s133 00:14:27.760 --> 00:14:39.839 Cyclomatic complexity uh is a measure that says how maintainable and testable the code is and it tries to base that on how many branches happen in the code and how deep the call stack goes s134 00:14:39.839 --> 00:14:47.760 and um and test coverage you know test coverage tries to tell you how much of the code is covered by test. s135 00:14:47.760 --> 00:14:56.880 Um, and the problem with that is when you give a number to the agent, the agent will optimize the out of that number. s136 00:14:56.880 --> 00:14:57.760 And it did. s137 00:14:57.760 --> 00:15:03.839 And I I got code bases, I got projects where I got to like a 100% or close to 100% of test coverage. s138 00:15:03.839 --> 00:15:05.600 But the actual tests are bad. s139 00:15:05.600 --> 00:15:09.760 They are like asserting particular string in particular output of the agent. s140 00:15:09.760 --> 00:15:12.079 They are not measuring anything of value. s141 00:15:12.079 --> 00:15:15.760 Um, and more so they are giving you a false sense of security. s142 00:15:15.760 --> 00:15:17.040 So be aware of that. s143 00:15:17.040 --> 00:15:24.240 beware of um the complexities uh of of actual qualitative metrics. s144 00:15:24.240 --> 00:15:25.920 Um and that's it. s145 00:15:25.920 --> 00:15:26.880 That's that's my call. s146 00:15:26.880 --> 00:15:33.040 My call to action for you is after this try to ask your agent for llinters and policies and see whether that helps you. s147 00:15:33.040 --> 00:15:35.680 Um try things and make the repo remember. s148 00:15:35.680 --> 00:15:37.279 And that's that's my time. s149 00:15:37.279 --> 00:15:39.519 Thank you.