WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ZD9-4fW2HhM NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ZD9-4fW2HhM.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.360 --> 00:00:04.560 Lately I've been building agents a lot for operational tasks. s2 00:00:04.560 --> 00:00:10.280 And while I was working one Friday night, I saw the sunset. s3 00:00:10.280 --> 00:00:13.000 And then dinner time came and went. s4 00:00:13.000 --> 00:00:14.800 And it hit me. s5 00:00:14.800 --> 00:00:17.640 I was in that familiar dev flow. s6 00:00:17.640 --> 00:00:21.160 And the thrill of building was back. s7 00:00:21.280 --> 00:00:26.160 Many of us who are coding with agents, we feel like this quiet sense of dread. s8 00:00:26.160 --> 00:00:34.440 Like they're kind of taking all of the fun parts of building and leaving us with the unglamorous work, but let me give you a little advice. s9 00:00:34.440 --> 00:00:35.960 Let them have it. s10 00:00:35.960 --> 00:00:42.480 Because if you go up just one layer, you'll find that the thrill is still there. s11 00:00:42.480 --> 00:00:50.520 When you're building agents, not just using them to write code, you start getting into architecting agentic systems. s12 00:00:50.520 --> 00:00:56.000 And you realize that the building blocks are different, but the discipline is the same. s13 00:00:56.000 --> 00:01:03.560 So, I find myself now flexing the same engineering muscles that I did pre-gen AI. s14 00:01:03.560 --> 00:01:06.000 And I'm having a blast with it. s15 00:01:06.000 --> 00:01:08.840 So, I'm going to walk through the flow of designing an agent. s16 00:01:08.840 --> 00:01:15.040 I'm going to show you where engineering skills still come into play. s17 00:01:15.040 --> 00:01:20.920 So, the agent is relocation scout, which is a house hunting agent. s18 00:01:20.920 --> 00:01:33.280 And if you did this as just a one-time prompt that like points the agent to some listings and asked it to rank them, I mean, that'll work, but you're likely not going to find a house in a day, right? s19 00:01:33.280 --> 00:01:39.280 So, you want to build this as an agentic system that you can reuse. s20 00:01:39.280 --> 00:01:42.320 One that can persist knowledge outside of the session. s21 00:01:42.320 --> 00:01:48.960 You know, it could reload or query that knowledge later to make decisions even within a fresh context. s22 00:01:48.960 --> 00:01:56.680 So, when thinking about how to design an agent, the first engineering skill that I exercised is systems thinking. s23 00:01:56.680 --> 00:01:59.520 So, an agent is not the system, right? s24 00:01:59.520 --> 00:02:01.280 It's part of the system. s25 00:02:01.280 --> 00:02:07.440 And that system has files and tools, humans, even other agents. s26 00:02:07.440 --> 00:02:15.480 So, Relocation Scout sits inside of something bigger and it pulls in listings and signals about the neighborhoods. s27 00:02:15.480 --> 00:02:22.200 It weighs them against what I care about, and then it hands me back a ranked short list. s28 00:02:22.200 --> 00:02:27.120 So, I often hear people say, "Just let your coding agent build it, right?" s29 00:02:27.120 --> 00:02:29.000 And I think that's a mistake. s30 00:02:29.000 --> 00:02:40.200 Like, yes, my coding agent can build it, but before allowing it to do so, I need to think about the whole environment, the entire system, right? s31 00:02:40.200 --> 00:02:43.400 I want to like think about what's this agent's job? s32 00:02:43.400 --> 00:02:45.200 What does it depend on? s33 00:02:45.200 --> 00:02:47.200 What happens if it breaks? s34 00:02:47.200 --> 00:02:55.320 And I want to treat it like any other component where it has boundaries and responsibilities, has dependencies, you know? s35 00:02:55.320 --> 00:02:57.400 And in ways that it can fail. s36 00:02:57.400 --> 00:03:02.080 And that whole thought process, that's engineering. s37 00:03:02.080 --> 00:03:05.480 The second skill is workflow design. s38 00:03:05.480 --> 00:03:08.840 So, traditional software is full of workflows. s39 00:03:08.840 --> 00:03:11.040 We got CICD pipelines, right? s40 00:03:11.040 --> 00:03:15.000 We got like ticket life cycles, you name it. s41 00:03:15.000 --> 00:03:19.200 Agentic systems, they need that same kind of design. s42 00:03:19.200 --> 00:03:26.080 As much as we all love the slash goal command, an agent needs more than a goal, it needs a path. s43 00:03:26.080 --> 00:03:33.532 When we say, "Review this listing," that's a goal, but the workflow is what defines what actually has to happen, right? s44 00:03:33.532 --> 00:03:41.560 [clears throat] For example, the agent has to gather what it needs, it needs to weigh the listing against my criteria, and then act, right? s45 00:03:41.560 --> 00:03:44.640 And every run ends one of three ways. s46 00:03:44.640 --> 00:03:48.360 Either it's going to stop, it's going to retry, or it's going to escalate. s47 00:03:48.360 --> 00:03:53.320 So, that path is what shapes the rest of the architecture. s48 00:03:53.320 --> 00:04:07.760 Once I see how work moves through the system, I can make better calls about what context the agent needs, what parts I want the agent to handle directly, and when like a tool or person should take over. s49 00:04:07.760 --> 00:04:11.959 We all know the danger of one giant thing that does everything, right? s50 00:04:11.959 --> 00:04:19.799 We scoff when we see one gigantic class or big old function that's doing too much, right? s51 00:04:19.799 --> 00:04:22.720 Or bloated service with a gazillion endpoints. s52 00:04:22.720 --> 00:04:25.120 We call these code smells. s53 00:04:25.120 --> 00:04:28.200 Well, agentic systems, they have their own version of this. s54 00:04:28.200 --> 00:04:30.280 It's the giant prompts. s55 00:04:30.280 --> 00:04:32.480 And this starts innocently enough. s56 00:04:32.480 --> 00:04:39.240 Like in a instructions file, maybe I tell the relocation scout how to size up a listing. s57 00:04:39.240 --> 00:04:40.720 Fair. s58 00:04:40.720 --> 00:04:43.200 But then I hit an edge case, so I go back. s59 00:04:43.200 --> 00:04:45.560 I add a note for that. s60 00:04:45.560 --> 00:04:49.320 And then I remember and a safety rule, right? s61 00:04:49.320 --> 00:04:52.000 So, of course, that has to go in there. s62 00:04:52.000 --> 00:04:56.280 I'm proud of myself that I even remember to put that in there, right? s63 00:04:56.280 --> 00:05:01.280 And then, oh yeah, there's like one more very important exception. s64 00:05:01.280 --> 00:05:05.760 And before you know it, that prompt is doing everything. s65 00:05:05.760 --> 00:05:11.200 And your engineering spidey sense already knows that this is messy. s66 00:05:11.200 --> 00:05:15.840 So, why aren't you taking a step back to decompose it, right? s67 00:05:15.840 --> 00:05:25.520 Decomposition means spotting the distinct jobs that are hiding inside of that one blob and pulling them apart into separate pieces. s68 00:05:25.520 --> 00:05:42.680 So, if I look at the prompts for relocation scout in its entirety, it includes a reusable process for pulling and normalizing a listing, and then it's going to have like a fixed format for how to write the short list. s69 00:05:42.680 --> 00:05:50.800 It has a little section in there for how to calculate the commute, and then a chunky subtask on how to research the neighborhood. s70 00:05:50.800 --> 00:05:56.560 That's four different jobs crammed into a single prompt. s71 00:05:56.560 --> 00:06:01.600 And then you wonder why your agent is drifting and not sticking to the script. s72 00:06:01.600 --> 00:06:04.680 The script is too long. s73 00:06:04.920 --> 00:06:14.280 So, I'm not saying that, you know, you need to split things up for the sake of it, but the point is to make each part easier to reason about, right? s74 00:06:14.280 --> 00:06:16.160 That way it's easier to task. s75 00:06:16.160 --> 00:06:19.680 It's easier to change things when you need to. s76 00:06:19.680 --> 00:06:23.240 Now, decomposition is about breaking the system apart. s77 00:06:23.240 --> 00:06:27.720 Separation of concerns is about putting each responsibility in the right place. s78 00:06:27.720 --> 00:06:38.320 And this is where building agents start to feel really familiar to me because in traditional software, we'd ask things like, should this live in the controller or the service layer? s79 00:06:38.320 --> 00:06:42.240 Or, you know, is this business logic or presentation? s80 00:06:42.240 --> 00:06:46.200 So, when building agents, you may have the same sort of questions. s81 00:06:46.200 --> 00:06:49.200 There's just different places to put things. s82 00:06:49.200 --> 00:06:54.480 So, the process to normalize the listing, should that stay buried in the prompt? s83 00:06:54.480 --> 00:06:57.440 Or maybe it actually become a skill, right? s84 00:06:57.440 --> 00:07:02.400 Um I want every listing in the short list formatted the same way. s85 00:07:02.400 --> 00:07:07.400 So, that structured output should probably be defined in a schema. s86 00:07:07.400 --> 00:07:10.680 Isn't that what you would do if you were coding the system yourself? s87 00:07:10.680 --> 00:07:12.120 I would. s88 00:07:12.120 --> 00:07:19.320 And then the piece that calculates the commute, that can go in a nice little boring script. s89 00:07:19.320 --> 00:07:26.120 And and then researching the neighborhood, that's meaty enough, should probably be handled by a sub agent. s90 00:07:26.120 --> 00:07:33.440 Now, you're using the best tools for the job, and it's clearer where to find things within the system. s91 00:07:33.440 --> 00:07:37.360 Modularity is important in agentic systems as well. s92 00:07:37.360 --> 00:07:47.280 Just like we have reusable functions and classes and libraries, Now, I'm also thinking about reusable agent capabilities. s93 00:07:47.280 --> 00:07:50.520 And the clearest example of this is an agent skill. s94 00:07:50.520 --> 00:07:58.200 So, making a skill to normalize listings comes in really handy when you need to expand the agents' duties. s95 00:07:58.200 --> 00:08:02.840 For example, what if I broaden my house search to three cities? s96 00:08:02.840 --> 00:08:05.920 Every one of those markets can load the same skill. s97 00:08:05.920 --> 00:08:09.160 So, I wrote it once and they all can reuse it. s98 00:08:09.160 --> 00:08:16.800 So, this has now basically become a component that I can reuse across agents or even share with other people. s99 00:08:16.800 --> 00:08:20.800 Kind of like the same way that we lean on packages. s100 00:08:20.800 --> 00:08:24.880 And then, sub-agents are another kind of reusable module. s101 00:08:24.880 --> 00:08:31.120 So, a lot of people that I talked to, they don't quite get the point of sub-agents. s102 00:08:31.120 --> 00:08:34.599 Architecturally, they're sort of like functions, right? s103 00:08:34.599 --> 00:08:37.520 So, you give them one specific task to do. s104 00:08:37.520 --> 00:08:40.080 You call them when it needs to be done. s105 00:08:40.080 --> 00:08:44.680 And they can do it really well because that's all that they have in scope, right? s106 00:08:44.680 --> 00:08:50.240 They are They're not carrying the context of the entire session with them. s107 00:08:50.240 --> 00:09:00.480 So, like our neighborhood research sub-agent, we can drop that into any market or workflow and it works, you know, for what it's supposed to do. s108 00:09:00.480 --> 00:09:02.680 It's good in any hood. s109 00:09:02.680 --> 00:09:08.600 Um but like everything, deciding like what should be a module, that takes some judgment, right? s110 00:09:08.600 --> 00:09:10.480 Not everything should be reused. s111 00:09:10.480 --> 00:09:15.000 Some instructions are local to a given workflow, right? s112 00:09:15.000 --> 00:09:19.040 Might not be worth abstracting because sometimes that costs more than it saves. s113 00:09:19.040 --> 00:09:22.640 But this is just another engineering decision here, right? s114 00:09:22.640 --> 00:09:26.680 Agentic systems, they have these same sorts of trade-offs. s115 00:09:26.680 --> 00:09:28.040 Algorithmic thinking. s116 00:09:28.040 --> 00:09:31.760 This is one of the most important skills in agentic system design. s117 00:09:31.760 --> 00:09:36.440 Just because an agent can do something doesn't mean that it should, right? s118 00:09:36.440 --> 00:09:40.920 Some tasks are better handled by plain code. s119 00:09:40.920 --> 00:09:58.760 For example, calculating that commute time or deduping listings that I've already seen, an agent's model is better at things like fuzzy, you know, fuzzy stuff, judgment, ambiguity, um reasoning over messy input. s120 00:09:58.760 --> 00:10:06.040 And ignoring this distinction is where I see a lot of agentic systems get more complicated than they used to be. s121 00:10:06.040 --> 00:10:16.760 So, you're using a model, you're handing it every part of the task to do, and then you're getting frustrated when the output differs every day. s122 00:10:16.760 --> 00:10:22.280 Um but some of this stuff can be handled by just regular code, right? s123 00:10:22.280 --> 00:10:24.680 It'll be cheaper, it'll be more reliable. s124 00:10:24.680 --> 00:10:28.280 I promise you AI did not invent automation, right? s125 00:10:28.280 --> 00:10:31.920 We can use code while still using these systems. s126 00:10:31.920 --> 00:10:38.280 So, my rule of thumb here is if a task has an exact answer, reach for code. s127 00:10:38.280 --> 00:10:44.200 If it needs interpretation or judgment, that's when you can get the agent to do it, right? s128 00:10:44.200 --> 00:10:51.040 So, use code for determinism, use agents for judgment, and then use humans for authority. s129 00:10:51.040 --> 00:11:01.400 So, the agent decides which listings are worth a closer look, the code crunches the commute, filters out the ones I've already seen, and then I'm the one who approves s130 00:11:01.400 --> 00:11:04.400 actually booking a tour of the house. s131 00:11:04.400 --> 00:11:08.920 Free form text is fine when the human is the only one reading it. s132 00:11:08.920 --> 00:11:16.800 But when another system has to act on the agent's output, then you're better off with a contract, usually. s133 00:11:16.800 --> 00:11:19.920 So, we already do this everywhere in software. s134 00:11:19.920 --> 00:11:25.640 Anytime two systems talk, there's an agreed-upon shape between them, yes? s135 00:11:25.640 --> 00:11:28.600 So, agentic systems, they need that same discipline. s136 00:11:28.600 --> 00:11:35.720 For example, when relocation scout scores a house, it shouldn't just handy back a message and call it a day, right? s137 00:11:35.720 --> 00:11:42.120 That's lovely for me to read in that moment, but that is a dead end for the system. s138 00:11:42.120 --> 00:11:49.200 If the decision is like buried in like one of our sessions, nothing downstream can reliably find that. s139 00:11:49.200 --> 00:11:55.320 So, instead, it gets written into a structured shape to the agent's memory. s140 00:11:55.320 --> 00:12:03.120 And I use uh Compendium Wiki for this for for my agent memory layer on most of my agents. s141 00:12:03.120 --> 00:12:10.720 Um but in here, there's a decision, a score, a reason, and because it's structured, that memory becomes queryable. s142 00:12:10.720 --> 00:12:19.960 So, later I can ask Relocation Scout like, "Hey, show me every house rated four or better that has a commute of 15 minutes of or less, right?" s143 00:12:19.960 --> 00:12:25.600 And it can actually pull that because the score and the commute, they live in known places. s144 00:12:25.600 --> 00:12:28.240 They're not trapped in a session convo. s145 00:12:28.240 --> 00:12:31.680 And it's not just me that needs to like get this information. s146 00:12:31.680 --> 00:12:38.320 My shortlist step within the system, it reads these same fields um without a human in the loop. s147 00:12:38.320 --> 00:12:42.080 So, the agent's output is another step's input. s148 00:12:42.080 --> 00:12:46.080 And so, the contract is what makes that handoff safe. s149 00:12:46.080 --> 00:12:53.640 And you know, the best part is that defining the shape forces you to get really clear and specific. s150 00:12:53.640 --> 00:13:02.400 Because if you can't say what the output should look like, then you probably don't yet fully understand what you're asking the agent to produce. s151 00:13:02.400 --> 00:13:06.400 So, a prompt can run once and be done, right? s152 00:13:06.400 --> 00:13:19.960 But a useful agentic system has to be able to work in messy realities where webhooks get fired twice, or a run doesn't complete for whatever reason, and you need to retry the flow. s153 00:13:19.960 --> 00:13:23.480 So, the agent has to keep track of its state. s154 00:13:23.480 --> 00:13:25.400 Was this action already taken? s155 00:13:25.400 --> 00:13:27.600 If so, did the input change, right? s156 00:13:27.600 --> 00:13:34.160 If not, did the session crash or something like what parts of this can I safely retry? s157 00:13:34.160 --> 00:13:36.040 And this is not an exception, right? s158 00:13:36.040 --> 00:13:38.000 This happens all the time. s159 00:13:38.000 --> 00:13:47.440 So, you have to design for idempotency, which is where you can run the same thing twice and the second run doesn't cause a mess. s160 00:13:47.440 --> 00:13:50.880 Um and we do this in traditional software often. s161 00:13:50.880 --> 00:13:59.800 Um but with the agents, they add a little trap here because you can't trust the model because its outputs can vary, right? s162 00:13:59.800 --> 00:14:09.480 So, a retry risks the agent actually like rewording the request just enough that it might look like a brand new task. s163 00:14:09.480 --> 00:14:12.520 So, you have to enforce this in the system. s164 00:14:12.520 --> 00:14:14.400 Let's look at an example with our agent. s165 00:14:14.400 --> 00:14:22.880 So, let's say a new listing comes in um and relocation scout wants to email my realtor to ask about a viewing. s166 00:14:22.880 --> 00:14:31.040 So, after that action, the agent must log it to memory um that it it sent that email, right? s167 00:14:31.040 --> 00:14:37.360 And then the agent goes to my calendar and wants to just block that time out just in case. s168 00:14:37.360 --> 00:14:41.320 But it crashes before it could take that action. s169 00:14:41.320 --> 00:14:44.839 So, that run is only half done. s170 00:14:44.839 --> 00:14:47.240 Later, a lint pass runs. s171 00:14:47.240 --> 00:14:59.240 By the way, you got to have a lint pass with these things to kind of keep them healthy, but um during the lint pass, it notices that went out, but the calendar was never blocked. s172 00:14:59.240 --> 00:15:02.200 So, it retries the task. s173 00:15:02.200 --> 00:15:05.600 But it better not email my realtor again, right? s174 00:15:05.600 --> 00:15:06.839 That already happened. s175 00:15:06.839 --> 00:15:11.839 I don't want to be the annoying client um you know, I'm sending all these emails. s176 00:15:11.839 --> 00:15:18.560 So, it just needs to finish the part that it that didn't happen, which [clears throat] was blocking my calendar. s177 00:15:18.560 --> 00:15:24.200 But it only knows this because it's checking what the system wrote down, right? s178 00:15:24.200 --> 00:15:30.920 So, if we run this again, then the agent just completes what's missing instead of making a mess. s179 00:15:30.920 --> 00:15:35.600 Threat modeling is a really important skill when designing agentic systems. s180 00:15:35.600 --> 00:15:39.000 Everyone is on edge about this stuff these days, right? s181 00:15:39.000 --> 00:15:42.400 But, security engineering already taught us the basics. s182 00:15:42.400 --> 00:15:51.760 We need to validate your inputs, you know, give the least privilege needed, um and draw boundaries around what an action can touch. s183 00:15:51.760 --> 00:15:54.480 So, agentic systems, they need all of that. s184 00:15:54.480 --> 00:16:00.840 Our relocation scout will consume a lot of content from strangers, right? s185 00:16:00.840 --> 00:16:12.360 Um the agent needs to read the listing copy from the seller, uh forum threads, and neighborhood reviews from anonymous people on the internet. s186 00:16:12.360 --> 00:16:23.040 So, we need to treat all of that as untrusted input and make it very clear to the agent that this is evidence, not instructions. s187 00:16:23.040 --> 00:16:33.680 And then, after considering the input, you also want to think about what boundaries should you put in place around what the agent is able to do. s188 00:16:33.680 --> 00:16:39.480 For example, our agent can read listings and it can build shortlists all day long. s189 00:16:39.480 --> 00:16:41.320 Knock yourself out, right? s190 00:16:41.320 --> 00:16:51.839 But, I don't want it autonomously emailing sellers or booking tours or, heaven forbid, submitting offers on my behalf, right? s191 00:16:51.839 --> 00:16:57.640 So, those actions need to be walled behind like my approval, right? s192 00:16:57.640 --> 00:17:07.199 And when you draw that wall, what you've done is reduced the blast radius, and hopefully, that minimizes your exposure to risk. s193 00:17:07.199 --> 00:17:13.760 Now, every engineer can relate to what it feels like to inherit a system that you can barely understand. s194 00:17:13.760 --> 00:17:25.880 This is one of the key reasons why I don't just have my coding agent design my other agents because I know it'll be thrown together in a way that's technically works, but it's not maintainable, s195 00:17:25.880 --> 00:17:26.360 right? s196 00:17:26.360 --> 00:17:33.880 There's going to be a giant prompt likely, and even if the agent does decompose, I don't know. s197 00:17:33.880 --> 00:17:38.440 I'm just not convinced that it's going to separate the concerns properly. s198 00:17:38.440 --> 00:17:39.840 You know, I got trust issues. s199 00:17:39.840 --> 00:17:41.040 What can I say? s200 00:17:41.040 --> 00:17:48.120 Um so in my agentic systems, I make sure to bake maintainability into the system itself. s201 00:17:48.120 --> 00:18:03.680 Every level of the system has an agent's in D file that explains the workflow and where the policy lives, um supporting resources like skills and scripts and sub agents and most importantly how to keep its memory s202 00:18:03.680 --> 00:18:04.760 up-to-date. s203 00:18:04.760 --> 00:18:13.840 So anyone, human or agent, can come in a system and get oriented without needing to reverse engineer a bunch of prompts. s204 00:18:13.840 --> 00:18:16.360 In fact, that's the test, really. s205 00:18:16.360 --> 00:18:26.320 I design my agent so that even in a fresh context, they can jump right into the system and start cold, knowing exactly what to do. s206 00:18:26.320 --> 00:18:28.080 And this also really helps. s207 00:18:28.080 --> 00:18:31.040 So whenever I need to modify the system, right? s208 00:18:31.040 --> 00:18:37.800 So I can pretty much grab any harness and say, "Update this agent to do XYZ." s209 00:18:37.800 --> 00:18:45.680 And because the system is so well designed, the chances of it being successful at that update are much higher. s210 00:18:45.680 --> 00:18:55.720 If it does happen to run into any issues when trying to update, that's a signal to me that I need to improve the maintainability of the system. s211 00:18:55.720 --> 00:18:58.480 So designing agents is software engineering. s212 00:18:58.480 --> 00:19:02.320 The primitives are different, but the discipline is the same. s213 00:19:02.320 --> 00:19:04.280 We still need to understand the system. s214 00:19:04.280 --> 00:19:08.200 We need to define the workflow and know what flows into it. s215 00:19:08.200 --> 00:19:20.760 We still need to break the problem down and put responsibilities in the right place, make the right things reusable, determine which actors are best suited for which jobs, define contracts, s216 00:19:20.760 --> 00:19:26.360 manage state, design for safety, and make the system understandable. s217 00:19:26.360 --> 00:19:32.160 This is why building agents can give you that same thrill of building software. s218 00:19:32.160 --> 00:19:33.840 We're still building. s219 00:19:33.840 --> 00:19:36.520 We just moved up a layer. s220 00:19:36.520 --> 00:19:38.920 Thanks so much.