Build Systems, Not Code - Angie Jones, Agentic AI Foundation

AI Engineer · 19 min · 220 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Lately I've been building agents a lot for operational tasks.
00:04And while I was working one Friday night, I saw the sunset.
00:10And then dinner time came and went.
00:13And it hit me.
00:14I was in that familiar dev flow.
00:17And the thrill of building was back.
00:21Many of us who are coding with agents, we feel like this quiet sense of dread.
00:26Like they're kind of taking all of the fun parts of building and leaving us with the unglamorous work, but let me give you a little advice.
00:34Let them have it.
00:35Because if you go up just one layer, you'll find that the thrill is still there.
00:42When you're building agents, not just using them to write code, you start getting into architecting agentic systems.
00:50And you realize that the building blocks are different, but the discipline is the same.
00:56So, I find myself now flexing the same engineering muscles that I did pre-gen AI.
01:03And I'm having a blast with it.
01:06So, I'm going to walk through the flow of designing an agent.
01:08I'm going to show you where engineering skills still come into play.
01:15So, the agent is relocation scout, which is a house hunting agent.
01:20And if you did this as just a one-time prompt that like points the agent to some listings and asked it to rank them, I mean, that'll work, but you're likely not going to find a house in a day, right?
01:33So, you want to build this as an agentic system that you can reuse.
01:39One that can persist knowledge outside of the session.
01:42You know, it could reload or query that knowledge later to make decisions even within a fresh context.
01:48So, when thinking about how to design an agent, the first engineering skill that I exercised is systems thinking.
01:56So, an agent is not the system, right?
01:59It's part of the system.
02:01And that system has files and tools, humans, even other agents.
02:07So, Relocation Scout sits inside of something bigger and it pulls in listings and signals about the neighborhoods.
02:15It weighs them against what I care about, and then it hands me back a ranked short list.
02:22So, I often hear people say, "Just let your coding agent build it, right?"
02:27And I think that's a mistake.
02:29Like, yes, my coding agent can build it, but before allowing it to do so, I need to think about the whole environment, the entire system, right?
02:40I want to like think about what's this agent's job?
02:43What does it depend on?
02:45What happens if it breaks?
02:47And I want to treat it like any other component where it has boundaries and responsibilities, has dependencies, you know?
02:55And in ways that it can fail.
02:57And that whole thought process, that's engineering.
03:02The second skill is workflow design.
03:05So, traditional software is full of workflows.
03:08We got CICD pipelines, right?
03:11We got like ticket life cycles, you name it.
03:15Agentic systems, they need that same kind of design.
03:19As much as we all love the slash goal command, an agent needs more than a goal, it needs a path.
03:26When we say, "Review this listing," that's a goal, but the workflow is what defines what actually has to happen, right?
03:33[clears throat] For example, the agent has to gather what it needs, it needs to weigh the listing against my criteria, and then act, right?
03:41And every run ends one of three ways.
03:44Either it's going to stop, it's going to retry, or it's going to escalate.
03:48So, that path is what shapes the rest of the architecture.
03:53Once I see how work moves through the system, I can make better calls about what context the agent needs, what parts I want the agent to handle directly, and when like a tool or person should take over.
04:07We all know the danger of one giant thing that does everything, right?
04:11We scoff when we see one gigantic class or big old function that's doing too much, right?
04:19Or bloated service with a gazillion endpoints.
04:22We call these code smells.
04:25Well, agentic systems, they have their own version of this.
04:28It's the giant prompts.
04:30And this starts innocently enough.
04:32Like in a instructions file, maybe I tell the relocation scout how to size up a listing.
04:39Fair.
04:40But then I hit an edge case, so I go back.
04:43I add a note for that.
04:45And then I remember and a safety rule, right?
04:49So, of course, that has to go in there.
04:52I'm proud of myself that I even remember to put that in there, right?
04:56And then, oh yeah, there's like one more very important exception.
05:01And before you know it, that prompt is doing everything.
05:05And your engineering spidey sense already knows that this is messy.
05:11So, why aren't you taking a step back to decompose it, right?
05:15Decomposition means spotting the distinct jobs that are hiding inside of that one blob and pulling them apart into separate pieces.
05:25So, if I look at the prompts for relocation scout in its entirety, it includes a reusable process for pulling and normalizing a listing, and then it's going to have like a fixed format for how to write the short list.
05:42It has a little section in there for how to calculate the commute, and then a chunky subtask on how to research the neighborhood.
05:50That's four different jobs crammed into a single prompt.
05:56And then you wonder why your agent is drifting and not sticking to the script.
06:01The script is too long.
06:04So, I'm not saying that, you know, you need to split things up for the sake of it, but the point is to make each part easier to reason about, right?
06:14That way it's easier to task.
06:16It's easier to change things when you need to.
06:19Now, decomposition is about breaking the system apart.
06:23Separation of concerns is about putting each responsibility in the right place.
06:27And this is where building agents start to feel really familiar to me because in traditional software, we'd ask things like, should this live in the controller or the service layer?
06:38Or, you know, is this business logic or presentation?
06:42So, when building agents, you may have the same sort of questions.
06:46There's just different places to put things.
06:49So, the process to normalize the listing, should that stay buried in the prompt?
06:54Or maybe it actually become a skill, right?
06:57Um I want every listing in the short list formatted the same way.
07:02So, that structured output should probably be defined in a schema.
07:07Isn't that what you would do if you were coding the system yourself?
07:10I would.
07:12And then the piece that calculates the commute, that can go in a nice little boring script.
07:19And and then researching the neighborhood, that's meaty enough, should probably be handled by a sub agent.
07:26Now, you're using the best tools for the job, and it's clearer where to find things within the system.
07:33Modularity is important in agentic systems as well.
07:37Just like we have reusable functions and classes and libraries, Now, I'm also thinking about reusable agent capabilities.
07:47And the clearest example of this is an agent skill.
07:50So, making a skill to normalize listings comes in really handy when you need to expand the agents' duties.
07:58For example, what if I broaden my house search to three cities?
08:02Every one of those markets can load the same skill.
08:05So, I wrote it once and they all can reuse it.
08:09So, this has now basically become a component that I can reuse across agents or even share with other people.
08:16Kind of like the same way that we lean on packages.
08:20And then, sub-agents are another kind of reusable module.
08:24So, a lot of people that I talked to, they don't quite get the point of sub-agents.
08:31Architecturally, they're sort of like functions, right?
08:34So, you give them one specific task to do.
08:37You call them when it needs to be done.
08:40And they can do it really well because that's all that they have in scope, right?
08:44They are They're not carrying the context of the entire session with them.
08:50So, like our neighborhood research sub-agent, we can drop that into any market or workflow and it works, you know, for what it's supposed to do.
09:00It's good in any hood.
09:02Um but like everything, deciding like what should be a module, that takes some judgment, right?
09:08Not everything should be reused.
09:10Some instructions are local to a given workflow, right?
09:15Might not be worth abstracting because sometimes that costs more than it saves.
09:19But this is just another engineering decision here, right?
09:22Agentic systems, they have these same sorts of trade-offs.
09:26Algorithmic thinking.
09:28This is one of the most important skills in agentic system design.
09:31Just because an agent can do something doesn't mean that it should, right?
09:36Some tasks are better handled by plain code.
09:40For example, calculating that commute time or deduping listings that I've already seen, an agent's model is better at things like fuzzy, you know, fuzzy stuff, judgment, ambiguity, um reasoning over messy input.
09:58And ignoring this distinction is where I see a lot of agentic systems get more complicated than they used to be.
10:06So, you're using a model, you're handing it every part of the task to do, and then you're getting frustrated when the output differs every day.
10:16Um but some of this stuff can be handled by just regular code, right?
10:22It'll be cheaper, it'll be more reliable.
10:24I promise you AI did not invent automation, right?
10:28We can use code while still using these systems.
10:31So, my rule of thumb here is if a task has an exact answer, reach for code.
10:38If it needs interpretation or judgment, that's when you can get the agent to do it, right?
10:44So, use code for determinism, use agents for judgment, and then use humans for authority.
10:51So, the agent decides which listings are worth a closer look, the code crunches the commute, filters out the ones I've already seen, and then I'm the one who approves
11:01actually booking a tour of the house.
11:04Free form text is fine when the human is the only one reading it.
11:08But when another system has to act on the agent's output, then you're better off with a contract, usually.
11:16So, we already do this everywhere in software.
11:19Anytime two systems talk, there's an agreed-upon shape between them, yes?
11:25So, agentic systems, they need that same discipline.
11:28For example, when relocation scout scores a house, it shouldn't just handy back a message and call it a day, right?
11:35That's lovely for me to read in that moment, but that is a dead end for the system.
11:42If the decision is like buried in like one of our sessions, nothing downstream can reliably find that.
11:49So, instead, it gets written into a structured shape to the agent's memory.
11:55And I use uh Compendium Wiki for this for for my agent memory layer on most of my agents.
12:03Um but in here, there's a decision, a score, a reason, and because it's structured, that memory becomes queryable.
12:10So, later I can ask Relocation Scout like, "Hey, show me every house rated four or better that has a commute of 15 minutes of or less, right?"
12:19And it can actually pull that because the score and the commute, they live in known places.
12:25They're not trapped in a session convo.
12:28And it's not just me that needs to like get this information.
12:31My shortlist step within the system, it reads these same fields um without a human in the loop.
12:38So, the agent's output is another step's input.
12:42And so, the contract is what makes that handoff safe.
12:46And you know, the best part is that defining the shape forces you to get really clear and specific.
12:53Because if you can't say what the output should look like, then you probably don't yet fully understand what you're asking the agent to produce.
13:02So, a prompt can run once and be done, right?
13:06But a useful agentic system has to be able to work in messy realities where webhooks get fired twice, or a run doesn't complete for whatever reason, and you need to retry the flow.
13:19So, the agent has to keep track of its state.
13:23Was this action already taken?
13:25If so, did the input change, right?
13:27If not, did the session crash or something like what parts of this can I safely retry?
13:34And this is not an exception, right?
13:36This happens all the time.
13:38So, you have to design for idempotency, which is where you can run the same thing twice and the second run doesn't cause a mess.
13:47Um and we do this in traditional software often.
13:50Um but with the agents, they add a little trap here because you can't trust the model because its outputs can vary, right?
13:59So, a retry risks the agent actually like rewording the request just enough that it might look like a brand new task.
14:09So, you have to enforce this in the system.
14:12Let's look at an example with our agent.
14:14So, let's say a new listing comes in um and relocation scout wants to email my realtor to ask about a viewing.
14:22So, after that action, the agent must log it to memory um that it it sent that email, right?
14:31And then the agent goes to my calendar and wants to just block that time out just in case.
14:37But it crashes before it could take that action.
14:41So, that run is only half done.
14:44Later, a lint pass runs.
14:47By the way, you got to have a lint pass with these things to kind of keep them healthy, but um during the lint pass, it notices that went out, but the calendar was never blocked.
14:59So, it retries the task.
15:02But it better not email my realtor again, right?
15:05That already happened.
15:06I don't want to be the annoying client um you know, I'm sending all these emails.
15:11So, it just needs to finish the part that it that didn't happen, which [clears throat] was blocking my calendar.
15:18But it only knows this because it's checking what the system wrote down, right?
15:24So, if we run this again, then the agent just completes what's missing instead of making a mess.
15:30Threat modeling is a really important skill when designing agentic systems.
15:35Everyone is on edge about this stuff these days, right?
15:39But, security engineering already taught us the basics.
15:42We need to validate your inputs, you know, give the least privilege needed, um and draw boundaries around what an action can touch.
15:51So, agentic systems, they need all of that.
15:54Our relocation scout will consume a lot of content from strangers, right?
16:00Um the agent needs to read the listing copy from the seller, uh forum threads, and neighborhood reviews from anonymous people on the internet.
16:12So, we need to treat all of that as untrusted input and make it very clear to the agent that this is evidence, not instructions.
16:23And then, after considering the input, you also want to think about what boundaries should you put in place around what the agent is able to do.
16:33For example, our agent can read listings and it can build shortlists all day long.
16:39Knock yourself out, right?
16:41But, I don't want it autonomously emailing sellers or booking tours or, heaven forbid, submitting offers on my behalf, right?
16:51So, those actions need to be walled behind like my approval, right?
16:57And when you draw that wall, what you've done is reduced the blast radius, and hopefully, that minimizes your exposure to risk.
17:07Now, every engineer can relate to what it feels like to inherit a system that you can barely understand.
17:13This is one of the key reasons why I don't just have my coding agent design my other agents because I know it'll be thrown together in a way that's technically works, but it's not maintainable,
17:25right?
17:26There's going to be a giant prompt likely, and even if the agent does decompose, I don't know.
17:33I'm just not convinced that it's going to separate the concerns properly.
17:38You know, I got trust issues.
17:39What can I say?
17:41Um so in my agentic systems, I make sure to bake maintainability into the system itself.
17:48Every level of the system has an agent's in D file that explains the workflow and where the policy lives, um supporting resources like skills and scripts and sub agents and most importantly how to keep its memory
18:03up-to-date.
18:04So anyone, human or agent, can come in a system and get oriented without needing to reverse engineer a bunch of prompts.
18:13In fact, that's the test, really.
18:16I design my agent so that even in a fresh context, they can jump right into the system and start cold, knowing exactly what to do.
18:26And this also really helps.
18:28So whenever I need to modify the system, right?
18:31So I can pretty much grab any harness and say, "Update this agent to do XYZ."
18:37And because the system is so well designed, the chances of it being successful at that update are much higher.
18:45If it does happen to run into any issues when trying to update, that's a signal to me that I need to improve the maintainability of the system.
18:55So designing agents is software engineering.
18:58The primitives are different, but the discipline is the same.
19:02We still need to understand the system.
19:04We need to define the workflow and know what flows into it.
19:08We still need to break the problem down and put responsibilities in the right place, make the right things reusable, determine which actors are best suited for which jobs, define contracts,
19:20manage state, design for safety, and make the system understandable.
19:26This is why building agents can give you that same thrill of building software.
19:32We're still building.
19:33We just moved up a layer.
19:36Thanks so much.