Building Great Agent Skills: The Missing Manual

AI Engineer · 20 min · 227 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Hello friends, I was dearly hoping to be able to come to the Air Engineer World's Fair, but family matters have intruded and I'm not able to make it.
00:07However, I will not be leaving you empty-handed.
00:09I'm going to give you the talk that I would have given in San Francisco.
00:13This talk is called The Missing Manual, How to Write Great Skills, and I think that the ability to distinguish good skills from bad skills is only getting more important.
00:22As developers, we seem to be pretty talented at finding different forms of hell for us to go to.
00:28In like a few years ago, we had tutorial hell, which is where you would go into a bunch of tutorials trying to learn something, not be able to piece it together, and sort of just get into this cycle you couldn't get out of.
00:40We had framework hell, where every other 10 minutes there was a JavaScript framework being announced, and you know, you had to learn the hot new thing all the time.
00:47And now, I think we have another version of hell, which is skill hell.
00:51Skill hell is where you have all of these skills available, freely available, that you can download, contribute to, you can figure out on your own, but you don't really know how the pieces all work together.
01:01You can't tell a good skill from a bad skill.
01:04And this means that people are trying to piece together these frameworks, trying to try everything that's out there all at once.
01:10And they sort of can't, or rather, they don't get the results that the skills themselves promise.
01:15This is true at an individual level, but it's also true at an organization level, too.
01:19Organizations have no way or no understanding on how to build good skills, how to take their operating procedures and turn them into things that an agent can do.
01:28you don't do that, then it's hard to get the bounty that skills can offer.
01:33Just one more skill, bro.
01:35That's kind of seems like what we're saying.
01:37And I feel a bit of guilt here, too, because we have Matt Pocock skills, which is my skills repo, which is one of the most popular engineering skill sets out there.
01:45And so, I feel like I want to help the people who use my skills get out of skill hell.
01:50So, how do we do it?
01:50How do we get out of this?
01:51Well, what is actually missing here?
01:54Well, in my opinion, the thing that we're missing is we don't know what makes a skill great.
01:58We can't yet look at a skill and go, "Okay, this skill is doing these good things and these bad things."
02:04There's no shared rubric, no framework for looking at a skill and making it better.
02:08And so, that's what I'm going to give you in this talk.
02:10I'm going to give you a skill checklist, a checklist of things you can look at inside the skill to make sure that it's doing what it says it's doing and ways you can improve it, ways you can write skills.
02:20This checklist looks like this.
02:22We start with the trigger of the skill, how the skill is invoked, and the decisions that you need to design there.
02:29Then the internal structure of the skill, how the skill is actually composed and laid out internally.
02:35Then number three is how do you actually steer using the skill?
02:38How do you get the skill to tell the agent what to do?
02:42Then four, how do you make the skill as small as possible?
02:45Because once we've got a working skill, we then need to basically maximize it, prune out all of the irrelevant stuff, prune out all of the no ops.
02:53There's one handy advantage of me not being in the room with you, which is you can immediately go and try this out because I've encoded all of this into a new skill in my repo called writing great skills.
03:03So, if you've got an immediate use case for this, then just go to this my skills repo, you know, just close this browser, get out of here, and go and use this skill
03:10to either improve your skills or write great new ones.
03:14But, let's now go through the checklist then.
03:15We have number one, the trigger, the way the skill is invoked.
03:19And in order to talk about this, I'm actually going to do a bit of comparison here, which is that my skills are often compared to another set of extremely
03:26popular engineering skills called superpowers.
03:28And I'm really often asked the question, "How do your skills compare to superpowers?
03:32What's the difference between them?"
03:34To understand that, we need to understand the difference between user invoked and model invoked skills.
03:39Anytime you have a skill, you can always invoke it manually.
03:43So, the skill sits on your file system, the agent will just be able to pull up the skill and understand what's in there.
03:49And you can always do that by communicating that to the agent.
03:52Doesn't always look like this forward slash depending on the harness, but that you can always use or invoke your skills.
03:58Another way that skills can be invoked is by the agent itself.
04:01These are called model invocable skills or model invoked skills.
04:04You can take a description, so the description of the skill always ends up in the agent's context, and the agent can look in that and go, "Okay, based on that description, I'm going to invoke the skill and I end up reading the skill.md
04:19file, which is where the meat of the skill is, into my context window.
04:23That's how you invoke a skill.
04:24That's what happens when a skill is invoked.
04:26So, this description serves as a kind of context pointer.
04:29It sits in the agent's context pointing to another file where the agent can go if it wants more context.
04:35But, that context pointer, you don't need to put it into the agent's context.
04:39It can just be invisible from the agent, and that is what we call a user invocable skill.
04:45So, some skills can only be invoked by the user because they don't have this context pointer.
04:50It's optional.
04:51For instance, we can see in my code base design here, this is a model invocable skill.
04:55It has a description that ends up in the agent's context window.
04:58But, if we look at my grill me skill instead, we can see it has disable model invocation true.
05:04This means that this little description here will only show to the user.
05:07It won't be visible to the agent.
05:09So, this then is tip number one.
05:10Decide if your skill is user invoked or model invoked.
05:14Now, you might think that model invoked skills are better, right?
05:17Because either the model can invoke it itself or the user can invoke it.
05:20It's more flexible.
05:22But, every time you add a model invoked skill into your agent's environment, it increases what I'm going to call the context load on that agent.
05:31It adds a new description, which is costing you tokens on every request, but also adding a different thing for the agent to think about.
05:39So, if you have a hundred model invoked skills, that's going to be a hundred descriptions inside the context for your agent.
05:45So, it seems to make sense then to either tamp down the number of model invoked skills or to just use all user invoked skills.
05:53But, user invoked skills have a different load, which is the more user invoked skills you have, the higher cognitive load on the user.
06:00In other words, the more things the user needs to keep in their head, the more skill you require from the pilot.
06:06And so, if we compare Matt Percot skills to Superpowers, Superpowers is primarily model invoked skills.
06:13It gives the agent superpowers.
06:16Whereas my skills, I much prefer to be in full control.
06:20That means I get to keep the context load on the agent as small as possible, but it does impose more of a cognitive load on me.
06:27So, I need to understand the skills really deeply in order to get the most use out.
06:31So, why have I done this?
06:32Why did I prefer user invoked skills?
06:34Well, every time you have a model invoked skill, it basically you get a cost in unpredictability.
06:41Because every time you have a context pointer pointing from one resource to another, the model may just choose not to follow it, you know, even if it's absolutely perfect for the task,
06:51it may just choose not to invoke the skill.
06:55I much prefer removing that level of unpredictability, imposing a bit more cognitive load on the user, and what you get is just you're removing a class of problem from even being a problem.
07:06Because this unpredictability leaves people to need to eval their skills to make sure they're being called at the right time, which is really nasty and it's a problem I prefer to avoid.
07:16But, what I'm hoping to show you here is that model invoked skills and user invoked skills both have their same costs.
07:22So, it's not an easy decision which one you choose.
07:24So, that then is the trigger, how the skill gets invoked.
07:28Now, let's talk about the structure, the internal layout of the skill.
07:31I think of there as being two main units that you need to put into most skills.
07:36These two units are the steps and the reference.
07:40The steps are the step-by-step procedure that the skill is going to walk through and the reference is any supporting information that helps it walk through those steps.
07:49You can have skills that have no steps and are only reference and you can have skills that are no reference and only a set of simple steps to walk through.
07:58But if you start thinking of skills as composed of these two units, it really helps just break them down a lot more.
08:03If we look at an example, one of my skills called 2 PRD creates a product requirements document out of the current context window.
08:10It's got three steps in it.
08:12So it finds the relevant context, it confirms the test seams with the user.
08:17So there's like a little human in the loop checkpoint there just to make sure we're not doing anything weird with the testing, which I find really important.
08:24And then we write the product requirements document.
08:26To handle those three steps, we've got two bits of reference material.
08:31We've got a little bit of reference on what is a test seam and then we've got a product requirements document template.
08:37So just a literal markdown template which is used to write the PRD.
08:42So this is a great way to write a skill from scratch.
08:45You work out if you need some steps, then you write those steps and you work out what reference material those steps need and you put it in a separate little spot in the skill, which is for reference material.
08:55However, there's a really important constraint that we need to think about, which is tip number three, we want to make the main skill.md file as small as possible.
09:04Every skill is composed of its description and then a skill.md file and then any reference material that branches off that.
09:11And this skill.md file, if we make it small, then we're saving in a bunch of different ways.
09:16Smaller skills are just easier to maintain, easier to audit, fewer words to think about.
09:21And every time you shave off a word, that is a token shaved, that multiple tokens shaved from your skills cost.
09:28So I do believe that small skills are really important both for maintainers and for users.
09:33One really useful way you can make your skill smaller is by thinking about the different branches of the skill, the different ways the skill can be used.
09:41Because if you have reference material that's only used in one branch, then that's a candidate for being removed from the main skill.md.
09:48For instance, if we look at my 2PRD here, we have two pieces of reference material, what is a test seam and the PRD template.
09:56Well, we need the PRD template every single time because we are always creating a PRD and we probably also need the what is a test seam information every time because we're always
10:05asking about the test seams.
10:07So, 2PRD, there's only one branch and all the reference material belongs on that branch, so it probably also belongs in the skill.md file.
10:15However, if we look at a different skill of mine, which is domain modeling, domain modeling does two things.
10:20It updates a local glossary called context.md and then it also creates architectural decision records.
10:27In other words, it's doing two different things or it might actually choose to do neither of these, in which case it doesn't need the template and it doesn't need the ADR template, either.
10:37So, in other words, domain modeling has two or maybe three branches and this means that we don't need to include the ADR template or the context.md template into the main skill.
10:47They can be moved into separate zones.
10:50The way you do that is you have the skill.md file, then you put it behind a context pointer and you point the context template to a separate markdown file inside the skills folder.
11:01That context pointer literally just says, "If you need the template or if you need to update the context.md file, go to this file."
11:08And I call that an external reference.
11:10It's a reference that's external to the skill.md that you can just easily reference the agent can pull in very easily because it's bundled along with the skill.
11:19So, this is a technique you can use for making the skill.md as small as possible, which is has so many benefits.
11:26Hide branching reference material behind context pointers.
11:29In other words, if you feel like your skill is going to be used in lots of different ways, then take the reference material that's relevant for those branches and hide them behind context pointers.
11:38So, that is structure.
11:39We need to think about making the skill.md super duper small.
11:43We need to think about the branches in our skill moving material out behind context pointers.
11:48And we need to think about steps and reference, which are the two main units inside a skill.
11:53Let's go next to steering, the actual ways we get the agent to do what we want it to do.
11:58And for me, steering comes down to one really cool technique, which is the kind of main thing I want you to get from this talk.
12:05This technique fixes this issue, which is the agent doesn't do what I want.
12:11In other words, I specify something in the skill, I think that I've been clear, and then it just doesn't do the thing.
12:16Now, I think the main reason this happens is because you're not using a technique called leading words.
12:22The idea of leading words, or light vert if you like literary theory, I suppose, is that there are certain words that pack in a bunch of meaning into a very small space.
12:33These leading words are really powerful with agents because you put the leading word in the skill itself in the text, and then the agent will repeat the leading word back to itself as part of its operations, as part of its thinking tokens,
12:47and as part of its output to you.
12:49And then, because it's re-emphasizing that word and that word hopefully describes what you want from the agent, that then goes and changes its behavior.
12:57Let's make this more concrete with an example.
13:00So, let's imagine that we have a problem, which is a classic problem with agents, which is that they code layer by layer.
13:06In other words, if you give them a big tranche of work to do, they will generally code up all of the database layer, then all of the schemas, then all of the API endpoints, then all of the front end.
13:15They don't do the sort of typical um human thing, which is to seek feedback early on, get something small working, and then expand out from there.
13:24Now, we can try to encourage the agent to do that by just saying, you know, don't code layer by layer, Um make sure that you create a small slice first and then go from there.
13:32But what if instead we use the leading word?
13:35We said, "vertical slice" is our leading word.
13:38We want to slice up the work instead of horizontal slices into vertical slices.
13:44A vertical slice is a pretty well-known terminology in development, and so this will hopefully trigger the agent's priors and it will understand what we mean.
13:52We don't just have to like have a two-word skill where it just says vertical slice.
13:56What we're doing is we're packing lots of meaning into a relatively short phrase that we then repeat throughout the skill.
14:02The cool thing about this technique is you can know if it's worked because you say vertical slice in your skill, and then you'll notice in the reasoning traces that it's saying, "Okay, we're going to do this as a thin vertical slice."
14:13Then you should get better implementation plans.
14:15Everyone I've explained this technique to sort of feels like, "Oh yeah, I've been doing that for a while.
14:20I've been using these little phrases to try to encourage the agent to do what I want."
14:25All I'm asking you now is to use those consistently within your skills and watch in the thinking traces as the agent adopts your way of doing it.
14:34So often if the agent isn't doing what you want, you need to make your leading words more consistent, more powerful, and look for others because, you know, English is a pretty wide API in terms of different functions you can call, different things you can experiment with, and there are many leading word candidates out there.
14:52And agents are actually pretty good at helping you think of them.
14:54Another little lever you can use with agents is sometimes the agent just doesn't do enough legwork.
15:01What I mean by this is that, okay, we're on a step, let's say, and maybe the step is to ask clarifying questions or to explore the code base, and the agent just doesn't do enough of it.
15:12It doesn't put enough effort into that particular step.
15:15A real classic case of this and something that I have found almost everywhere it is plan mode.
15:22Because in plan mode, we have two steps.
15:24We have ask clarifying questions and then create a plan.
15:28And what I have found in every single implementation of plan mode I've tried is that ask clarifying questions just, you know, it doesn't ever do enough legwork.
15:36It sees that its ultimate goal is to create a plan and so it just does a small amount of legwork with ask clarifying questions, ask you a couple of things, and then eagerly creates the plan.
15:46So, what was my solution here?
15:48Instead of doing plan mode, I instead have a skill called grill with docs, which is kind of my ask clarifying questions phase.
15:55And then, I split that up into a separate skill.
15:58So, I split the planning into its own skill.
16:02So, grill with docs now is its own skill where the agent only sees that part of the process.
16:08And then, after grill with docs completes, we then go and do 2 PRD.
16:12In other words, we have step one and step two, but the agent only sees one step at a time.
16:18So, this is a really cool technique for increasing legwork on the step that you're on by hiding the future goal, hiding the future steps.
16:25It's not always necessary to split skills into individual steps, but in particular cases where you really want an extra chunk of legwork, it really there's no technique like it.
16:37It works very, very well.
16:38So, that is steering using leading words to capture what you want in small reusable tokens and then making sure that it's doing the right amount of legwork per step.
16:47So, let's head now into pruning.
16:50Now, pruning really is just a quick fire set of failure modes, different things that you can get wrong.
16:55And the first is fairly obvious is we do not want massive skills.
17:00Massive skills are usually a kind of symptom of something else going wrong.
17:04So, a symptom of one of these other failure modes.
17:06And the first one is pretty simple.
17:08Don't repeat yourself.
17:10You need to make sure you're watching out for duplication.
17:13And in general, I like to have every part of the skill to have a single source of truth.
17:19In other words, if you have a piece of reference material like the PRD template, let's say, or something even smaller like what is a test seam, you make sure that you don't repeat that in several places or like cover multiple steps in multiple places.
17:32Just make sure each part has a single source of truth and you're not repeating yourself even across reference material, too.
17:38The next way that skills get big is via sediment.
17:41And sediment is just a classic thing when people are working on the same set of docs, really, which is that everyone starts contributing to a shared markdown file.
17:52People add their own stuff.
17:53They don't feel brave enough to delete and modify anyone else's.
17:57And so you just end up with this huge amount of sediment with often irrelevant material for the skill, especially stuff that hasn't been laid out properly.
18:05With a skill with a lot of sediments, you really need to look at structure.
18:08That's the first thing you need to do.
18:10You need to make sure that the stuff that's been added is relevant for all branches.
18:13If it's not, then move it into the correct branches.
18:16Or if it's just totally irrelevant, maybe just remove it or kill it.
18:20Or maybe there's stuff in there that's totally stale, in which case you just need to kill it dead.
18:24The next failure mode is really common when an agent writes your skills, which are no-ops.
18:30So things inside the skill that appear to do something but don't actually influence the agent's behavior inside the context of the skill.
18:38Let's imagine we have an implement skill and we have an entire paragraph of the skill that tells the agent to write a long detailed commit message.
18:45What would happen if you just deleted that paragraph?
18:48Well, the agent would probably still write a decent like long commit message.
18:52People ask me a lot how I get my skills so small, and it's just using these techniques, using deletion tests, using um making sure that I compact things into leading words, I don't have anything irrelevant in there, and I don't have any sediment.
19:06And that finally brings us to the full sweep of things.
19:09Number one, we check the trigger.
19:11We make sure that it's firing at the right times.
19:13We check whether we're imposing context load or cognitive load.
19:17With structure, we think about branches.
19:19We think about structuring things into steps and reference.
19:23And we make sure that material that's only relevant for one branch is outside of the main skill.md.
19:29With steering, we're thinking about condensing text down into leading words and watching those leading words appear in the reasoning traces.
19:37And we're also thinking about legwork.
19:38Should we break this skill down further to increase its focus on the current phase by hiding the future phase phase from it.
19:46And with pruning, we're doing a final pruning pass over the entire skill, watching out for sediments, watching out for crud, and watching out especially for no-ops.
19:54Now, all of this stuff, the best way to get started with this framework is inside this skill, inside the writing great skills skill.
20:02You can check it out from my papago skills, download it, use it to improve your own skills, and maybe even use it to run over some community authored skills so you can check that the skills that you're actually pulling in are any good.
20:15If you want to follow along with my stuff, then I have a newsletter up on aihero.dev.
20:19And my plans for the next few months are to release an AI coding crash course, which is an intro to a lot of the stuff I've been talking about and how you get off the ground working with engineering and AI.
20:30I hope that what I've given you is enough to help you escape from skill hell or at least try to make the bitter journey out of there.
20:38I'm so sorry not to be able to attend in person, but thanks for watching.
20:41I'll see you very soon.