Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind

AI Engineer · 21 min · 244 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Yes, so hi everyone.
00:13My name is Philip.
00:14I'm based out of Germany.
00:16I'm part of the Google DeepMind team mostly working on Gemini API and agents.
00:20And we are going to talk about why you should not ship skills without eval.
00:24And maybe before we start, I need a little bit of your help.
00:27So if you could raise your hands if you use coding agents to write code.
00:32So yeah, hopefully every every hand goes up, right?
00:35And do you use skills with it?
00:38Okay.
00:39Do you have evals for those skills?
00:41Okay, yeah, that's um not a lot of hands.
00:44Everyone uses skills, no one has evals.
00:46Hopefully we can fix that today.
00:47And like very important is like why checks fail in productions.
00:51And um Skill Bench is a very popular and nice like eval or benchmark which um index like over like 50,000 skills from like it up and like try to look into him and almost none of those skills had evals.
01:04Most of them were AI written um not really tested and it's very hard to know if your skill is good or bad because like agents are really non-deterministic.
01:14So you might not know if your uh task fails because your skill is bad or if your task fails because it's way too challenging for the model.
01:21So um very important um before we go into it there's I want to like really make sure that we know the difference between the agents we use and the agents we build.
01:31Um most of us use agents for writing code, doing productivity work.
01:35That's the agents we use.
01:36It's like anti-gravity, cursor, Claude code.
01:39And there are you are the engineer and you have context about skills, right?
01:43If you write some prompt to I don't know, like help me build a new Gemini API feature.
01:49And if your agent does not invoke the skill on the first time, you will notice it very quickly.
01:54You stop your your task and reprompt it or like use slash commands for for triggering those skills.
02:00When you build an agent inside your application for consumer or customers, they have no idea about what a skill is.
02:07They don't start their prompt with use customer support skill to like help me refund or use refund skill to help me solve my problem.
02:16So, there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills in like the context there.
02:27And what is a skill?
02:28I mean, every one of us knows hopefully in by now what a skill is.
02:31It's like basically really a folder with a skills.md file in it and then some additional assets to make that skill really work.
02:38And the big difference with skills is that they work on progressive disclosure.
02:42So, most of the skills start very small.
02:45So, you have the title and a description.
02:48The description is normally part of the model's context.
02:51So, the model knows when to use the skill.
02:54Second layer is we have a skills body with more instructions, more details, and hopefully more references to external files.
03:01And then you can really go deep in those reference files where there's all of the context the model needs to discover to to solve the task.
03:07And I like to differentiate between two kinds of skills.
03:11So, they are capability skills and preference skills.
03:13Capability skills teach models something they cannot do consistently at the moment.
03:19Maybe it's like, I don't know, like tracing some logs, creating a new React app.
03:24And those capability skills are temporary.
03:27So, the better our model gets, the more likely it is that we can remove those skills.
03:32And Evals will tell us when we can retire skill and when not.
03:35And then we have preference skills.
03:37Those are more durable, mostly encode some references.
03:41So, if you have a specific workflow in your team or a specific style language or other preferences which are very specific to your company, Um will have or create
03:53preference skills and those uh preference skills are then protected with e-walls where because most of like the foundation models might not uh integrate the knowledge which is very specific to your use case or your domain.
04:06And preference skills are very valuable, so we really want to make sure that those are working and we don't like update our agents to uh degrade performance.
04:15So, do skills work?
04:17Yes, they do work and I going back to a skills bench which has an update of 1.1 which has evaluated all kinds of open and closed models in different harnesses
04:27showing that skills on average improve the performance by roughly 15%.
04:32Skills bench covers around 100 different tasks uh based on like coding and also productivity across different languages.
04:40It's uh openly available and they have a very nice website, a very nice leaderboard, are also very open for uh community contributions.
04:47And then they did a second analysis on self-generated or AI-generated skills, right?
04:53It's very easy if you are in a coding agent and you work on something, tell the model create a skill and then it writes a skill.md file.
05:00We maybe look at it very closely.
05:02It roughly covers what we want to do and then we just accept it and start using it.
05:08And what I found out is that human-written skills are the best we can provide.
05:15Uh AI-generated skills can uh impact performance negatively.
05:20And that skills or skills.md files should be below 500 lines of words.
05:25So, if you have your laptop open and have a skill available, if you open that and if it's above 500 lines, you should definitely look at the skill after our session.
05:34And um the last topic about what is a skill and how a skill works, uh we have different ways of triggering our skill, right?
05:42We can have a model-triggered skill meaning uh based on the context and the description, the model decides to use or read a skill to, uh uh, get more context to solve a task.
05:52And then there are user-invoked skills.
05:54And I think people underestimate how powerful user-invoked skills are.
05:59Um, and they most of the time just accept, uh, the overhead by pro- like adding it into the context.
06:06I have like many user-invoked skills for more workflow type of tasks like creating a pull request, uh, staging documentation, and all like of the very uh, normal dev work which could be run in a script
06:21should most likely be a user-invoked, uh, skill.
06:25And when you build agents for customer, you don't have those user-invoked skills.
06:29We are only working in the model-invoked skills, and that's where we are like focusing on for the small eval section we are going to look in a second.
06:36So, writing skills, um, is an important topic.
06:40Uh, we're going to look at eight, um, examples on how or tips on how you can write good skills.
06:48And most importantly, if you work with model-invoked skills is the description because the description are most of the time two sentences we provide to the system instruction to help the model
06:58know when it should use a skill or not.
07:00And it's bad if your description is too weak because then it might trigger too often or the it might not be triggered if you need it.
07:09So, very important is the the why and the how for the model.
07:13So, why it should use that skill and then how it should use that skill.
07:16Um, very common is like use that skill if you are working on a React application, for example.
07:21And then, of course, the when.
07:23And we should write directives instead of essays.
07:26So, we should not say something like, "Hey, the Interactions API is recommended for multi-chat, um, multi-chat because it handles like session state and it's like where you should be way more directive like use the Interactions API if you're working on like a chat application.
07:42So, you need to give the model like clear instructions and directives on when it should use the skill and how it should use the skill.
07:50And similar to what we have seen in the skills bench results, we should keep the skill lean and layer information.
07:57So, the description is the cost you always pay on every model invocation.
08:02So, on every model call, the description is part of the model context.
08:05So, you always pay that 100 200 tokens cost and you don't want to have a super long description because then you always have to pay that.
08:12When you have a very long skill MD file, it will be always read into context when the model decides to read the the skill or to use the skill.
08:20Uh which can be expensive as well.
08:22That's why we want to keep it as concise as possible, but still include all of the reference and details for the model to solve the task.
08:29And then of course, the layer three is like we can have those reference files where the model needs to like um go really deep into a very specific task.
08:37And a good example for this is like if you are working in like maybe a multi-cloud environment and you have a skill to deploy your application, you might need instruction to deploying to AWS and deploying to Google Cloud.
08:48Those should not be part of your skill MD file.
08:50Those should be references.
08:52That you have a reference for AWS, reference for Google Cloud, maybe a reference for Azure so that the model can basically explore based on the context where it should go to get all of that information.
09:03Then we should set the right level of freedom.
09:05Um I see many people very clearly describing the exact workflow in a skill.
09:11Step one, go there.
09:12Step two, do this.
09:13Step three, do this.
09:14If you have those type of use cases, you should not use skills.
09:17You maybe you should write a script because if the the process or the workflow is always the same, you don't need to waste models and tokens for that exercise.
09:27You can create a script.
09:28You can tell the model use that script to run a specific workflow.
09:31So, rather define goals and constraints.
09:34So, if you need to like deploy to your update or stage your documentation, describe how the model can do that.
09:42Or like for your database updating a config, you should not say like read the config, update the port, and then like deploy again.
09:49The model knows what to do.
09:50Just like hey, if we need to change the config, here's the file, make the change.
09:55Then uh don't skip negative cases.
09:57So, we always look at the when we want to use this skill, but most of the time we don't look at when we don't want to use this skill.
10:04So, if we have a description for our skill which says use it for web development tasks, it might over trigger.
10:12Maybe you work with React, maybe you also work with Angular, and the model always loads the skill if you are working in like a web development environment, but if you are very specific for like hey, only use that skill for React components or for Tailwind CSS,
10:27then the model knows hey, that's very specific for one to use.
10:31And with Evals, we can also identify those.
10:34Um And then test early.
10:36So, that's what we are going to look at.
10:37We should really try to test when you create a new skill.
10:40Always try to create 10 of 20 prompts.
10:43I like to create five for like the happy path.
10:46So, when do I want to use that skill?
10:48Five when I don't want to use that skill just to make sure the model is not over triggering the skill and confusing itself.
10:54And then if you have already some customer or production traces, try to include those as well because nothing is better than than real-world data.
11:03And then tip seven which is quite new and I have to give all credits to Matt.
11:07So, if you don't know Matt, he's a great AI educator and you should definitely follow him.
11:11He published a tweet and also a skill on like killing all of the no-ops.
11:15And what he found is that AI generated skills tend to include a lot of no-ops.
11:20And no-ops basically is an instruction which does nothing to change the agent's behavior.
11:26It's like before making an implementation easy to read.
11:29Like the model knows how it when it should make something easy to read or write clear high-quality code.
11:36I mean, like that's what we expect from the model to do without telling it really.
11:39So, um definitely look at those no-ops.
11:41He have has published a very good skill in in his like skills repository.
11:46Uh and then last but not least, uh know when you should retire skill.
11:51Um skills are not there to live forever.
11:54Models get better, behaviors change, expectation change, um the environment changes.
12:00So, um always try to run evals with and without the skill enabled.
12:05And if the model achieves the performance without even like triggering the skill, you know you can retire that skill, save the cost uh for your tokens, and then also
12:15um don't keep like it redundant.
12:18So, save cost at the end and maintenance also as well.
12:21And to look at a little bit of a practical example and also how you can create your own small eval or eval harness for skills.
12:29Um earlier this year we wanted to create a new skill for the Gemini Interactions API.
12:34So, the Gemini Interactions API is our new interface for working with Gemini models and with agents.
12:40And the Interactions API was released after the last training of Gemini.
12:45So, the model or Gemini 3 and like 3.1 or even 3.5 has no context about what is the the Gemini Interactions API.
12:53So, we decided, "Okay, let's look uh at creating a skill to help the model create good code for the Interactions API, to use the latest models."
13:03And to do that, we created 117 test cases.
13:07Those are uh based on like data we see uh from real users trying to generate uh Gemini code from um synthetic generated uh test cases, and also from like feedback we see people like, "Hey, the model is like using Gemini 2.0
13:21even if we are already on 3.0." And the the end result was that we improved the the performance up to like almost 90% for generating valid interactions API code with the latest Gemini models.
13:36And to do this, we basically only needed like two very simple uh assets.
13:43So, one of that was a JSON file with all of our test cases.
13:49And it's very like no clear structure.
13:51It's like, "Hey, we have a prompt."
13:53That's basically what we expect the user to provide.
13:55We have a language because we wanted to test the skill against TypeScript and Python.
14:00Uh we have a should trigger.
14:01That's basically there to tell us if the agent should read the skill or not read the skill.
14:07And then we have different expected checks.
14:09Uh we look at them in a little bit.
14:11Those are basically uh very simple asserts uh for that prompt if it should trigger or not.
14:17And then we have a very basic Python script which runs a coding agent.
14:20In this case, it was the Gemini CLI which uh passes the output and returns runs it so we can like take a look at the the outcome whether we have valid code for the interactions API
14:31or not.
14:32And uh most of the tests or evals for skills can be regex.
14:39It's like very amazing how good of regex you can write using coding agents.
14:45And it's really for us it was all about, "Okay, do we use the correct SDK?
14:49Do we use the correct um model?
14:51Do we use the correct methods?
14:53Do we use any old patterns?"
14:55And we created um very basic asserts for all of those cases, which are very cheap to run.
15:01So, we can run our skill against the evals many times.
15:04So, if a new model releases, we have a very easy way to update those asserts to the latest model IDs.
15:10And it's very cheap to run for it well because we don't need to use like LLM as a judge.
15:15But of course, you can use LLM as a judge if you have like more complex skills which need to look at the whole traces or the whole steps taken.
15:23And a very easy case is like you just create LLM as a judge with a rubric on like what you want to look at, and then like take the output, put it through the LLM as a judge, try to get a pass or a fail, and then if it fails, look at the data,
15:37and then like try to um improve your skill based on that.
15:41And that's also how we now uh eval skills at Google DeepMind.
15:45So, the we don't use YAML, but it's like just as like uh an example.
15:51Uh we have um tests or evals alongside every skill we have internally at Google DeepMind.
15:59Um every test has multiple cases with like a prompt.
16:02Uh we all run them in like clear workspaces, so you can define your workspace or environment if it should include additional files like your application environments.
16:12You have uh startup commands, which basically preloads or installs libraries into the environment.
16:19And then you have script evals or data.
16:21Those are those regex where we look at all of the traces to see what the skill triggered, was a certain command run, was a certain CLI run.
16:29And then we also have LLM as a judge where we have some expectations, which are basically matched against like hey, did it trigger the skill, did it run a certain bash command to like
16:39also evaluate it.
16:40And we run them on every change to the skill.
16:43So, if a change happens to or like a diff to the skill file, the eval will be run, and there will also be a result, and the change will not be merged if it is not improving the test cases.
16:56So, we always have those regression tests for every change to the skill, and you can only change the skill if it improves the eval or add new evals.
17:05And um yeah, that that's how we we we basically manage it.
17:11And then last but not least, uh 10 examples for best practices for skills.
17:16You don't need to take photos.
17:17They are in the blog post I can share later.
17:20So, um the I mean, we had it many, many times.
17:24The the skill skill description is very important.
17:26Uh we have seen 50% of the failures uh because the skill was not triggered correctly because the prompt of the user was not uh detailed enough for the model to understand, "Hey, I need to use that skill to solve that task."
17:38And especially if you build agents for others, they are not aware of the skill descriptions you have for your model and for your skill.
17:46So, they might write something very um shallow and then the model needs to know, "Okay, I need to trigger that skill."
17:54Um we should write directors over passive information.
17:57So, we should always think about it.
17:59You should tell the agent what to do or not what to do and not just like, "Hey, if you feel happy today, please use the skill."
18:06Um include negative tests.
18:08Uh we always forget negative tests.
18:10Start small.
18:11Even like 10 to 20 skill eval samples are better than nothing.
18:15You will be surprised on how much you will find even from like five to 10 examples.
18:20And then like definitely create outcomes, not paths.
18:23Uh we don't want to test if the model loads the skill on like the first turn.
18:27We really want to test if it can achieve the task based on the prompt.
18:31And if it loads the skill, it loads the skill.
18:33If not, then not.
18:34If it um loads the skill after five turns, that's also okay.
18:38Then we want to have isolated runs because coding agents are very good at finding or cheating.
18:45So, if you run inside uh your existing environment, it might look up previous chats or it might look up some other executions and then like try to cheat it and get the context from the skill without even using the skill.
18:59Then definitely run more than one trial when running evals.
19:02Like agents, our models are non-deterministic.
19:06Maybe the first one works, the second one doesn't.
19:08So, always run the to six uh trials per case and to measure reliability Uh um test across different harnesses if you work with like or if you have employees
19:19or um people working with like different harnesses, not only just evaluate against Claude or anti-gravity.
19:25If you have people working with cursor, try to include them as well because agent harnesses behave differently and of course model behaves differently.
19:32So, maybe your skill is very good with a Gemini but very bad with Codex and then you have uh customers, consumers using your harness with Codex and then it fails.
19:41And then um create your evals.
19:43So, if your um model is good enough that it doesn't need the skill anymore, keep that eval.
19:50You don't need to throw that eval away because you throw the skill away.
19:53You can keep that eval to make sure that the model or the agent keeps the performance and as soon as you start seeing some degradation, you can reintroduce the skill.
20:01You can maybe tweak some other tools or pieces to keep like the the performance up and then really detect when you can retire skill and you will be very surprised with all of the model updates
20:12how fast you can retire skill which you might need it like six months ago but not today anymore.
20:18And I have some homework for you.
20:20So, if you are back from holiday on Monday, um pick uh the most used skill and write five test prompts.
20:28Uh you can also use your coding agent and ask it to see look at your trajectories which are my most used skills and then try to create some some skills.
20:36It's you have seen it's like very easy to write your eval harness.
20:40It's like a JSON or YAML file and then like some Python script which runs your coding agent or your agent harness and then like look at the outcome.
20:48Definitely uh try to look at the removing no-ops.
20:51Maybe it does not change the eval performance but it helps you save cost because all of the tokens which are not helpful or not changing the agent behavior are money you will like spend.
21:03So, look at um writing great skills uh from Matt.
21:07It's you can find it on on GitHub and then also run ablation test.
21:11So, run always evals with your skill loaded and without your skill loaded.
21:16Only that way you will know when you can retire skill or if a skill is really helpful for your performance.
21:21So, don't ship skills without evals.
21:25Thank you.
21:25[applause]