WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=0vphxNt4wyk

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/0vphxNt4wyk.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:13.480
Yes, so hi everyone.

s3
00:00:13.480 --> 00:00:14.320
My name is Philip.

s4
00:00:14.320 --> 00:00:16.160
I'm based out of Germany.

s5
00:00:16.160 --> 00:00:20.080
I'm part of the Google DeepMind team mostly working on Gemini API and agents.

s6
00:00:20.080 --> 00:00:24.840
And we are going to talk about why you should not ship skills without eval.

s7
00:00:24.840 --> 00:00:27.840
And maybe before we start, I need a little bit of your help.

s8
00:00:27.840 --> 00:00:32.759
So if you could raise your hands if you use coding agents to write code.

s9
00:00:32.759 --> 00:00:35.080
So yeah, hopefully every every hand goes up, right?

s10
00:00:35.080 --> 00:00:38.360
And do you use skills with it?

s11
00:00:38.360 --> 00:00:39.320
Okay.

s12
00:00:39.320 --> 00:00:41.880
Do you have evals for those skills?

s13
00:00:41.880 --> 00:00:44.240
Okay, yeah, that's um not a lot of hands.

s14
00:00:44.240 --> 00:00:46.280
Everyone uses skills, no one has evals.

s15
00:00:46.280 --> 00:00:47.960
Hopefully we can fix that today.

s16
00:00:47.960 --> 00:00:51.520
And like very important is like why checks fail in productions.

s17
00:00:51.520 --> 00:01:04.120
And um Skill Bench is a very popular and nice like eval or benchmark which um index like over like 50,000 skills from like it up and like try to look into him and almost none of those skills had evals.

s18
00:01:04.120 --> 00:01:14.760
Most of them were AI written um not really tested and it's very hard to know if your skill is good or bad because like agents are really non-deterministic.

s19
00:01:14.760 --> 00:01:21.600
So you might not know if your uh task fails because your skill is bad or if your task fails because it's way too challenging for the model.

s20
00:01:21.600 --> 00:01:31.040
So um very important um before we go into it there's I want to like really make sure that we know the difference between the agents we use and the agents we build.

s21
00:01:31.040 --> 00:01:35.480
Um most of us use agents for writing code, doing productivity work.

s22
00:01:35.480 --> 00:01:36.520
That's the agents we use.

s23
00:01:36.520 --> 00:01:39.800
It's like anti-gravity, cursor, Claude code.

s24
00:01:39.800 --> 00:01:43.640
And there are you are the engineer and you have context about skills, right?

s25
00:01:43.640 --> 00:01:49.520
If you write some prompt to I don't know, like help me build a new Gemini API feature.

s26
00:01:49.520 --> 00:01:54.440
And if your agent does not invoke the skill on the first time, you will notice it very quickly.

s27
00:01:54.440 --> 00:02:00.320
You stop your your task and reprompt it or like use slash commands for for triggering those skills.

s28
00:02:00.320 --> 00:02:07.600
When you build an agent inside your application for consumer or customers, they have no idea about what a skill is.

s29
00:02:07.600 --> 00:02:16.520
They don't start their prompt with use customer support skill to like help me refund or use refund skill to help me solve my problem.

s30
00:02:16.520 --> 00:02:27.120
So, there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills in like the context there.

s31
00:02:27.120 --> 00:02:28.520
And what is a skill?

s32
00:02:28.520 --> 00:02:31.400
I mean, every one of us knows hopefully in by now what a skill is.

s33
00:02:31.400 --> 00:02:38.560
It's like basically really a folder with a skills.md file in it and then some additional assets to make that skill really work.

s34
00:02:38.560 --> 00:02:42.959
And the big difference with skills is that they work on progressive disclosure.

s35
00:02:42.959 --> 00:02:45.800
So, most of the skills start very small.

s36
00:02:45.800 --> 00:02:48.600
So, you have the title and a description.

s37
00:02:48.600 --> 00:02:51.920
The description is normally part of the model's context.

s38
00:02:51.920 --> 00:02:54.040
So, the model knows when to use the skill.

s39
00:02:54.040 --> 00:03:01.160
Second layer is we have a skills body with more instructions, more details, and hopefully more references to external files.

s40
00:03:01.160 --> 00:03:07.840
And then you can really go deep in those reference files where there's all of the context the model needs to discover to to solve the task.

s41
00:03:07.840 --> 00:03:11.120
And I like to differentiate between two kinds of skills.

s42
00:03:11.120 --> 00:03:13.680
So, they are capability skills and preference skills.

s43
00:03:13.680 --> 00:03:19.239
Capability skills teach models something they cannot do consistently at the moment.

s44
00:03:19.239 --> 00:03:24.160
Maybe it's like, I don't know, like tracing some logs, creating a new React app.

s45
00:03:24.160 --> 00:03:27.120
And those capability skills are temporary.

s46
00:03:27.120 --> 00:03:32.320
So, the better our model gets, the more likely it is that we can remove those skills.

s47
00:03:32.320 --> 00:03:35.920
And Evals will tell us when we can retire skill and when not.

s48
00:03:35.920 --> 00:03:37.600
And then we have preference skills.

s49
00:03:37.600 --> 00:03:41.640
Those are more durable, mostly encode some references.

s50
00:03:41.640 --> 00:03:53.520
So, if you have a specific workflow in your team or a specific style language or other preferences which are very specific to your company, Um will have or create

s51
00:03:53.520 --> 00:04:06.360
preference skills and those uh preference skills are then protected with e-walls where because most of like the foundation models might not uh integrate the knowledge which is very specific to your use case or your domain.

s52
00:04:06.360 --> 00:04:15.560
And preference skills are very valuable, so we really want to make sure that those are working and we don't like update our agents to uh degrade performance.

s53
00:04:15.560 --> 00:04:17.079
So, do skills work?

s54
00:04:17.079 --> 00:04:27.880
Yes, they do work and I going back to a skills bench which has an update of 1.1 which has evaluated all kinds of open and closed models in different harnesses

s55
00:04:27.880 --> 00:04:32.480
showing that skills on average improve the performance by roughly 15%.

s56
00:04:32.480 --> 00:04:40.280
Skills bench covers around 100 different tasks uh based on like coding and also productivity across different languages.

s57
00:04:40.280 --> 00:04:47.720
It's uh openly available and they have a very nice website, a very nice leaderboard, are also very open for uh community contributions.

s58
00:04:47.720 --> 00:04:53.840
And then they did a second analysis on self-generated or AI-generated skills, right?

s59
00:04:53.840 --> 00:05:00.480
It's very easy if you are in a coding agent and you work on something, tell the model create a skill and then it writes a skill.md file.

s60
00:05:00.480 --> 00:05:02.400
We maybe look at it very closely.

s61
00:05:02.400 --> 00:05:08.800
It roughly covers what we want to do and then we just accept it and start using it.

s62
00:05:08.800 --> 00:05:15.720
And what I found out is that human-written skills are the best we can provide.

s63
00:05:15.720 --> 00:05:20.280
Uh AI-generated skills can uh impact performance negatively.

s64
00:05:20.280 --> 00:05:25.440
And that skills or skills.md files should be below 500 lines of words.

s65
00:05:25.440 --> 00:05:34.600
So, if you have your laptop open and have a skill available, if you open that and if it's above 500 lines, you should definitely look at the skill after our session.

s66
00:05:34.600 --> 00:05:42.919
And um the last topic about what is a skill and how a skill works, uh we have different ways of triggering our skill, right?

s67
00:05:42.919 --> 00:05:52.720
We can have a model-triggered skill meaning uh based on the context and the description, the model decides to use or read a skill to, uh uh, get more context to solve a task.

s68
00:05:52.720 --> 00:05:54.320
And then there are user-invoked skills.

s69
00:05:54.320 --> 00:05:59.960
And I think people underestimate how powerful user-invoked skills are.

s70
00:05:59.960 --> 00:06:06.720
Um, and they most of the time just accept, uh, the overhead by pro- like adding it into the context.

s71
00:06:06.720 --> 00:06:21.440
I have like many user-invoked skills for more workflow type of tasks like creating a pull request, uh, staging documentation, and all like of the very uh, normal dev work which could be run in a script

s72
00:06:21.440 --> 00:06:25.000
should most likely be a user-invoked, uh, skill.

s73
00:06:25.000 --> 00:06:29.440
And when you build agents for customer, you don't have those user-invoked skills.

s74
00:06:29.440 --> 00:06:36.000
We are only working in the model-invoked skills, and that's where we are like focusing on for the small eval section we are going to look in a second.

s75
00:06:36.000 --> 00:06:40.800
So, writing skills, um, is an important topic.

s76
00:06:40.800 --> 00:06:48.040
Uh, we're going to look at eight, um, examples on how or tips on how you can write good skills.

s77
00:06:48.040 --> 00:06:58.800
And most importantly, if you work with model-invoked skills is the description because the description are most of the time two sentences we provide to the system instruction to help the model

s78
00:06:58.800 --> 00:07:00.919
know when it should use a skill or not.

s79
00:07:00.919 --> 00:07:09.200
And it's bad if your description is too weak because then it might trigger too often or the it might not be triggered if you need it.

s80
00:07:09.200 --> 00:07:13.040
So, very important is the the why and the how for the model.

s81
00:07:13.040 --> 00:07:16.760
So, why it should use that skill and then how it should use that skill.

s82
00:07:16.760 --> 00:07:21.080
Um, very common is like use that skill if you are working on a React application, for example.

s83
00:07:21.080 --> 00:07:23.040
And then, of course, the when.

s84
00:07:23.040 --> 00:07:26.400
And we should write directives instead of essays.

s85
00:07:26.400 --> 00:07:42.960
So, we should not say something like, "Hey, the Interactions API is recommended for multi-chat, um, multi-chat because it handles like session state and it's like where you should be way more directive like use the Interactions API if you're working on like a chat application.

s86
00:07:42.960 --> 00:07:50.400
So, you need to give the model like clear instructions and directives on when it should use the skill and how it should use the skill.

s87
00:07:50.400 --> 00:07:57.480
And similar to what we have seen in the skills bench results, we should keep the skill lean and layer information.

s88
00:07:57.480 --> 00:08:02.000
So, the description is the cost you always pay on every model invocation.

s89
00:08:02.000 --> 00:08:05.920
So, on every model call, the description is part of the model context.

s90
00:08:05.920 --> 00:08:12.960
So, you always pay that 100 200 tokens cost and you don't want to have a super long description because then you always have to pay that.

s91
00:08:12.960 --> 00:08:20.680
When you have a very long skill MD file, it will be always read into context when the model decides to read the the skill or to use the skill.

s92
00:08:20.680 --> 00:08:22.400
Uh which can be expensive as well.

s93
00:08:22.400 --> 00:08:29.280
That's why we want to keep it as concise as possible, but still include all of the reference and details for the model to solve the task.

s94
00:08:29.280 --> 00:08:37.159
And then of course, the layer three is like we can have those reference files where the model needs to like um go really deep into a very specific task.

s95
00:08:37.159 --> 00:08:48.600
And a good example for this is like if you are working in like maybe a multi-cloud environment and you have a skill to deploy your application, you might need instruction to deploying to AWS and deploying to Google Cloud.

s96
00:08:48.600 --> 00:08:50.800
Those should not be part of your skill MD file.

s97
00:08:50.800 --> 00:08:52.120
Those should be references.

s98
00:08:52.120 --> 00:09:03.240
That you have a reference for AWS, reference for Google Cloud, maybe a reference for Azure so that the model can basically explore based on the context where it should go to get all of that information.

s99
00:09:03.240 --> 00:09:05.720
Then we should set the right level of freedom.

s100
00:09:05.720 --> 00:09:11.480
Um I see many people very clearly describing the exact workflow in a skill.

s101
00:09:11.480 --> 00:09:12.720
Step one, go there.

s102
00:09:12.720 --> 00:09:13.680
Step two, do this.

s103
00:09:13.680 --> 00:09:14.720
Step three, do this.

s104
00:09:14.720 --> 00:09:17.680
If you have those type of use cases, you should not use skills.

s105
00:09:17.680 --> 00:09:27.360
You maybe you should write a script because if the the process or the workflow is always the same, you don't need to waste models and tokens for that exercise.

s106
00:09:27.360 --> 00:09:28.320
You can create a script.

s107
00:09:28.320 --> 00:09:31.440
You can tell the model use that script to run a specific workflow.

s108
00:09:31.440 --> 00:09:34.240
So, rather define goals and constraints.

s109
00:09:34.240 --> 00:09:42.080
So, if you need to like deploy to your update or stage your documentation, describe how the model can do that.

s110
00:09:42.080 --> 00:09:49.440
Or like for your database updating a config, you should not say like read the config, update the port, and then like deploy again.

s111
00:09:49.440 --> 00:09:50.720
The model knows what to do.

s112
00:09:50.720 --> 00:09:55.080
Just like hey, if we need to change the config, here's the file, make the change.

s113
00:09:55.080 --> 00:09:57.160
Then uh don't skip negative cases.

s114
00:09:57.160 --> 00:10:04.160
So, we always look at the when we want to use this skill, but most of the time we don't look at when we don't want to use this skill.

s115
00:10:04.160 --> 00:10:12.480
So, if we have a description for our skill which says use it for web development tasks, it might over trigger.

s116
00:10:12.480 --> 00:10:27.320
Maybe you work with React, maybe you also work with Angular, and the model always loads the skill if you are working in like a web development environment, but if you are very specific for like hey, only use that skill for React components or for Tailwind CSS,

s117
00:10:27.320 --> 00:10:31.280
then the model knows hey, that's very specific for one to use.

s118
00:10:31.280 --> 00:10:34.320
And with Evals, we can also identify those.

s119
00:10:34.320 --> 00:10:36.320
Um And then test early.

s120
00:10:36.320 --> 00:10:37.960
So, that's what we are going to look at.

s121
00:10:37.960 --> 00:10:40.520
We should really try to test when you create a new skill.

s122
00:10:40.520 --> 00:10:43.520
Always try to create 10 of 20 prompts.

s123
00:10:43.520 --> 00:10:46.040
I like to create five for like the happy path.

s124
00:10:46.040 --> 00:10:48.600
So, when do I want to use that skill?

s125
00:10:48.600 --> 00:10:54.960
Five when I don't want to use that skill just to make sure the model is not over triggering the skill and confusing itself.

s126
00:10:54.960 --> 00:11:03.240
And then if you have already some customer or production traces, try to include those as well because nothing is better than than real-world data.

s127
00:11:03.240 --> 00:11:07.480
And then tip seven which is quite new and I have to give all credits to Matt.

s128
00:11:07.480 --> 00:11:11.160
So, if you don't know Matt, he's a great AI educator and you should definitely follow him.

s129
00:11:11.160 --> 00:11:15.320
He published a tweet and also a skill on like killing all of the no-ops.

s130
00:11:15.320 --> 00:11:20.920
And what he found is that AI generated skills tend to include a lot of no-ops.

s131
00:11:20.920 --> 00:11:26.400
And no-ops basically is an instruction which does nothing to change the agent's behavior.

s132
00:11:26.400 --> 00:11:29.480
It's like before making an implementation easy to read.

s133
00:11:29.480 --> 00:11:36.200
Like the model knows how it when it should make something easy to read or write clear high-quality code.

s134
00:11:36.200 --> 00:11:39.240
I mean, like that's what we expect from the model to do without telling it really.

s135
00:11:39.240 --> 00:11:41.560
So, um definitely look at those no-ops.

s136
00:11:41.560 --> 00:11:46.040
He have has published a very good skill in in his like skills repository.

s137
00:11:46.040 --> 00:11:51.280
Uh and then last but not least, uh know when you should retire skill.

s138
00:11:51.280 --> 00:11:54.840
Um skills are not there to live forever.

s139
00:11:54.840 --> 00:12:00.600
Models get better, behaviors change, expectation change, um the environment changes.

s140
00:12:00.600 --> 00:12:05.600
So, um always try to run evals with and without the skill enabled.

s141
00:12:05.600 --> 00:12:15.480
And if the model achieves the performance without even like triggering the skill, you know you can retire that skill, save the cost uh for your tokens, and then also

s142
00:12:15.480 --> 00:12:18.400
um don't keep like it redundant.

s143
00:12:18.400 --> 00:12:21.440
So, save cost at the end and maintenance also as well.

s144
00:12:21.440 --> 00:12:29.800
And to look at a little bit of a practical example and also how you can create your own small eval or eval harness for skills.

s145
00:12:29.800 --> 00:12:34.960
Um earlier this year we wanted to create a new skill for the Gemini Interactions API.

s146
00:12:34.960 --> 00:12:40.360
So, the Gemini Interactions API is our new interface for working with Gemini models and with agents.

s147
00:12:40.360 --> 00:12:45.320
And the Interactions API was released after the last training of Gemini.

s148
00:12:45.320 --> 00:12:53.960
So, the model or Gemini 3 and like 3.1 or even 3.5 has no context about what is the the Gemini Interactions API.

s149
00:12:53.960 --> 00:13:03.600
So, we decided, "Okay, let's look uh at creating a skill to help the model create good code for the Interactions API, to use the latest models."

s150
00:13:03.600 --> 00:13:07.000
And to do that, we created 117 test cases.

s151
00:13:07.000 --> 00:13:21.880
Those are uh based on like data we see uh from real users trying to generate uh Gemini code from um synthetic generated uh test cases, and also from like feedback we see people like, "Hey, the model is like using Gemini 2.0

s152
00:13:21.880 --> 00:13:36.560
even if we are already on 3.0." And the the end result was that we improved the the performance up to like almost 90% for generating valid interactions API code with the latest Gemini models.

s153
00:13:36.560 --> 00:13:43.360
And to do this, we basically only needed like two very simple uh assets.

s154
00:13:43.360 --> 00:13:49.360
So, one of that was a JSON file with all of our test cases.

s155
00:13:49.360 --> 00:13:51.880
And it's very like no clear structure.

s156
00:13:51.880 --> 00:13:53.280
It's like, "Hey, we have a prompt."

s157
00:13:53.280 --> 00:13:55.480
That's basically what we expect the user to provide.

s158
00:13:55.480 --> 00:14:00.720
We have a language because we wanted to test the skill against TypeScript and Python.

s159
00:14:00.720 --> 00:14:01.680
Uh we have a should trigger.

s160
00:14:01.680 --> 00:14:07.000
That's basically there to tell us if the agent should read the skill or not read the skill.

s161
00:14:07.000 --> 00:14:09.920
And then we have different expected checks.

s162
00:14:09.920 --> 00:14:11.480
Uh we look at them in a little bit.

s163
00:14:11.480 --> 00:14:17.480
Those are basically uh very simple asserts uh for that prompt if it should trigger or not.

s164
00:14:17.480 --> 00:14:20.680
And then we have a very basic Python script which runs a coding agent.

s165
00:14:20.680 --> 00:14:31.320
In this case, it was the Gemini CLI which uh passes the output and returns runs it so we can like take a look at the the outcome whether we have valid code for the interactions API

s166
00:14:31.320 --> 00:14:32.360
or not.

s167
00:14:32.360 --> 00:14:39.480
And uh most of the tests or evals for skills can be regex.

s168
00:14:39.480 --> 00:14:45.120
It's like very amazing how good of regex you can write using coding agents.

s169
00:14:45.120 --> 00:14:49.240
And it's really for us it was all about, "Okay, do we use the correct SDK?

s170
00:14:49.240 --> 00:14:51.400
Do we use the correct um model?

s171
00:14:51.400 --> 00:14:53.920
Do we use the correct methods?

s172
00:14:53.920 --> 00:14:55.680
Do we use any old patterns?"

s173
00:14:55.680 --> 00:15:01.600
And we created um very basic asserts for all of those cases, which are very cheap to run.

s174
00:15:01.600 --> 00:15:04.640
So, we can run our skill against the evals many times.

s175
00:15:04.640 --> 00:15:10.280
So, if a new model releases, we have a very easy way to update those asserts to the latest model IDs.

s176
00:15:10.280 --> 00:15:15.280
And it's very cheap to run for it well because we don't need to use like LLM as a judge.

s177
00:15:15.280 --> 00:15:23.760
But of course, you can use LLM as a judge if you have like more complex skills which need to look at the whole traces or the whole steps taken.

s178
00:15:23.760 --> 00:15:37.600
And a very easy case is like you just create LLM as a judge with a rubric on like what you want to look at, and then like take the output, put it through the LLM as a judge, try to get a pass or a fail, and then if it fails, look at the data,

s179
00:15:37.600 --> 00:15:41.840
and then like try to um improve your skill based on that.

s180
00:15:41.840 --> 00:15:45.440
And that's also how we now uh eval skills at Google DeepMind.

s181
00:15:45.440 --> 00:15:51.440
So, the we don't use YAML, but it's like just as like uh an example.

s182
00:15:51.440 --> 00:15:59.040
Uh we have um tests or evals alongside every skill we have internally at Google DeepMind.

s183
00:15:59.040 --> 00:16:02.440
Um every test has multiple cases with like a prompt.

s184
00:16:02.440 --> 00:16:12.520
Uh we all run them in like clear workspaces, so you can define your workspace or environment if it should include additional files like your application environments.

s185
00:16:12.520 --> 00:16:19.280
You have uh startup commands, which basically preloads or installs libraries into the environment.

s186
00:16:19.280 --> 00:16:21.120
And then you have script evals or data.

s187
00:16:21.120 --> 00:16:29.000
Those are those regex where we look at all of the traces to see what the skill triggered, was a certain command run, was a certain CLI run.

s188
00:16:29.000 --> 00:16:39.480
And then we also have LLM as a judge where we have some expectations, which are basically matched against like hey, did it trigger the skill, did it run a certain bash command to like

s189
00:16:39.480 --> 00:16:40.880
also evaluate it.

s190
00:16:40.880 --> 00:16:43.920
And we run them on every change to the skill.

s191
00:16:43.920 --> 00:16:56.040
So, if a change happens to or like a diff to the skill file, the eval will be run, and there will also be a result, and the change will not be merged if it is not improving the test cases.

s192
00:16:56.040 --> 00:17:05.760
So, we always have those regression tests for every change to the skill, and you can only change the skill if it improves the eval or add new evals.

s193
00:17:05.760 --> 00:17:11.079
And um yeah, that that's how we we we basically manage it.

s194
00:17:11.079 --> 00:17:16.800
And then last but not least, uh 10 examples for best practices for skills.

s195
00:17:16.800 --> 00:17:17.880
You don't need to take photos.

s196
00:17:17.880 --> 00:17:20.120
They are in the blog post I can share later.

s197
00:17:20.120 --> 00:17:24.360
So, um the I mean, we had it many, many times.

s198
00:17:24.360 --> 00:17:26.680
The the skill skill description is very important.

s199
00:17:26.680 --> 00:17:38.240
Uh we have seen 50% of the failures uh because the skill was not triggered correctly because the prompt of the user was not uh detailed enough for the model to understand, "Hey, I need to use that skill to solve that task."

s200
00:17:38.240 --> 00:17:46.440
And especially if you build agents for others, they are not aware of the skill descriptions you have for your model and for your skill.

s201
00:17:46.440 --> 00:17:54.760
So, they might write something very um shallow and then the model needs to know, "Okay, I need to trigger that skill."

s202
00:17:54.760 --> 00:17:57.400
Um we should write directors over passive information.

s203
00:17:57.400 --> 00:17:59.400
So, we should always think about it.

s204
00:17:59.400 --> 00:18:06.080
You should tell the agent what to do or not what to do and not just like, "Hey, if you feel happy today, please use the skill."

s205
00:18:06.080 --> 00:18:08.040
Um include negative tests.

s206
00:18:08.040 --> 00:18:10.200
Uh we always forget negative tests.

s207
00:18:10.200 --> 00:18:11.240
Start small.

s208
00:18:11.240 --> 00:18:15.680
Even like 10 to 20 skill eval samples are better than nothing.

s209
00:18:15.680 --> 00:18:20.880
You will be surprised on how much you will find even from like five to 10 examples.

s210
00:18:20.880 --> 00:18:23.200
And then like definitely create outcomes, not paths.

s211
00:18:23.200 --> 00:18:27.040
Uh we don't want to test if the model loads the skill on like the first turn.

s212
00:18:27.040 --> 00:18:31.440
We really want to test if it can achieve the task based on the prompt.

s213
00:18:31.440 --> 00:18:33.320
And if it loads the skill, it loads the skill.

s214
00:18:33.320 --> 00:18:34.560
If not, then not.

s215
00:18:34.560 --> 00:18:38.160
If it um loads the skill after five turns, that's also okay.

s216
00:18:38.160 --> 00:18:45.840
Then we want to have isolated runs because coding agents are very good at finding or cheating.

s217
00:18:45.840 --> 00:18:59.240
So, if you run inside uh your existing environment, it might look up previous chats or it might look up some other executions and then like try to cheat it and get the context from the skill without even using the skill.

s218
00:18:59.240 --> 00:19:02.880
Then definitely run more than one trial when running evals.

s219
00:19:02.880 --> 00:19:06.200
Like agents, our models are non-deterministic.

s220
00:19:06.200 --> 00:19:08.840
Maybe the first one works, the second one doesn't.

s221
00:19:08.840 --> 00:19:19.000
So, always run the to six uh trials per case and to measure reliability Uh um test across different harnesses if you work with like or if you have employees

s222
00:19:19.000 --> 00:19:25.000
or um people working with like different harnesses, not only just evaluate against Claude or anti-gravity.

s223
00:19:25.000 --> 00:19:32.840
If you have people working with cursor, try to include them as well because agent harnesses behave differently and of course model behaves differently.

s224
00:19:32.840 --> 00:19:41.840
So, maybe your skill is very good with a Gemini but very bad with Codex and then you have uh customers, consumers using your harness with Codex and then it fails.

s225
00:19:41.840 --> 00:19:43.720
And then um create your evals.

s226
00:19:43.720 --> 00:19:50.400
So, if your um model is good enough that it doesn't need the skill anymore, keep that eval.

s227
00:19:50.400 --> 00:19:53.040
You don't need to throw that eval away because you throw the skill away.

s228
00:19:53.040 --> 00:20:01.160
You can keep that eval to make sure that the model or the agent keeps the performance and as soon as you start seeing some degradation, you can reintroduce the skill.

s229
00:20:01.160 --> 00:20:12.400
You can maybe tweak some other tools or pieces to keep like the the performance up and then really detect when you can retire skill and you will be very surprised with all of the model updates

s230
00:20:12.400 --> 00:20:18.120
how fast you can retire skill which you might need it like six months ago but not today anymore.

s231
00:20:18.120 --> 00:20:20.480
And I have some homework for you.

s232
00:20:20.480 --> 00:20:28.520
So, if you are back from holiday on Monday, um pick uh the most used skill and write five test prompts.

s233
00:20:28.520 --> 00:20:36.960
Uh you can also use your coding agent and ask it to see look at your trajectories which are my most used skills and then try to create some some skills.

s234
00:20:36.960 --> 00:20:40.280
It's you have seen it's like very easy to write your eval harness.

s235
00:20:40.280 --> 00:20:48.000
It's like a JSON or YAML file and then like some Python script which runs your coding agent or your agent harness and then like look at the outcome.

s236
00:20:48.000 --> 00:20:51.400
Definitely uh try to look at the removing no-ops.

s237
00:20:51.400 --> 00:21:03.240
Maybe it does not change the eval performance but it helps you save cost because all of the tokens which are not helpful or not changing the agent behavior are money you will like spend.

s238
00:21:03.240 --> 00:21:07.240
So, look at um writing great skills uh from Matt.

s239
00:21:07.240 --> 00:21:11.880
It's you can find it on on GitHub and then also run ablation test.

s240
00:21:11.880 --> 00:21:16.240
So, run always evals with your skill loaded and without your skill loaded.

s241
00:21:16.240 --> 00:21:21.920
Only that way you will know when you can retire skill or if a skill is really helpful for your performance.

s242
00:21:21.920 --> 00:21:25.080
So, don't ship skills without evals.

s243
00:21:25.080 --> 00:21:25.779
Thank you.

s244
00:21:25.779 --> 00:21:27.779
[applause]