WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=SQMCtZX3trg

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/SQMCtZX3trg.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.040 --> 00:00:15.440
Hello everybody.

s2
00:00:15.839 --> 00:00:18.240
How's it going?

s3
00:00:19.039 --> 00:00:25.920
Okay, I think we still have some people trickling in, but I'm super excited to be here.

s4
00:00:28.640 --> 00:00:37.760
Okay, first off, let me reload these slides because my claw code was still building something on it.

s5
00:00:39.360 --> 00:00:40.480
Um, okay.

s6
00:00:40.480 --> 00:00:42.719
So, the contrast is a little bit low.

s7
00:00:42.719 --> 00:00:44.079
Um, so please bear with me.

s8
00:00:44.079 --> 00:00:49.120
I'm going to try to cover uh what you cannot read as much as possible.

s9
00:00:49.120 --> 00:00:51.680
Uh, I also have heard that the Wi-Fi is not the strongest.

s10
00:00:51.680 --> 00:01:01.440
So um while it is a workshop uh hopefully you just do take away a lot of the lessons um and then can apply it whenever you want to.

s11
00:01:01.440 --> 00:01:04.799
Um but I do have a sample repo if you want to follow along.

s12
00:01:04.799 --> 00:01:08.640
Um okay first of all hi my name is Paul.

s13
00:01:08.640 --> 00:01:11.280
Um I'm really glad you found your way into this room.

s14
00:01:11.280 --> 00:01:14.880
Um I'm the author of Impeccable.

s15
00:01:14.880 --> 00:01:17.119
Who here has used Impeccable by any chance?

s16
00:01:17.119 --> 00:01:18.799
Can I see some hands?

s17
00:01:18.799 --> 00:01:20.400
Okay, a few people.

s18
00:01:20.400 --> 00:01:21.200
Nice.

s19
00:01:21.200 --> 00:01:29.200
Um, so for those of you who have not used Impeccable, Impeccable is a skill that I've built for myself mainly.

s20
00:01:29.200 --> 00:01:36.320
Um, I've built this large enterprise app uh over the last year and it has like lots of different views, states, whatever.

s21
00:01:36.320 --> 00:01:42.479
Um, and I wanted to design really quickly with my agents with codecs with claw.

s22
00:01:42.479 --> 00:01:50.720
But I noticed that uh even though it gets me quickly to something that I can look at, normalizing it back to the design system was really really challenging.

s23
00:01:50.720 --> 00:01:59.280
So the first skill that I've built for myself was called normalize and it kind of like brought it brought whatever claw designed back to the design system.

s24
00:01:59.280 --> 00:02:10.720
That's how I started and I also used Anthropics funed design skill like maybe many of you um when I first got going and from there on it kind of expanded into more and more design skills

s25
00:02:10.720 --> 00:02:17.200
that allowed me to turn clawed code and then codecs and other harnesses more into a design harness.

s26
00:02:17.200 --> 00:02:23.520
Um at some point I decided maybe other people might find uh um might find this useful as well.

s27
00:02:23.520 --> 00:02:30.000
So I open sourced it as open source skill uh released it and it turns out a lot of you liked it.

s28
00:02:30.000 --> 00:02:34.319
Um so if you haven't checked it out yet um uh yeah give it a go.

s29
00:02:34.319 --> 00:02:36.480
Uh it's on impeccable.style.

s30
00:02:36.480 --> 00:02:40.400
Um but today won't be a talk about impeccable per se.

s31
00:02:40.400 --> 00:02:45.120
It will be about what I learned from making these skills because it kind of escalated.

s32
00:02:45.120 --> 00:02:49.599
It started with like a simple prompt and then uh went all the way to what it is today.

s33
00:02:49.599 --> 00:02:57.599
Um, a lot of people have looked at the code of Impeccable and they see like a whole bunch of scripts in the scripts folder and they're like, "What is all this stuff?"

s34
00:02:57.599 --> 00:03:02.319
And so, um, I wanted to share some of my knowledge that I've gained with you.

s35
00:03:02.319 --> 00:03:04.480
So, let's get into it.

s36
00:03:04.480 --> 00:03:08.720
Uh, let's talk about the dark arts of skill engineering.

s37
00:03:13.882 --> 00:03:15.882
[panting]

s38
00:03:16.319 --> 00:03:18.000
Okay.

s39
00:03:18.000 --> 00:03:23.040
Um yeah, the production value of this talk is out of this roof.

s40
00:03:23.040 --> 00:03:26.720
Um okay, so first of all, you've all seen this kind of design.

s41
00:03:26.720 --> 00:03:33.680
Um this is actually a real design uh built with uh the front end design skill uh and clawed code.

s42
00:03:33.680 --> 00:03:36.720
Um and you probably all have seen a design like this.

s43
00:03:36.720 --> 00:03:40.000
This is for like a fake kids reader, iPad reader app.

s44
00:03:40.000 --> 00:03:42.319
Um you have italic serif.

s45
00:03:42.319 --> 00:03:47.200
Uh you have like some capitalized uh hero.

s46
00:03:47.200 --> 00:03:51.599
Um, and you have like a eyebrow text, like a kicker, whatever you want to call it.

s47
00:03:51.599 --> 00:03:53.440
The top of it, like a weird label.

s48
00:03:53.440 --> 00:03:58.959
Um, you have beige, I call it claw beige, claw beige, uh, backgrounds.

s49
00:03:58.959 --> 00:04:04.239
Um, and now it's not necessarily a bad design, right?

s50
00:04:04.239 --> 00:04:08.000
But I think you all can point this out and say like, well, this is clearly AI generated.

s51
00:04:08.000 --> 00:04:09.760
It's clearly slop.

s52
00:04:09.760 --> 00:04:12.239
Um, and it turns out slop is a moving target.

s53
00:04:12.239 --> 00:04:21.919
Um you've all seen you might have thinking might have been thinking of slop as purple gradients but uh we've kind of moved on since that uh into cloud beige.

s54
00:04:21.919 --> 00:04:25.520
Um okay so this is where I started.

s55
00:04:25.520 --> 00:04:28.639
I started with a system prompt and a prayer basically.

s56
00:04:28.639 --> 00:04:36.960
Um, so I I started using the front end design skill uh like many of you and it's like 55 lines of named bands.

s57
00:04:36.960 --> 00:04:39.759
Uh, no scripts, no routing, pure pros, right?

s58
00:04:39.759 --> 00:04:42.160
Uh, and you just hope for the best.

s59
00:04:42.160 --> 00:04:46.639
Um, that sometimes worked and sometime and most of the time it didn't.

s60
00:04:46.639 --> 00:04:50.560
So, um, for example, I'm just going to read some of this.

s61
00:04:50.560 --> 00:04:53.759
So, um, if I mean maybe it's really readable, I don't know.

s62
00:04:53.759 --> 00:05:08.320
Um but for example in the font and design skill you have sentences like never use generic AI aesthetics overused fonts like inter robboto arial system fonts um or cliched color schemes particularly purple gradients and white backgrounds

s63
00:05:08.320 --> 00:05:24.720
uh never converge on common choices like space grotesque for example um now there are two problems with this approach the first one it over applies and then a ban just relocates the model to the next cluster and I'll show you why this is a problem a a little bit further down the road.

s64
00:05:24.720 --> 00:05:30.400
But really, you tell it not to use in, it just uses the next best font it finds in its latent space.

s65
00:05:30.400 --> 00:05:33.120
And so it doesn't actually make it more creative.

s66
00:05:33.120 --> 00:05:35.680
It just, again, this is why I said slop is a moving target.

s67
00:05:35.680 --> 00:05:37.520
It kind of picks the next best thing.

s68
00:05:37.520 --> 00:05:49.680
Um, I have uh learned my lesson here um the hard way because uh I don't know if you noticed, but the reason why we got purple gradients in the first place is because of Tailwind.

s69
00:05:49.680 --> 00:05:53.199
Taywin's default sample pages theme whatever was purple.

s70
00:05:53.199 --> 00:05:57.440
Um well it turns out I've turned the web orange before many years before that.

s71
00:05:57.440 --> 00:06:04.160
I created a framework called jQuery UI and the first default theme of jQuery UI was orange.

s72
00:06:04.160 --> 00:06:06.000
So overnight I call it the web orange.

s73
00:06:06.000 --> 00:06:08.639
I thought people would modify the theme but now they didn't.

s74
00:06:08.639 --> 00:06:12.000
Um so I learned my lesson.

s75
00:06:12.000 --> 00:06:16.000
Um okay the median is the model's gravity.

s76
00:06:16.000 --> 00:06:22.960
Even 250 lines of like artisal, crafted, beautiful skill pros cannot change this.

s77
00:06:22.960 --> 00:06:24.160
It's just not enough.

s78
00:06:24.160 --> 00:06:25.680
It doesn't help enough, right?

s79
00:06:25.680 --> 00:06:27.520
It's nowhere near enough.

s80
00:06:27.520 --> 00:06:31.039
Um, and I learned this the hard way and hopefully you don't have to.

s81
00:06:31.039 --> 00:06:42.639
Um, my overall thesis for this talk is that prompting is sort of like the starter level, but harness engineering is where you should end up.

s82
00:06:42.639 --> 00:06:45.120
Like you should reframe when you're building skills.

s83
00:06:45.120 --> 00:06:52.639
You should think about okay skills the same way as MCP is an extension to the coding harness or whatever harness you're in.

s84
00:06:52.639 --> 00:06:59.680
Uh it's not just a prompt that you package it's something more than that or you should at least conceptually think about it more than that.

s85
00:06:59.680 --> 00:07:03.280
It is extending the harness of whoever is using that thing.

s86
00:07:03.280 --> 00:07:06.400
Uh and it also has more capabilities as just prompting.

s87
00:07:06.400 --> 00:07:10.240
Uh and when I thought about that way it sort of like clicked for me.

s88
00:07:10.240 --> 00:07:23.440
uh prompting is a spell harnessing the magic and we'll talk about nine different dark arts today that I learned uh in the process of building impeccable um we'll make

s89
00:07:23.440 --> 00:07:36.000
uh sub agents argue with each other uh we'll talk about how to force divergence as opposed to convergence routing like a model um and basic you've seen this probably um most modern

s90
00:07:36.000 --> 00:07:43.360
models are mixture of expert architectures and impeccable is built like a mixture of expert skill.

s91
00:07:43.360 --> 00:07:45.680
Um, we we give them memory.

s92
00:07:45.680 --> 00:07:49.840
We'll create scripts that talk back and I I promise this will make sense.

s93
00:07:49.840 --> 00:07:54.080
Uh, hooks that fight back live wire the browser and use more of the harness.

s94
00:07:54.080 --> 00:07:57.840
Compile to every harness and design for the weakest model.

s95
00:07:57.840 --> 00:07:59.120
Let's get into it.

s96
00:07:59.120 --> 00:08:02.400
Number one, made it make it argue.

s97
00:08:02.400 --> 00:08:09.599
So if you're building something like a critique skill or code review skill, um here's one huge issue.

s98
00:08:09.599 --> 00:08:22.560
Uh if you are I mean you probably noticed this when you're working with claw code or codeex doesn't matter, but if you ask codeex or claw code to review its own work, it will usually rate it as very high.

s99
00:08:22.560 --> 00:08:26.319
It's like I mean like I've built this, I've done a great job, right?

s100
00:08:26.319 --> 00:08:28.479
Like it's like grading your own homework.

s101
00:08:28.479 --> 00:08:30.240
Doesn't make any sense, right?

s102
00:08:30.240 --> 00:08:33.599
um it anchors on what it already created.

s103
00:08:33.599 --> 00:08:36.640
Um now that's not great.

s104
00:08:36.640 --> 00:08:39.760
Um what can you do in order to solve this?

s105
00:08:39.760 --> 00:08:43.440
Well, you can make a model argue with another model, right?

s106
00:08:43.440 --> 00:08:45.600
Adversarial prompting is also called.

s107
00:08:45.600 --> 00:08:50.320
So you have two sub agents and they never see each other's work.

s108
00:08:50.959 --> 00:08:52.320
And here's why this matters.

s109
00:08:52.320 --> 00:08:56.880
So in Impeccable, there's a critique command that actually critiques your design.

s110
00:08:56.880 --> 00:08:58.720
And you can point it to your landing page.

s111
00:08:58.720 --> 00:09:00.800
You can point it to anything.

s112
00:09:00.800 --> 00:09:03.839
And there are two particular failure scenarios.

s113
00:09:03.839 --> 00:09:07.360
The first one, and this is almost impossible to read, so I'll explain it.

s114
00:09:07.360 --> 00:09:11.440
The first one is um a strong page, right?

s115
00:09:11.440 --> 00:09:17.920
So, it's a really good-look page, but there's a whole bunch of maybe like uh deterministic errors.

s116
00:09:17.920 --> 00:09:28.320
And impeccable actually has a deterministic engine like a design llin llin llin llin llin llin llin llin llin llinter that can detect things like bad contrast can detect things like

s117
00:09:28.320 --> 00:09:38.080
you know too many fonts um you know maybe things that are too close to the edge of an element so it detects some of I would say polish issues

s118
00:09:38.080 --> 00:09:49.279
um but you could have this really beautiful website and then the detector runs and it's doing that as part of the same skill and the same model thread and then the model

s119
00:09:49.279 --> 00:09:56.080
uh just sees the detector output and say like well I guess there's 500 issues therefore this design must be bad.

s120
00:09:56.080 --> 00:09:56.959
Now that's one.

s121
00:09:56.959 --> 00:09:58.240
The other one is the opposite.

s122
00:09:58.240 --> 00:10:06.000
The other one is it's actually a really terrible page or maybe like an empty page but there are no detected issues by the by the deterministic detector.

s123
00:10:06.000 --> 00:10:10.880
So the model is like we didn't find any issues so this must be great design.

s124
00:10:10.880 --> 00:10:13.279
So both of those are not amazing.

s125
00:10:13.279 --> 00:10:21.040
Uh what you want is and this is what uh impeccable critique skill does it uh it kind of combines two things.

s126
00:10:21.040 --> 00:10:25.839
It spawns two sub agents and they are blind to each other.

s127
00:10:25.839 --> 00:10:28.240
Um and that's how you get to a balance critique.

s128
00:10:28.240 --> 00:10:35.040
So the first sub agent um does um um act like a design director.

s129
00:10:35.040 --> 00:10:37.440
So it's an L&M um that acts like a design director.

s130
00:10:37.440 --> 00:10:42.160
And so um it looks for hierarchy, it looks for slop, it looks for uristics.

s131
00:10:42.160 --> 00:10:48.320
And so it does a critique the way a human would with the browser tools that are you that are available to it.

s132
00:10:48.320 --> 00:10:54.480
The second sub agent runs determinist deterministic detector and also collects browser evidence.

s133
00:10:54.480 --> 00:11:06.399
And then once both of those results come in, the main thread synthesizes both into one into one critique and that produces a much more balanced result.

s134
00:11:06.399 --> 00:11:14.560
And before I go on, um, I realized I actually have not, uh, have not shown you where the sample repo of this lives.

s135
00:11:14.560 --> 00:11:17.680
So, let me bring this up real quick.

s136
00:11:17.680 --> 00:11:23.839
Um, if you want to clone this and you have a decent enough internet connection, go ahead if you like.

s137
00:11:23.839 --> 00:11:30.480
Um, so this is, um, uh, PBA Aus/impeccable miners talks.

s138
00:11:30.480 --> 00:11:40.560
Um the talk lives here but also in the dark arts folder there is a starter folder uh and a demos folder.

s139
00:11:40.560 --> 00:11:46.399
Demos uh has a pretty average median page that you can manipulate.

s140
00:11:46.399 --> 00:11:57.200
Um and then uh in the starter kit you have enough to build sort of a a mini impeccable if you want to if you want to follow along or or try it out yourself.

s141
00:11:57.200 --> 00:12:00.720
And so as part of this um there's a troy here too.

s142
00:12:00.720 --> 00:12:06.720
Uh you'll you'll follow along sort of the checkpoints the dark arts and build something yourself.

s143
00:12:06.720 --> 00:12:10.399
I would suggest you know if you like to multitask great.

s144
00:12:10.399 --> 00:12:12.720
Um you can apply this by the way to anything.

s145
00:12:12.720 --> 00:12:14.320
You can do like a code review thing.

s146
00:12:14.320 --> 00:12:17.360
You can do um it doesn't have to be the design skill.

s147
00:12:17.360 --> 00:12:21.120
Um but I wanted to have something for you to play.

s148
00:12:21.120 --> 00:12:23.920
Uh back to the deck.

s149
00:12:24.560 --> 00:12:28.560
So two blind opinions beat one confident guess.

s150
00:12:28.560 --> 00:12:34.959
Um you can use this again I already said code review design review but also security audits once is a good example.

s151
00:12:34.959 --> 00:12:47.839
Uh or a plan creating a really good plan RFC critique where you have multiple LLM judges argue with each other before it gets good or ranking outputs is a good good example.

s152
00:12:47.839 --> 00:12:50.480
Now, here's the problem, though.

s153
00:12:50.480 --> 00:12:54.639
Codeex, why why you don't let me do this?

s154
00:12:54.639 --> 00:12:55.839
It's bad.

s155
00:12:55.839 --> 00:13:01.440
Uh, it turns out a Codeex never created these sub agents when I first tried this.

s156
00:13:01.440 --> 00:13:03.519
Um, and I bang my head against the wall.

s157
00:13:03.519 --> 00:13:05.040
I'm like, why is this?

s158
00:13:05.040 --> 00:13:09.920
It turns out Codex has a different permission model than claude code and other harnesses.

s159
00:13:09.920 --> 00:13:19.760
In codeex, you have to explicitly as a user request the use of sub aents for anything in the harness to use sub aents.

s160
00:13:19.760 --> 00:13:23.200
So if you're distributing a skill, you're out of luck.

s161
00:13:23.200 --> 00:13:35.040
The only way to make this work as far as I know today is to actually tell the model, okay, if you have sub agents capabilities, but you do not have permission,

s162
00:13:35.040 --> 00:13:38.240
please stop right here and ask the user.

s163
00:13:38.240 --> 00:13:41.920
And so that's pretty much the only way you can get codecs to comply.

s164
00:13:41.920 --> 00:13:56.000
So in impeccable um if you see something that you know makes you go huh um it's probably because of that like you know through um lots and lots of issues that people filed and um a lot of testing on on my end also

s165
00:13:56.000 --> 00:14:01.600
um a lot of this obscure knowledge got into the skills so that it works truly across harnesses.

s166
00:14:01.600 --> 00:14:05.040
Um for instance here you see kind of pseudo code of how this would work.

s167
00:14:05.040 --> 00:14:09.440
And then also I uh this is another really important thing.

s168
00:14:09.440 --> 00:14:17.839
Uh very often if codeex realizes it can get away with something it will do it.

s169
00:14:17.839 --> 00:14:24.000
So um if there is no punishment for not spawning sub agents it will simply not spawn them.

s170
00:14:24.000 --> 00:14:25.760
It's like well this is the easier route.

s171
00:14:25.760 --> 00:14:27.760
I will take this easier route.

s172
00:14:27.760 --> 00:14:39.040
So what you have to say is actually if you cannot use sub agents you must say that you are giving the user a degraded experience and codeex hates that

s173
00:14:39.680 --> 00:14:42.880
so use that to your advantage.

s174
00:14:42.880 --> 00:14:47.040
Um so um you can watch them argue.

s175
00:14:47.040 --> 00:14:54.000
Um now I did not pre-record an actual example here because I'm like let's do it live.

s176
00:14:54.000 --> 00:15:10.000
Um, so we're going to go into cursor and um I'm going to do like critique and let's hopefully it'll work.

s177
00:15:10.000 --> 00:15:28.160
I don't know if composer sponsor sub agents uh well enough but let's see uh composer by the way if you haven't used it is a really fast well balanced model

s178
00:15:28.160 --> 00:15:43.120
so it's kind of uh kind of neat for for work that you want to show on stage in particular um okay so now it's doing something here Um, okay.

s179
00:15:43.120 --> 00:15:47.519
This this repository I think has an old version of packable doesn't sponsor agents.

s180
00:15:47.519 --> 00:15:49.680
I see this is the unfortunate.

s181
00:15:49.680 --> 00:15:52.320
Well, maybe it does.

s182
00:16:01.120 --> 00:16:07.120
I'm not sure if it did or not, but I at least want to show you what the type of critique looks like.

s183
00:16:20.000 --> 00:16:20.880
Uh, all right.

s184
00:16:20.880 --> 00:16:23.839
Now it's asking me a bunch of questions of what I actually want to create.

s185
00:16:23.839 --> 00:16:26.160
I'm going to skip this.

s186
00:16:26.160 --> 00:16:35.040
Uh, and now I get a design critique um on what's working, what the priorities issues are, persona red flags.

s187
00:16:35.040 --> 00:16:41.120
Now the actual thing that I wanted to show you unfortunately couldn't be seen in this particular thread but we can come back to it.

s188
00:16:41.120 --> 00:16:51.920
Um if you run this in in cloud code or codeex on the most recent version you should very clearly see I mean in cloud code it's very easy to see the sub aents running and doing its work.

s189
00:16:51.920 --> 00:16:59.680
So it will spin up two sub agents and you see it at the bottom of the cloud code thread uh doing its thing.

s190
00:16:59.839 --> 00:17:03.839
Okay level number two uh our Ben just moves the problem.

s191
00:17:03.839 --> 00:17:05.039
We talked about this already.

s192
00:17:05.039 --> 00:17:08.640
Um, you ban enter into the model graphs graph space grotesque.

s193
00:17:08.640 --> 00:17:09.679
How do you solve that?

s194
00:17:09.679 --> 00:17:11.760
How do you force divergence?

s195
00:17:11.760 --> 00:17:15.839
Well, a ban only moves the model around inside its own cluster.

s196
00:17:15.839 --> 00:17:23.199
And what I've built for impeccable and for a bunch of other skills that I've released is what I call an anti-attractor.

s197
00:17:23.199 --> 00:17:28.640
Uh, and the anti-attractor works by creating a random seed of sorts.

s198
00:17:28.640 --> 00:17:39.440
And that can come from user input or it can come from a script that it can run that produces something that is completely unexpected to the model because that's what you want.

s199
00:17:39.440 --> 00:17:50.880
And so in this case for instance it would be font selection and instead of selecting this save next step prediction font it went into a completely different space through a different seat.

s200
00:17:50.880 --> 00:17:57.760
Uh there are three techniques that you know are easy to hard and work differently.

s201
00:17:57.760 --> 00:18:02.640
The first one um is is something you can do right right now.

s202
00:18:02.640 --> 00:18:06.320
It's the most simple one and it's to shave the save pics.

s203
00:18:06.320 --> 00:18:16.640
So basically tell the model, okay, name your top three fonts and then the model is like okay I got the top three fonts and then you're like now throw them away.

s204
00:18:16.640 --> 00:18:18.320
And like but why?

s205
00:18:18.320 --> 00:18:25.280
Um but then you basically shave shaved off the uh the next token uh that is predicted.

s206
00:18:25.280 --> 00:18:34.080
Um and you do that three times and then now you get to a different space of the uh like a further away in the latent space right um now that's

s207
00:18:34.080 --> 00:18:44.240
doable but now at some point you still get convergence so this is just you know a limited technique the second technique is to generate a lot of different things

s208
00:18:44.240 --> 00:18:55.200
and then have a sub agent rank I've done this for a shader library that I've created called radiant shaders and um the goal here was to create around a hundred different shaders

s209
00:18:55.200 --> 00:19:04.000
And the problem is every time I I would say you know create a new shader or create ideas for 10 new shaders I would get the same repeating ideas.

s210
00:19:04.000 --> 00:19:06.160
I solved this with two in two different ways.

s211
00:19:06.160 --> 00:19:10.480
The first one is I created something unexpected a random seed a creative seed.

s212
00:19:10.480 --> 00:19:12.480
In this case I used celebrities.

s213
00:19:12.480 --> 00:19:20.000
I said like well what would Ryana look like as a shader um or what would Beyonce look like as a shader?

s214
00:19:20.000 --> 00:19:24.080
And um and then the model was like let me think about that.

s215
00:19:24.080 --> 00:19:33.600
So that's the first thing and then I said well generate a hundred of these ideas and then spawn a sub agent that ranks all of those ideas uh and that's important.

s216
00:19:33.600 --> 00:19:43.760
It has to be a sub agent because the sub agent doesn't know anything uh from the prior context of the session um and uh and can then completely change the order.

s217
00:19:43.760 --> 00:19:47.360
The third one is to create a random seat from a script.

s218
00:19:47.360 --> 00:20:02.320
So in impeccable for example when you first start a project it calls a script called color.js JS and color.js has um over a 100 hand selected uh they're not complete color pallets

s219
00:20:02.320 --> 00:20:13.840
but they are primary colors and they're kind of like a starting point of a palette and it reads that and then the model uses that as a creative spark to build a pallet around it for you.

s220
00:20:13.840 --> 00:20:15.760
You can still say you I don't like this.

s221
00:20:15.760 --> 00:20:20.720
I mean I don't like what it proposed but it it turns it into a different direction.

s222
00:20:20.720 --> 00:20:34.960
So those are all ways to create divergence and when you design something with impeccable the same brief depending on the users's input and you know color script that runs etc can produce

s223
00:20:34.960 --> 00:20:41.919
vastly different results because of that because I didn't want to have the whole internet look like uh everything else.

s224
00:20:41.919 --> 00:20:45.440
Number three um here's the problem.

s225
00:20:45.440 --> 00:20:50.080
If you cram every everything into one skill, it kind of blurs them.

s226
00:20:50.080 --> 00:20:53.200
The instruction following becomes not very good enough anymore.

s227
00:20:53.200 --> 00:21:02.720
So if you're building some general purpose skill and you expand it and expand it and expand it, at some point uh it becomes really really blurry to the model.

s228
00:21:02.720 --> 00:21:06.080
Um here's a concrete example of this.

s229
00:21:06.080 --> 00:21:08.720
the anthopic front end design skill.

s230
00:21:08.720 --> 00:21:19.360
Uh the former version of it, they just shipped a new version three weeks ago, but the former version had a line that I read earlier that says avoid system fonts.

s231
00:21:19.360 --> 00:21:21.919
That's okay for landing page design.

s232
00:21:21.919 --> 00:21:25.520
But for product UI, oftent times you want it to feel as native as possible.

s233
00:21:25.520 --> 00:21:28.159
So system fonts are actually the thing that you want.

s234
00:21:28.159 --> 00:21:29.440
So how do you solve this?

s235
00:21:29.440 --> 00:21:33.600
Um you can say I mean you have like this giant if else block in a skill.

s236
00:21:33.600 --> 00:21:36.320
say like, well, if the user wants a landing page, do this.

s237
00:21:36.320 --> 00:21:38.559
If the user wants a product, do this.

s238
00:21:38.559 --> 00:21:44.400
But that becomes really convoluted, wastes a lot of tokens, and uh honestly doesn't work very well.

s239
00:21:44.400 --> 00:21:56.080
So, impeccable started as a lot of different subs skills and now has this um mixture of experts model that routes internally um both in terms of uh capabilities.

s240
00:21:56.080 --> 00:22:04.400
So you can call impeccable critique or impeccable polish and you get a different MD file loaded behind the scenes for that particular job.

s241
00:22:04.400 --> 00:22:06.960
So it's not just one giant skill MD.

s242
00:22:06.960 --> 00:22:20.720
Um, but also, and this is something not a lot of people know, behind the scenes, Impeccable decides based on your brief and what you input whether you're trying to design something brandandy,

s243
00:22:20.720 --> 00:22:26.799
so like a landing page or something that wants to attract attention or whether it's the actual product that you're designing.

s244
00:22:26.799 --> 00:22:35.200
So, it switches registers and then loads completely different rules for those two registers because product design and brand design are very very different.

s245
00:22:35.200 --> 00:22:40.960
Um, so that's also something that I would recommend you doing if you're building a larger scope.

s246
00:22:40.960 --> 00:22:51.840
This works for bu big multi-tool skills, works for context on demand type of skills, uh, per audience behavior, agent toolkits, that kind of thing.

s247
00:22:51.840 --> 00:22:55.679
Number four, everyone starts from zero.

s248
00:22:55.679 --> 00:22:58.080
Skills by default don't have long-term memory.

s249
00:22:58.080 --> 00:23:02.159
Uh, they don't really compound over time, but you can make it so.

s250
00:23:02.159 --> 00:23:05.760
So um you have a skill folder and you can save things in that skill folder.

s251
00:23:05.760 --> 00:23:15.280
In fact in claude you even have an environment variable that uh resolves to the actual directory uh that you can save things in which is nice.

s252
00:23:15.280 --> 00:23:18.400
No other harness supports this right now I believe.

s253
00:23:18.400 --> 00:23:20.559
Um but you can you can hack around that.

s254
00:23:20.559 --> 00:23:25.120
Impeccable uses a impeccable folder in the current repository route.

s255
00:23:25.120 --> 00:23:28.559
Um, but you can also save things directly in the skill folder.

s256
00:23:28.559 --> 00:23:32.799
Um, and maybe ask the user to ignore them.

s257
00:23:32.799 --> 00:23:33.840
Uh, how could this work?

s258
00:23:33.840 --> 00:23:43.840
So, for example, if you're running a critique in impeccable, that critique is saved as a file in that folder and by default it's get ignored.

s259
00:23:43.840 --> 00:23:48.720
But then if you then later on say, well, okay, I just ran a critique.

s260
00:23:48.720 --> 00:23:50.240
I'd like to polish my page.

s261
00:23:50.240 --> 00:24:02.320
Even if you do it in another session, it actually uses that prior critique as a signal to understand what have we found out about this page and it can look at all prior critiques and see sort of the progression of the page.

s262
00:24:02.320 --> 00:24:07.280
Uh for example, you could have said in one of the critiques, you know, I don't agree with this critique.

s263
00:24:07.280 --> 00:24:12.559
I don't think you're right and I think I really like my instrument fonts.

s264
00:24:12.559 --> 00:24:14.880
Um and then the model would be like, okay, no problem.

s265
00:24:14.880 --> 00:24:16.799
I'm gonna mark this for later.

s266
00:24:16.799 --> 00:24:18.960
And the skill is now smart enough.

s267
00:24:18.960 --> 00:24:37.760
the skill has uh built context to realize okay well that's the user preference so I'm going to respect it going forward um so compound engineering um really uh is an interesting theme for skills as well you can make skills aware of prior sessions with that technique

s268
00:24:37.760 --> 00:24:53.120
um so make the runs compound this works really well for resumable gradients uh resumable agents progress tracking u multi session refactors migrations that kind of thing Um uh for instance, one of the things that I do all the time is refactor my code.

s269
00:24:53.120 --> 00:24:55.120
And um how do I do that?

s270
00:24:55.120 --> 00:25:05.120
By having a sort of like a skill that spawns itself across multiple sessions and tackles one file at a time.

s271
00:25:05.120 --> 00:25:11.600
So I basically tell it, okay, here's your TSX file or whatever um for today's session.

s272
00:25:11.600 --> 00:25:21.919
um and now refactor everything around this file and linking into that file and and then it sort of builds up context over time until it's completely finished with the whole codebase.

s273
00:25:22.320 --> 00:25:24.559
Okay, number five.

s274
00:25:24.559 --> 00:25:27.760
Uh buried rules get skimmed.

s275
00:25:27.760 --> 00:25:33.200
We talked a bit about this before, but this is a little bit of a different point I'm trying to make.

s276
00:25:33.200 --> 00:25:47.200
Um now especially with weaker models and now if you're building a skill for yourself and you're only running opus or you're only running codecs u this isn't that that big of an issue right you you know which model you run

s277
00:25:47.200 --> 00:25:58.400
uh you know if it works with GPD55 for example uh I'm good because that's the only model I use now if you want to distribute your skill to lots of users

s278
00:25:58.400 --> 00:26:03.600
uh this is where things get kind of hairy because some of those users might be running on it.

s279
00:26:03.600 --> 00:26:05.279
Some of them might be running Haiku.

s280
00:26:05.279 --> 00:26:06.880
Some of them might be running Grock.

s281
00:26:06.880 --> 00:26:07.679
I don't know.

s282
00:26:07.679 --> 00:26:10.880
Um uh some of them might be running Gemini.

s283
00:26:10.880 --> 00:26:12.400
You know, you never know.

s284
00:26:12.400 --> 00:26:14.640
Uh sometimes I meet somebody who does.

s285
00:26:14.640 --> 00:26:21.279
Um but um uh really that's where it gets complicated, right?

s286
00:26:21.279 --> 00:26:32.159
Um because you need to build for the lowest common denominator and ideally for the for the one model that is the weakest as at instruction following.

s287
00:26:32.159 --> 00:26:36.799
For example, GPD5 mini is not a very good rule follower.

s288
00:26:36.799 --> 00:26:40.880
Uh there are things even in impeccable that don't work with GPD5 mini.

s289
00:26:40.880 --> 00:26:48.159
Um it consistently doesn't load certain MD files that I thought it uh it consistently doesn't spin up the life mode.

s290
00:26:48.159 --> 00:26:57.679
So there are uh boundaries to instruction following across these models and it gets especially bad with longer skills that have lots of rules.

s291
00:26:57.679 --> 00:27:00.240
So how do you work around this?

s292
00:27:00.240 --> 00:27:04.159
Well, in impeccable, impeccable really is kind of bionic of sorts.

s293
00:27:04.159 --> 00:27:05.760
It's really not just pros.

s294
00:27:05.760 --> 00:27:11.440
It is a combination of scripts that run in line within the skill at certain times.

s295
00:27:11.440 --> 00:27:13.919
Uh, and then pros around it.

s296
00:27:13.919 --> 00:27:19.279
For example, every time you call impeccable, it runs a file called context.

s297
00:27:19.279 --> 00:27:20.240
MJS.

s298
00:27:20.240 --> 00:27:22.880
And the context MJS does a couple of things.

s299
00:27:22.880 --> 00:27:31.840
The first thing is uh if there is a product MD which is impeccables uh it's almost like design MD but it is for product strategy.

s300
00:27:31.840 --> 00:27:42.240
So wants to understand who's the target audience or what do you want to achieve with this thing which is oftentimes more important in a in a design interview than you know

s301
00:27:42.240 --> 00:27:44.559
how how round do you want your borders to be.

s302
00:27:44.559 --> 00:27:46.400
Um but it supports both.

s303
00:27:46.400 --> 00:27:55.120
It supports product MD and design MD and by default context M.JS JS brings these files together and then spits them into the session.

s304
00:27:55.120 --> 00:27:57.039
Now that's not exciting.

s305
00:27:57.039 --> 00:28:09.120
But when those files are not available, it will actually give the uh the skill structured JSON and say like by the way there is no product MD and here's exactly what you should do about it.

s306
00:28:09.120 --> 00:28:12.880
Or here's another thing that context MJS does.

s307
00:28:12.880 --> 00:28:17.760
It actually makes impeccable self update if there's a new version of impeccable now with your permission.

s308
00:28:17.760 --> 00:28:28.799
So we will it will ask you but it will say hey by the way there's an update available for the impeccable skill uh and here's what you should do now to ask the user whether whether they want to update impeccable.

s309
00:28:28.799 --> 00:28:36.640
So it's it's overloaded in many ways um and it will always tell the model the exact instructions on what to do next.

s310
00:28:36.640 --> 00:28:45.520
And the really interesting thing about this is that I found that that works significantly better than some random rule in the pros of the main scope.

s311
00:28:45.520 --> 00:28:58.080
when you put something out uh from the exit value uh from the standard out of a script uh somehow the model will follow it a lot uh more than before.

s312
00:28:58.080 --> 00:29:06.240
Uh so that could be environment uh aware setup dynamic onboarding repo state gating adaptive flows anything really.

s313
00:29:06.240 --> 00:29:15.360
Um actually before I end this session one of the shortcomings of this technique and this is something to be aware of is prompt caching.

s314
00:29:15.360 --> 00:29:28.000
So this works super super well to keep a skill uh sort of like you know flowing in the right direction instruction following but it does so at the expense of prompt caching.

s315
00:29:28.000 --> 00:29:41.279
Um, if you need prompt caching, if you run the skill many many times, uh, and you want the whole thing to be cached, this is not a good technique to use, but I found it to be very useful in really interactive scenarios.

s316
00:29:41.919 --> 00:29:46.720
All right, number six, hooks that fight back.

s317
00:29:46.720 --> 00:29:51.039
It's something I shipped quite recently and um, and I really like it.

s318
00:29:51.039 --> 00:29:53.840
I want to show you what I mean by that.

s319
00:29:53.840 --> 00:30:01.120
Um, so a lot of people have impeccable their systems, but sometimes they forget to run it.

s320
00:30:01.120 --> 00:30:02.960
Sometimes they're like, you know, I don't know.

s321
00:30:02.960 --> 00:30:05.919
I mean, I mean, I wish Codex is actually pretty good.

s322
00:30:05.919 --> 00:30:10.640
Some of the harnesses are pretty good consistently looping in the right skill.

s323
00:30:10.640 --> 00:30:21.039
Um, but because it now bundles as one skill, oftent times, uh, the harnesses forget to simply call impeccable when you don't explicitly mention it.

s324
00:30:21.039 --> 00:30:26.559
So, now you're building some front-end code and maybe it doesn't follow your design system or whatever.

s325
00:30:26.559 --> 00:30:30.480
Um now that can be solved with hooks.

s326
00:30:30.480 --> 00:30:35.360
Who who has used hooks before in cloud code or codeex?

s327
00:30:35.360 --> 00:30:36.000
A few people.

s328
00:30:36.000 --> 00:30:37.120
Okay, nice.

s329
00:30:37.120 --> 00:30:43.840
Um so this skill that I've built here, Impeccable ships design hooks.

s330
00:30:43.840 --> 00:30:49.360
So I've basically built a design lint that runs under the hood and ships with the skill.

s331
00:30:49.360 --> 00:31:04.240
When you install impeccable uh these hooks install into cloud code, cursor um codeex and github copilot uh and they will keep the model uh exactly where it needs to be.

s332
00:31:04.240 --> 00:31:05.919
So the hooks come to you.

s333
00:31:05.919 --> 00:31:08.880
It's a guardrail that fires on every edit.

s334
00:31:08.880 --> 00:31:15.679
Um and um there are some there are some differences between the different providers here.

s335
00:31:15.679 --> 00:31:19.520
So the hook syntax for codex and clot code is not the same.

s336
00:31:19.520 --> 00:31:22.159
Um and also the behavior is not the same.

s337
00:31:22.159 --> 00:31:28.080
So for instance we found out that with weaker models slightly weaker models like composer and cursor.

s338
00:31:28.080 --> 00:31:36.480
Uh you kind of want to use a um pre-tool use hook that prevents writing of code as opposed to a post tool use hook.

s339
00:31:36.480 --> 00:31:47.679
Post tool use basically happens right after the agent has written a file for example and then it it tells you hey by the way like the contrast of these colors is bad or

s340
00:31:47.679 --> 00:31:53.840
um you know you have a purple gradient in here and then ideally the model is smart enough to actually fix it.

s341
00:31:53.840 --> 00:32:04.960
Um some models don't follow those instructions very well and so if you do a pre-tool who use hook you are actively preventing the writing of this file in the first place.

s342
00:32:04.960 --> 00:32:11.360
So, it's a much more heavy-handed approach, but we needed to do that for certain models and certain hardnesses.

s343
00:32:11.360 --> 00:32:20.720
Um, but this is nice and what's even nicer about it is that you can personalize it to your design system and your use case or whether let's say you use it for code reviews.

s344
00:32:20.720 --> 00:32:31.279
You can personalize it with your own ESLint rules uh with your own uh synthetic syntax guidelines etc. uh and then expand it from there.

s345
00:32:31.279 --> 00:32:37.279
Um, so passive guardrails beat a command no one remembers to run.

s346
00:32:37.279 --> 00:32:43.440
Um, so these are passive guardrails that always keep you in the right lane uh on track.

s347
00:32:43.440 --> 00:32:45.840
Again, that works for linting for formatting.

s348
00:32:45.840 --> 00:32:57.039
Um, of course, if you're using cloud code or codeex, it already uses uh some of the llinters for things like syntax formatting, but design linting is a whole different uh game.

s349
00:32:57.039 --> 00:33:09.360
But I would really encourage you to uh try out hooks in combination with a skill and think about okay well my skill does this how can I create a feedback loop a validation loop

s350
00:33:09.360 --> 00:33:14.480
that uses hooks uh to actually keep me on the right lane.

s351
00:33:14.480 --> 00:33:26.399
Um okay so here's it's hard to show loop uh hooks in action but um you know if you can see this this is roughly how it would happen in an agent.

s352
00:33:26.399 --> 00:33:31.840
Um so uh for instance in this case I would use let's say Gemini does this all the time.

s353
00:33:31.840 --> 00:33:37.120
Gemini uh creates animations on images like crazy.

s354
00:33:37.120 --> 00:33:42.240
Um it will animate any image and it will usually do a hover zoom in effect.

s355
00:33:42.240 --> 00:33:43.679
Um it loves that.

s356
00:33:43.679 --> 00:33:46.000
Um and that's something that impeccable flags.

s357
00:33:46.000 --> 00:33:54.320
And in this case it the hook would fire silently usually that's why I built this fake demo because you can't usually see it.

s358
00:33:54.320 --> 00:33:58.640
Um, and then it will tell the model, hey, by the way, here was a violation.

s359
00:33:58.640 --> 00:34:01.120
The experience of this is that oftentimes you don't have to do anything.

s360
00:34:01.120 --> 00:34:04.000
The model just course corrects and fixes itself.

s361
00:34:04.000 --> 00:34:16.399
Um, now one important thing if you do this and you ship it to users, very important to add a way to create ignore rules or something like that because oftent times these hooks

s362
00:34:16.399 --> 00:34:20.240
have false positives as well and you want a way to configure those hooks.

s363
00:34:20.240 --> 00:34:23.200
Um, otherwise gets really annoying very quickly.

s364
00:34:23.200 --> 00:34:31.760
Um, impeccable ships with these design hooks that uh allow you to create ignore rules at a file basis within a CSS rule.

s365
00:34:31.760 --> 00:34:37.520
So like many granular levels uh to exclude certain files for example.

s366
00:34:37.919 --> 00:34:40.720
Okay, level seven.

s367
00:34:41.040 --> 00:34:44.399
Now you can't really tune pixels through a chat box.

s368
00:34:44.399 --> 00:34:49.599
Uh now this might not be relevant if you're not building a design skill, but I think the general point is relevant.

s369
00:34:49.599 --> 00:35:02.000
So if you think about a skill as harness engineering versus prompting then you think about the harness as a whole right you're living in cloud code for example or you're living in codeex

s370
00:35:02.000 --> 00:35:14.960
or you're living in github copilot now what capabilities of that harness that you can exploit to make the best user experience for your use case that's the question you should ask yourself

s371
00:35:14.960 --> 00:35:25.440
for example uh uh codeex on desktop now has an inapp browser built into the the actual um uh app.

s372
00:35:25.440 --> 00:35:28.000
Can you use this inapp browser in some interesting ways?

s373
00:35:28.000 --> 00:35:32.240
Can you use the browser screenshot tool in some interesting ways?

s374
00:35:32.240 --> 00:35:33.839
Uh and in my case, I could.

s375
00:35:33.839 --> 00:35:47.280
I realized, hey, there's probably a way to connect the inapp browser uh and spin up the development server and just load the page there and then kind of connect it to the main thread in some ways.

s376
00:35:47.280 --> 00:35:55.200
So I can allow the user to visually iterate on that page um instead of in the chat.

s377
00:35:55.200 --> 00:36:02.800
Um and so in Impeccable what this looks like is uh it's not using MCP.

s378
00:36:02.800 --> 00:36:12.160
Um it's simply spinning up a live polar a little server that uh looks for input and in inserts a snippet into your development server.

s379
00:36:12.160 --> 00:36:28.880
Um it then on the page when you do something on the page it sends an event back to that actual polar uh using uh serverside events and then uh and this is I think the the the clever bit maybe or the the bit that makes it all work.

s380
00:36:28.880 --> 00:36:31.440
Um the polar then stops.

s381
00:36:31.440 --> 00:36:34.160
So the polar uh ends itself.

s382
00:36:34.160 --> 00:36:35.760
There's a standard out message.

s383
00:36:35.760 --> 00:36:37.200
We talked about standard out before, right?

s384
00:36:37.200 --> 00:36:47.200
the exit value of this thing and uh the model reads that message and realizes oh something happened I better do something.

s385
00:36:47.200 --> 00:36:52.079
So in this case in the skill itself I give it instructions on how to handle this event.

s386
00:36:52.079 --> 00:37:02.880
I say like well if this event comes in you should probably build some design for this particular section of the page and then you should send it back to this polar

s387
00:37:02.880 --> 00:37:05.599
so that it arrives on the user side.

s388
00:37:05.599 --> 00:37:11.440
And so this is a direct connection between one harness capability and another harness capability.

s389
00:37:11.440 --> 00:37:14.640
So the chat thread and the inapp browser.

s390
00:37:14.640 --> 00:37:20.160
Um and uh yeah this is kind of how how it looks like on a diagram.

s391
00:37:20.160 --> 00:37:24.320
But I think the best way to experience with it is to is to see it.

s392
00:37:24.320 --> 00:37:28.320
So let me bring this up.

s393
00:37:35.119 --> 00:37:36.880
Okay, cursor.

s394
00:37:36.880 --> 00:37:40.000
Uh, I think I'm already in live mode here.

s395
00:37:40.000 --> 00:37:40.800
Um, okay.

s396
00:37:40.800 --> 00:37:44.079
So, I I booted up live mode already.

s397
00:37:44.079 --> 00:37:45.200
I'm now in picker mode.

s398
00:37:45.200 --> 00:37:47.440
I get this little bar here at the bottom.

s399
00:37:47.440 --> 00:37:51.280
Uh, and as you can see, I can pick anything on this page.

s400
00:37:51.280 --> 00:37:58.800
Um, I get now I get this little overlay bar and um I can select all sorts of subcomands within the skill.

s401
00:37:58.800 --> 00:38:03.920
So these are basically translating to MD files that live within the skill.

s402
00:38:03.920 --> 00:38:06.960
Uh I can select the amount of variance I want.

s403
00:38:06.960 --> 00:38:10.000
And then um I can hit go.

s404
00:38:10.000 --> 00:38:26.320
And now um here in the thread you can see that it picked up the actual signal in the main thread uh because the polar stopped h and it now knows hopefully exactly what it needs to do to first wrap this element

s405
00:38:26.320 --> 00:38:28.000
in some special tag.

s406
00:38:28.000 --> 00:38:35.680
then it knows how to create uh variants and uh that are marked up in a special way with CSS.

s407
00:38:35.680 --> 00:38:37.280
Uh and now it did that.

s408
00:38:37.280 --> 00:38:42.160
So now as you can see the uh the thing updated immediately.

s409
00:38:42.160 --> 00:38:49.839
I now get these three variants and I can click through and then if I like one of them I can click accept and accept it.

s410
00:38:49.839 --> 00:38:55.119
If I don't like like one of them I hit uh escape and I'm back in this normal mode.

s411
00:38:55.119 --> 00:39:05.839
Um so this shows sort of like uh how to exploit an a harness capability uh in an effective way for one problem space in this case design.

s412
00:39:05.839 --> 00:39:12.079
Um you can also uh insert elements with this thing um and sort of click into anything here.

s413
00:39:12.079 --> 00:39:15.200
You can um you can draw on top of this and leave comments.

s414
00:39:15.200 --> 00:39:17.599
You can leave annotations if you want.

s415
00:39:17.599 --> 00:39:30.960
Um you can dictate uh you can uh steer the whole page by simply writing into this and then uh again this goes back to the main agent and uh it becomes a steering signal for the whole page.

s416
00:39:30.960 --> 00:39:33.680
Um and you can also visualize lots of things this way.

s417
00:39:33.680 --> 00:39:43.359
I mean you might have read uh Tariq's uh blog post about this about how HTML is a is a really cool way to communicate as opposed to markdown.

s418
00:39:43.359 --> 00:39:49.839
Um I agree and I think also like uh design MD is much better visualized as HTML.

s419
00:39:49.839 --> 00:40:02.960
In this case you see the M design MD of this you know not great website for demonstration purposes but uh um you can use this to your advantage as well if you hijack the inapp browser and use it to your advantage.

s420
00:40:02.960 --> 00:40:06.880
Um so this is how I make use of it.

s421
00:40:07.119 --> 00:40:10.880
Okay number eight.

s422
00:40:10.880 --> 00:40:12.800
It worked on my machine.

s423
00:40:12.800 --> 00:40:18.560
Um well I mean everybody who who's a developer here knows this problem.

s424
00:40:18.560 --> 00:40:22.320
Um this is this hits really hard when you ship a scale.

s425
00:40:22.320 --> 00:40:35.119
Uh there are so many times uh I kind of saw this argument on X was like hey bro just sim link just you know sim link.cloud and all your problems will be gone.

s426
00:40:35.119 --> 00:40:36.720
Um, well, that's great.

s427
00:40:36.720 --> 00:40:41.839
If you're building a simple skill and if you're doing it for yourself, by all means, go for it, right?

s428
00:40:41.839 --> 00:40:44.160
Sim link your claw MD to agents.m MD.

s429
00:40:44.160 --> 00:40:44.800
Amazing.

s430
00:40:44.800 --> 00:40:48.160
Like, sim link the out of everything.

s431
00:40:48.160 --> 00:40:56.480
But it's not great if you're trying to ship a skill to lots of users because again, we just talked about a whole lot of differences these hardes have.

s432
00:40:56.480 --> 00:40:58.000
I'm going to talk about more differences.

s433
00:40:58.000 --> 00:41:03.839
And I know it's annoying because it would be great to simulate those things, but unfortunately, we don't live in that world.

s434
00:41:03.839 --> 00:41:07.440
And unfortunately, Anthropica has still not adopted agents.

s435
00:41:07.440 --> 00:41:08.720
MD.

s436
00:41:08.720 --> 00:41:12.240
So, uh, what are the actual differences?

s437
00:41:12.240 --> 00:41:14.560
For example, we talked about sub agents already.

s438
00:41:14.560 --> 00:41:18.000
We talked about how, well, on the bright side, they're widely supported.

s439
00:41:18.000 --> 00:41:22.960
Now, um, but who can spawn one is very very different.

s440
00:41:22.960 --> 00:41:26.400
So, with claude, you can programmatically do it very easily.

s441
00:41:26.400 --> 00:41:27.680
Codeex needs the user.

s442
00:41:27.680 --> 00:41:28.560
Okay.

s443
00:41:28.560 --> 00:41:32.480
Um, in cursor, it's agent chosen most of the time.

s444
00:41:32.480 --> 00:41:44.960
Um so there are clear differences also if you want to predefine these agents codeex has a different syntax for that uh than claude and and uh and cursor etc.

s445
00:41:44.960 --> 00:41:47.440
Another one is the ask user tool.

s446
00:41:47.440 --> 00:41:53.760
So one of the coolest tools in and the um cla code harness is the ask user question tool.

s447
00:41:53.760 --> 00:41:57.599
Uh it's a really nice tool that you can use to ask the user a question right?

s448
00:41:57.599 --> 00:42:01.920
It brings up this menu say like hey what would you like to do and then you pick some option.

s449
00:42:01.920 --> 00:42:05.440
Well, turns out Codex has a tool like this.

s450
00:42:05.440 --> 00:42:06.560
That's the good news.

s451
00:42:06.560 --> 00:42:10.240
The bad news is that tool is only available in plan mode.

s452
00:42:10.240 --> 00:42:14.560
So, again, big differences between how these things work.

s453
00:42:14.560 --> 00:42:16.640
Uh, and um, what does that mean?

s454
00:42:16.640 --> 00:42:24.720
That means that if you're not running codecs in plan mode, but your skill wants to ask questions, most of the time it simply doesn't.

s455
00:42:24.720 --> 00:42:29.920
It will simply infer from the current context and not ask any questions to the user.

s456
00:42:29.920 --> 00:42:31.359
Uh, which is not great.

s457
00:42:31.359 --> 00:42:39.359
So there's a lot of sentences in the impeccable skill that specifically say if you're codeex you have to stop and ask questions.

s458
00:42:39.359 --> 00:42:42.400
No, you're not smart enough to infer the context.

s459
00:42:42.400 --> 00:42:45.359
Um so if you see lines like this, that's why.

s460
00:42:45.359 --> 00:42:48.240
Another one is background jobs.

s461
00:42:48.240 --> 00:42:52.160
And there's also something you learn through the hard way by doing this.

s462
00:42:52.160 --> 00:42:57.280
For example, this live mode that I just showed you, it's spawning a background task.

s463
00:42:57.280 --> 00:43:00.480
So it's running a shell in a background task.

s464
00:43:00.480 --> 00:43:13.599
Uh and that that's cool because you can keep using the session um and then when the background task finishes the model is automatically waken up gets a gets the message back and then can do something and re react to it.

s465
00:43:13.599 --> 00:43:19.359
Uh where codeex cannot codeex and other harnesses do not react when a background task finishes.

s466
00:43:19.359 --> 00:43:28.000
You actually have to manually say hey by the way this background task can you take a look at what it did and that's not great right if you're doing an automation like this.

s467
00:43:28.000 --> 00:43:33.040
So there are differences in how these tasks are spawned and uh and how they work.

s468
00:43:33.040 --> 00:43:43.280
So that's why if you're using the live mode in cursor or in codeex, it creates a foreground task and it keeps the actual chat thread blocked.

s469
00:43:43.280 --> 00:43:46.000
Uh not ideal, but it makes it actually work.

s470
00:43:46.000 --> 00:43:49.680
So there are subtle differences on how these tasks are spawned.

s471
00:43:49.680 --> 00:43:50.960
Watchers is another example.

s472
00:43:50.960 --> 00:43:52.319
Tail watch exists now.

s473
00:43:52.319 --> 00:43:52.960
That's really cool.

s474
00:43:52.960 --> 00:43:57.440
I mean most of the harnesses have a way to watch for instance a log file.

s475
00:43:57.440 --> 00:44:02.240
Um but those are throttled way harder than uh simply spawning a background task.

s476
00:44:02.240 --> 00:44:04.000
Edit hooks, we talked about this already.

s477
00:44:04.000 --> 00:44:11.280
Um they are different and so lots and lots of behavioral differences, but there's also model differences.

s478
00:44:11.280 --> 00:44:18.079
So for example, in my case, um they all have different tails in the ways they're overfitted.

s479
00:44:18.079 --> 00:44:23.280
Uh for example, Gemini, again, I mentioned this loves to animate pictures.

s480
00:44:23.280 --> 00:44:24.480
It just loves it.

s481
00:44:24.480 --> 00:44:29.599
Um, you have to tell it not to animate pictures if you don't want a hover effect on every picture.

s482
00:44:29.599 --> 00:44:30.720
Doesn't matter where it is.

s483
00:44:30.720 --> 00:44:31.920
It loves it.

s484
00:44:31.920 --> 00:44:35.040
Um, Codeex loves bad letter spacing.

s485
00:44:35.040 --> 00:44:37.200
I don't know why, but it does.

s486
00:44:37.200 --> 00:44:41.520
Um, Codex also loves extremely rounded borders.

s487
00:44:41.520 --> 00:44:44.160
Uh, it will round anything you thought it.

s488
00:44:44.160 --> 00:44:45.200
Uh, it loves it.

s489
00:44:45.200 --> 00:44:48.319
Doesn't matter if it's hospital website or a kids website.

s490
00:44:48.319 --> 00:44:50.400
Uh, it also loves hairline borders.

s491
00:44:50.400 --> 00:44:54.480
And so there are specific tails that are unique to every model.

s492
00:44:54.480 --> 00:44:56.319
And that's not just for design.

s493
00:44:56.319 --> 00:44:57.760
It's for architecture.

s494
00:44:57.760 --> 00:44:58.960
It's for code architecture.

s495
00:44:58.960 --> 00:45:02.480
It's for, you know, preferred npm packages.

s496
00:45:02.480 --> 00:45:05.920
Now, every model is overfitted in different ways.

s497
00:45:05.920 --> 00:45:09.359
Uh finding out how to overfit it usually happens by accident.

s498
00:45:09.359 --> 00:45:13.520
In my case, I have a pretty extensive EVA harness that I run behind the scenes.

s499
00:45:13.520 --> 00:45:16.640
In fact, every line of impeccable is ablation tested.

s500
00:45:16.640 --> 00:45:19.920
So I test every single line and see what it does across all models.

s501
00:45:19.920 --> 00:45:32.000
I don't expect you to do that but it is very good to know that uh that the models are different and are following instructions differently and and the behav behavior the harness behavior is different as well.

s502
00:45:32.000 --> 00:45:42.319
Um and so what impeccable does it creates harness specific and model specific builds for every single uh model.

s503
00:45:42.319 --> 00:45:49.760
Um, you might not have to go all this way for your own purposes, but I just wanted to show you how far you can go with this.

s504
00:45:49.760 --> 00:46:02.560
Uh, for example, it actually has a um substitute variable that picks the right user question tool depending on the hardness or it has these XML blocks for Gemini, for codeex,

s505
00:46:02.560 --> 00:46:09.119
etc. Uh, that will actually insert specific overfitting avoidance rules for the given models.

s506
00:46:09.119 --> 00:46:17.200
Because it turns out if you if you tell Claude not to let her space too much, it will let space in the exact opposite direction.

s507
00:46:17.200 --> 00:46:21.200
So you can't just include it all in the same skill.

s508
00:46:21.200 --> 00:46:32.000
Um and that's why you know you can you know if you if you instrument this way enough you can actually get to this right once ship to all of them uh skill that actually works everywhere.

s509
00:46:32.000 --> 00:46:37.119
It's a lot of work but it does pay off and allows you to create beautiful pictures like this.

s510
00:46:37.119 --> 00:46:53.280
Um now the only other problem is that uh typical install methods like for um uh MPX skills for instance if you've been using MPX skills do not honor um different directories for different harnesses.

s511
00:46:53.280 --> 00:46:58.720
So they actually just take the first directory and then copy it or sim link it into all sorts of folders.

s512
00:46:58.720 --> 00:47:04.720
Um that's why if you go to the impeccable website uh I've built my own CLI to solve this problem.

s513
00:47:04.720 --> 00:47:06.480
Uh that's why it doesn't use impact skills.

s514
00:47:06.480 --> 00:47:13.359
So, I think the committee hasn't quite yet gotten to the point where uh this is a this is an accepted idea and it's annoying.

s515
00:47:13.359 --> 00:47:13.760
I get it.

s516
00:47:13.760 --> 00:47:18.800
It's annoying to compile for different harnesses, but uh I found it worthwhile.

s517
00:47:18.800 --> 00:47:23.520
Finally, um again, build for the lowest common denominator.

s518
00:47:23.520 --> 00:47:29.280
Um a weaker model has opinions just fine, but what it loses is the discipline to follow yours.

s519
00:47:29.280 --> 00:47:38.560
Um so uh codeex for example uh and GPT specifically loves the word gate.

s520
00:47:38.560 --> 00:47:42.240
If you've built a skill in codeex before, it loves gates.

s521
00:47:42.240 --> 00:47:45.359
Um whenever you say, "Hey, why didn't you follow these instructions?"

s522
00:47:45.359 --> 00:47:47.520
You're like, "Well, I think we need a gate."

s523
00:47:47.520 --> 00:47:51.599
Um so I gave it what it loves the most, gates.

s524
00:47:51.599 --> 00:47:54.319
Um but I only do that for codeex.

s525
00:47:54.319 --> 00:47:59.040
So there's a codeex MD that gets loaded on the fly for codeex.

s526
00:47:59.040 --> 00:48:12.000
um and GPT uh and then uh it actually follows like you know okay here are your eight gates you have to pass every single gate and you are not allowed to compress those gates.

s527
00:48:12.000 --> 00:48:15.520
Um that's really important because it loves compressing these instructions as well.

s528
00:48:15.520 --> 00:48:19.440
I'll just skim over it and say like well I guess I do one and two and five and good.

s529
00:48:19.440 --> 00:48:29.839
Um and so um the way you solve this is by actually having it lock every single result of every gate and say like well I just passed gate one great success.

s530
00:48:29.839 --> 00:48:40.880
Um and the most important lesson from this is if the gate can be skipped it will be I mentioned this before right if the model can wiggle itself out out of a difficult situation

s531
00:48:40.880 --> 00:48:43.280
it will absolutely do that.

s532
00:48:43.280 --> 00:48:49.760
uh it will not do all the all the things it needs to do to uh to complete the end result.

s533
00:48:49.760 --> 00:48:53.119
Um so be careful, make it unskippable.

s534
00:48:53.119 --> 00:48:56.240
Um so we just built a harness extension.

s535
00:48:56.240 --> 00:49:01.119
We went from prompting all the way to building a monster.

s536
00:49:01.119 --> 00:49:09.119
Um but uh I think it turned out to be pretty powerful in my case and I wanted to share what I've learned on the way.

s537
00:49:09.119 --> 00:49:12.160
Uh I don't expect you to use all of those techniques.

s538
00:49:12.160 --> 00:49:16.480
I think some of them are pretty exotic and maybe not applicable to every use case.

s539
00:49:16.480 --> 00:49:21.760
Um, but I hope that you find value in uh some of the advice that I've given today.

s540
00:49:21.760 --> 00:49:24.079
Uh, so we've done a whole bunch of things today.

s541
00:49:24.079 --> 00:49:25.839
Nine things a prompt can't do.

s542
00:49:25.839 --> 00:49:33.440
We made it much more deterministic um and uh and made impactable better for that reason.

s543
00:49:33.440 --> 00:49:40.480
Uh if you like to try it out yourself um again you can clone the repository for this talk.

s544
00:49:40.480 --> 00:49:43.359
you can clone uh impeccable minus talks.

s545
00:49:43.359 --> 00:49:50.000
Um but of course it also is useful to just take a look at the actual skill and see how it's built.

s546
00:49:50.000 --> 00:49:58.960
Um the project is completely open source license under Apache 2. Um you can install impeccable MPX impeccable skills install.

s547
00:49:58.960 --> 00:50:02.160
Um and check out the source code on GitHub.

s548
00:50:02.160 --> 00:50:06.400
Um with that um I'm at the end of it.

s549
00:50:06.400 --> 00:50:08.559
Thank you.

s550
00:50:09.104 --> 00:50:11.104
[applause]

s551
00:50:12.800 --> 00:50:18.720
And uh now I think we have uh about 10 minutes for any questions that you have.

s552
00:50:19.040 --> 00:50:21.760
Does anybody have questions?

s553
00:50:21.760 --> 00:50:23.760
Yes.

s554
00:50:25.920 --> 00:50:26.319
Oh, sorry.

s555
00:50:26.319 --> 00:50:27.839
What was that?

s556
00:50:27.839 --> 00:50:29.280
A link

s557
00:50:29.280 --> 00:50:30.319
to the repository.

s558
00:50:30.319 --> 00:50:30.880
Yeah.

s559
00:50:30.880 --> 00:50:35.305
Um so the the this is hard to see but

s560
00:50:35.305 --> 00:50:35.680
[music]

s561
00:50:35.680 --> 00:50:39.680
uh let me let me put it up here.

s562
00:50:39.680 --> 00:50:45.599
Um this is the repository uh for the talks.

s563
00:50:45.599 --> 00:50:47.520
Yeah.

s564
00:50:47.520 --> 00:50:48.640
Awesome.

s565
00:50:48.640 --> 00:50:52.160
My question was you mentioned that.

s566
00:51:03.200 --> 00:51:14.960
Yeah, I the question is I mentioned that it breaks prompt caching um the the actual sort of trick the technique to actually get uh something back from a script

s567
00:51:14.960 --> 00:51:16.400
um within a skill.

s568
00:51:16.400 --> 00:51:19.760
And the reason is because the result is dynamic, right?

s569
00:51:19.760 --> 00:51:21.040
It could be anything.

s570
00:51:21.040 --> 00:51:25.920
Um, so unless the result is always the same, it's a dynamic shell execution.

s571
00:51:25.920 --> 00:51:29.280
Um, so it gets inserted into the into the thread.

s572
00:51:29.280 --> 00:51:33.040
Now the now to be fair, the skill will still be cached.

s573
00:51:33.040 --> 00:51:46.240
So the skill will still be cached but uh but I guess I'm differentiating between the skill with inline uh you know static content versus the skill with sort of like a dynamic instruction to call out.

s574
00:51:46.240 --> 00:51:51.359
So this part will not get cached yeah that was that was my main point.

s575
00:51:51.359 --> 00:51:53.359
Yeah.

s576
00:51:59.200 --> 00:51:59.520
Yeah.

s577
00:51:59.520 --> 00:52:03.359
what is the process on how I evaluate and iterate on this skill?

s578
00:52:03.359 --> 00:52:09.280
Um, so I the process is pretty involved.

s579
00:52:09.280 --> 00:52:12.079
Um, let me see.

s580
00:52:12.079 --> 00:52:16.400
Uh, see if I can bring this up on screen.

s581
00:52:16.400 --> 00:52:23.599
Uh, uh, okay.

s582
00:52:23.599 --> 00:52:24.400
Here we go.

s583
00:52:24.400 --> 00:52:26.160
So, here's a glimpse.

s584
00:52:26.160 --> 00:52:28.960
Oh, no. Okay, I shut down the server.

s585
00:52:28.960 --> 00:52:29.680
That's fine.

s586
00:52:29.680 --> 00:52:31.920
Um, okay.

s587
00:52:31.920 --> 00:52:33.040
I'll just voice over.

s588
00:52:33.040 --> 00:52:47.520
So, um, yeah, I mentioned I built an uh Evals harness and, um, and so I've created myself a harness that, uh, re closely recreates the conditions and the tools

s589
00:52:47.520 --> 00:52:49.520
of every harness that I care about.

s590
00:52:49.520 --> 00:52:53.920
So, for instance, it uses the clawed code SDK.

s591
00:52:56.079 --> 00:52:56.400
Yes.

s592
00:52:56.400 --> 00:52:59.440
Uh sorry guys, can you uh can you lower your volume a little bit?

s593
00:52:59.440 --> 00:53:04.640
Um because uh people are still trying to hear the questions.

s594
00:53:04.720 --> 00:53:06.800
Um thank you.

s595
00:53:06.800 --> 00:53:08.640
So how do I how do I test this?

s596
00:53:08.640 --> 00:53:09.280
How do I build it?

s597
00:53:09.280 --> 00:53:10.240
So this it's a combination.

s598
00:53:10.240 --> 00:53:14.400
So first of all, Impeccable has a ton of end to-end tests in the repository.

s599
00:53:14.400 --> 00:53:20.319
Um uh that's both L&M driven tests as well as um endto-end playright tests.

s600
00:53:20.319 --> 00:53:25.440
So that's one uh and that's useful for things like testing the live mode scripts for example.

s601
00:53:25.440 --> 00:53:29.359
Um, but then beyond that, how do I test that it actually works?

s602
00:53:29.359 --> 00:53:31.200
Uh, well, I've built a EVA harness.

s603
00:53:31.200 --> 00:53:39.920
That one is not open source yet, but I built a EVAL harness that closely replicates every um every model harness that I care about right now.

s604
00:53:39.920 --> 00:53:51.440
Specifically, right now uh uh cloud codecs in Gemini and um and I'm trying to expand to more and it also recreates the tools like for instance browser screenshot tools

s605
00:53:51.440 --> 00:53:53.440
uh or something along those lines.

s606
00:53:53.440 --> 00:54:01.040
And um and then it also recreates um the because some parts of impeccable are interactive.

s607
00:54:01.040 --> 00:54:08.880
In the initialization of impeccable oftent times the user gets asked so you know what would you what would you like your page not to feel like?

s608
00:54:08.880 --> 00:54:16.160
And so you get these interactive like back and forth and so I've built this LLM that acts as the user against the other LLM.

s609
00:54:16.160 --> 00:54:20.000
And so it does like an interactive you know back and forth turn.

s610
00:54:20.000 --> 00:54:26.720
Um so I've built that harness and then I've built a um mixture of expert design judge that runs on top of it.

s611
00:54:26.720 --> 00:54:30.880
So basically give it eyes uh to evaluate each result.

s612
00:54:30.880 --> 00:54:37.839
Uh and then I can run uh across 20 different niches like for instance Italian restaurant.

s613
00:54:37.839 --> 00:54:51.200
Um I run across all models that I care about GPD55, Opus, Sonnet um and do like five to 10 tests um for each of those uh for each skill release to see you know how it changed.

s614
00:54:51.200 --> 00:55:02.640
I also run against competitors for instance I run against the front end design skill to see does it make a difference um um and and and how does it make it worse or better

s615
00:55:02.640 --> 00:55:14.319
um and then beyond that I'm doing ablation testing that's harder and more expensive I would say um so I don't recommend it for everyone but this the ablation testing so every

s616
00:55:14.319 --> 00:55:22.559
you'll see this in the source code of impeccable every rule has sort of an XML tag that says like you know a unique identifier of that particular line.

s617
00:55:22.559 --> 00:55:35.920
Um, and that that will be used by the harness to then do a test where it removes that line um runs the evolves against all models and then adds the line back in

s618
00:55:35.920 --> 00:55:41.839
and then uh uh uses the detection engine of impeccable the deterministic one to see did it actually change.

s619
00:55:41.839 --> 00:55:42.000
Right?

s620
00:55:42.000 --> 00:55:42.160
Right.

s621
00:55:42.160 --> 00:55:51.040
So if there's a line that says, "Hey, don't don't do like gray on colorful backgrounds for for contrast purposes."

s622
00:55:51.040 --> 00:55:57.839
Um there's a there's an ablation test and then a deterministic check or feedback loop that tests against it.

s623
00:55:57.839 --> 00:56:09.040
So uh in short quite involved um but uh but I really it started you know vibes based uh and now it's really uh truly um well tested.

s624
00:56:09.040 --> 00:56:10.640
Yeah.

s625
00:56:10.640 --> 00:56:10.960
Yep.

s626
00:56:10.960 --> 00:56:13.040
Go ahead.

s627
00:56:15.760 --> 00:56:16.480
Sorry,

s628
00:56:16.480 --> 00:56:18.720
you set up for evaluating taste.

s629
00:56:18.720 --> 00:56:20.720
Yes.

s630
00:56:22.240 --> 00:56:23.280
Yes.

s631
00:56:23.280 --> 00:56:30.640
Um I do have I do have um evals for evaluating taste, but I don't think they work particularly well.

s632
00:56:30.640 --> 00:56:34.240
Um I just talked to uh Ben from Contra about this.

s633
00:56:34.240 --> 00:56:44.960
Um I don't think I I mean I know I know um some of my colleagues might disagree um but I don't think taste can be solved at a model level.

s634
00:56:44.960 --> 00:56:57.280
Um I actually think it's a it's a fundamentally human thing um because taste is scars and unique and once everybody uses the same taste uh it becomes ubiquitous and then we don't think it's tasteful anymore.

s635
00:56:57.280 --> 00:57:01.920
So it's it I think it's uh um I think it's hard.

s636
00:57:01.920 --> 00:57:06.480
Um and I also think the models are particularly bad at evaluating taste.

s637
00:57:06.480 --> 00:57:15.119
So for example um there are certain things that the models can evaluate well like hey is this is is the correct thing in the first viewport right?

s638
00:57:15.119 --> 00:57:28.160
So functional stuff that works but what doesn't work and here's one example I've built again this mixture of judges and one judge rates whether the first viewport looks great right and is effective.

s639
00:57:28.160 --> 00:57:41.440
Um, and one of the tells is that uh, Gemini, for example, the more stuff there is in the first viewport, the higher it rates it, right?

s640
00:57:41.440 --> 00:57:42.640
This is just a general rule.

s641
00:57:42.640 --> 00:57:47.359
Like if you just cram the viewport full, it gives it a higher ranking.

s642
00:57:47.359 --> 00:57:51.520
And so there's a interesting example of like, you know, the models are often maximalist, right?

s643
00:57:51.520 --> 00:57:53.920
They like, well, more is more, I guess.

s644
00:57:53.920 --> 00:57:59.599
Um, and so oftent times I build judges that actually invert the response of the model.

s645
00:57:59.599 --> 00:58:02.880
Uh, which is really strange, but it works.

s646
00:58:02.880 --> 00:58:05.359
Um, where it sort of judges something very high.

s647
00:58:05.359 --> 00:58:08.160
I'm like, okay, that's definitely not a good design.

s648
00:58:08.160 --> 00:58:21.280
Um, so anyway, I don't think it's solved and I don't think it's solvable, but um, I do have, I would say, a tool that gives you the design director eyes that works marginally better than random.

s649
00:58:21.280 --> 00:58:29.040
And that's good enough for me for like a first pass and then I use my own human eyes to evaluate results and annotate them.

s650
00:58:30.400 --> 00:58:31.119
Any other questions?

s651
00:58:31.119 --> 00:58:32.400
Yeah, over here.

s652
00:58:32.400 --> 00:58:35.760
What do you say is the future for skills?

s653
00:58:36.160 --> 00:58:37.599
The future for skills.

s654
00:58:37.599 --> 00:58:42.480
So I would say that's a that's a broad question.

s655
00:58:42.480 --> 00:58:48.640
Um yeah.

s656
00:58:50.960 --> 00:58:51.359
Yeah.

s657
00:58:51.359 --> 00:58:56.160
So I'll first answer for impeccable and for me.

s658
00:58:56.160 --> 00:59:04.480
So in the in the case of impeccable I think we're we're definitely outgrowing uh the skill platform kind of what's possible with skills.

s659
00:59:04.480 --> 00:59:07.599
I think the the live mode is a good example of that.

s660
00:59:07.599 --> 00:59:14.720
the live mode um was sort of like a Jurassic Park experiment to see like can I do this?

s661
00:59:14.720 --> 00:59:23.040
And the answer is yes is um I think it's it's it's working better than I expected but um it still has a lot of problems.

s662
00:59:23.040 --> 00:59:28.720
I mean it would be way better to do this in a first party harness integration or like a like a first party tool.

s663
00:59:28.720 --> 00:59:35.040
Um so I think there are limits that I'm hitting where um where skills might not be effective anymore.

s664
00:59:35.040 --> 00:59:42.000
I think in general I would say most skills should probably be written by the individual users.

s665
00:59:42.000 --> 00:59:50.799
I think those that actually go go through the effort of packaging a skill and sharing it with others need to invest more time than they currently do.

s666
00:59:50.799 --> 00:59:52.480
So I guess that's my hot take.

s667
00:59:52.480 --> 01:00:02.960
I think right now I've I've seen plenty of skills that are distributed that uh do not work well in a model that the author didn't use for example, right?

s668
01:00:02.960 --> 01:00:09.280
And so I think I think we just have to raise the bar of what's acceptable to ship uh to to to people.

s669
01:00:09.280 --> 01:00:11.839
I mean again this is like the works on my machine thing.

s670
01:00:11.839 --> 01:00:17.440
Um I would rather see less skills in the ecosystem that are really battle tested and proven.

s671
01:00:17.440 --> 01:00:23.280
Um and I I hope we're shifting towards that because right now it's sort of like a wide west.

s672
01:00:23.280 --> 01:00:23.520
Yeah.

s673
01:00:23.520 --> 01:00:25.280
Go ahead.

s674
01:00:25.280 --> 01:00:33.119
There's no like way to test it all.

s675
01:00:37.280 --> 01:00:39.359
There's no common way to test the skills.

s676
01:00:39.359 --> 01:00:40.000
Yeah.

s677
01:00:40.000 --> 01:00:41.440
And that could be a opportunity.

s678
01:00:41.440 --> 01:00:42.160
That's a good point.

s679
01:00:42.160 --> 01:00:42.720
Yeah.

s680
01:00:42.720 --> 01:00:45.599
I guess I could I do have the tool for that.

s681
01:00:45.599 --> 01:00:46.426
That's true.

s682
01:00:46.426 --> 01:00:46.480
[laughter]

s683
01:00:46.480 --> 01:00:48.640
Yes, I could do something with it.

s684
01:00:48.640 --> 01:00:49.280
Yeah.

s685
01:00:49.280 --> 01:01:00.160
uh right now it's purely built for my own purposes but uh but yeah the same is true for for instance like the the impeccable installer and compiler um I don't think most people know that it exists

s686
01:01:00.160 --> 01:01:07.200
uh that it can compile to every harness and that it has these substitution techniques and stuff like this like I could probably release that standalone as well.

s687
01:01:07.200 --> 01:01:09.440
Yeah, it's a good point.

s688
01:01:09.440 --> 01:01:11.920
Yeah, go ahead.

s689
01:01:17.839 --> 01:01:20.400
MCP having skills on the server.

s690
01:01:20.400 --> 01:01:23.040
How would that work?

s691
01:01:25.359 --> 01:01:26.079
Oh, I see.

s692
01:01:26.079 --> 01:01:26.480
Yeah.

s693
01:01:26.480 --> 01:01:32.160
Um, to be honest, I haven't tried it out yet.

s694
01:01:32.160 --> 01:01:34.640
Um, or I haven't really read too much into it.

s695
01:01:34.640 --> 01:01:49.119
I think MCP in general um you know I worry greatly about context pollution and um I do that with skills too and I think um I'm not using MCP a lot for that reason

s696
01:01:49.119 --> 01:01:52.079
um because it polluted my context many times.

s697
01:01:52.079 --> 01:01:57.280
Um how do skills work in MCP?

s698
01:02:05.520 --> 01:02:07.520
Oh, you can download a skill from MCP server.

s699
01:02:07.520 --> 01:02:07.760
Yeah.

s700
01:02:07.760 --> 01:02:09.760
Okay.

s701
01:02:20.799 --> 01:02:22.799
Yeah.

s702
01:02:24.079 --> 01:02:24.240
Yeah.

s703
01:02:24.240 --> 01:02:24.480
Yeah.

s704
01:02:24.480 --> 01:02:26.480
Yeah.

s705
01:02:27.280 --> 01:02:30.160
So, what's the recommended way of packaging them and distributing them?

s706
01:02:30.160 --> 01:02:30.319
Yeah.

s707
01:02:30.319 --> 01:02:31.680
It's a good good topic.

s708
01:02:31.680 --> 01:02:37.680
Um so of course like the harnesses and the and the frontier labs have their own ways.

s709
01:02:37.680 --> 01:02:43.839
I mean um Codex has a marketplace that you can use for distribution plug-in marketplace.

s710
01:02:43.839 --> 01:02:46.480
Um uh Claude Code has a marketplace as well.

s711
01:02:46.480 --> 01:02:49.119
I think they started with the marketplace technique.

s712
01:02:49.119 --> 01:02:51.200
Those marketplaces don't work particularly well.

s713
01:02:51.200 --> 01:02:54.000
I mean the Claude code one for sure doesn't work particularly well.

s714
01:02:54.000 --> 01:03:04.319
I know this for a fact because I mean the update me mechanism often doesn't work and people are like well my skill doesn't update and oftentimes there's a caching issue.

s715
01:03:04.319 --> 01:03:12.400
So it my experience has been hit or miss with the native methods of distributing and then of course it only for that particular provider.

s716
01:03:12.400 --> 01:03:16.240
That's why projects like uh skills.sh exist.

s717
01:03:16.240 --> 01:03:26.480
But again, the problem with MPX skills right now, it does it it does it it doesn't allow for like, you know, more advanced skill use cases like, you know, compiled for every different harness.

s718
01:03:26.480 --> 01:03:31.200
I have a pull request um in the in the repository.

s719
01:03:31.200 --> 01:03:41.119
Um and I've uh I've bugged Andrew a couple times about it, but um he uh he still has to get it merged or agree to agree with me on that, I guess.

s720
01:03:41.119 --> 01:03:43.520
Um I think uh we're still discussing.

s721
01:03:43.520 --> 01:03:47.920
Um but yeah, MPX skills I think is a great project in general.

s722
01:03:47.920 --> 01:03:51.839
I think it'd be great if we could sort of like standardize around it.

s723
01:03:51.839 --> 01:03:55.520
Um there's also one from Microsoft trying to do that.

s724
01:03:55.520 --> 01:03:56.720
Um a project from Microsoft.

s725
01:03:56.720 --> 01:03:58.079
I forgot the name of it.

s726
01:03:58.079 --> 01:04:02.160
Um but there's definitely no no industry standard for distribution yet.

s727
01:04:02.160 --> 01:04:07.440
Um yeah, I'm not I don't love having to maintain my own CLI installer.

s728
01:04:07.440 --> 01:04:08.720
I would rather not.

s729
01:04:08.720 --> 01:04:09.680
Uh it's annoying.

s730
01:04:09.680 --> 01:04:20.079
Um but uh it does make it so it plays safe with all harnesses installed, the hooks in the right part of the system, etc. So it's um Yeah.

s731
01:04:20.079 --> 01:04:22.079
Yeah.

s732
01:04:22.079 --> 01:04:22.400
Okay.

s733
01:04:22.400 --> 01:04:27.520
I think I'm way out of time, but come up and speak with me if you like.

s734
01:04:27.520 --> 01:04:30.799
Yeah, I would say I'll end it here, but yeah, come come up if you like.

s735
01:04:30.799 --> 01:04:33.059
Um let me just uh Thank you.

s736
01:04:33.059 --> 01:04:35.059
[applause]

s737
01:04:50.089 --> 01:04:52.089
[music]