WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7jjudsEhBtM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7jjudsEhBtM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.040 --> 00:00:15.599
Hi everyone, I am Yogi.

s3
00:00:15.599 --> 00:00:19.520
I work at Faxet as principal AI engineer.

s4
00:00:19.520 --> 00:00:23.600
We are a financial data and research company.

s5
00:00:23.600 --> 00:00:34.880
I'm going to talk about how to build skill centric agentic products and I'm going to post slides so you don't have to keep uh taking photos.

s6
00:00:34.880 --> 00:00:38.719
So that's my exandle yogi not the bear.

s7
00:00:38.719 --> 00:00:47.920
Um so let's connect there and uh let's begin.

s8
00:00:49.760 --> 00:01:02.640
So in the last year's talk in this very conference I talked about blueprints and what blueprints were really a simple set of steps or recipe that you can hand over

s9
00:01:02.640 --> 00:01:20.320
to agents so that agent doesn't have to discover its path every time and when I look back it was simply a skill in a very naive form and a lot has changed since Then

s10
00:01:20.320 --> 00:01:28.960
Antropic has shipped skills in last year October and even warned us not to build agents.

s11
00:01:28.960 --> 00:01:38.640
But on serious note since Antropic had already open source the skills there was no point in trying to maintain our own standards.

s12
00:01:38.640 --> 00:01:44.799
So we moved away from blueprints and just adopted skills fully.

s13
00:01:44.799 --> 00:01:54.320
And this talk really inspired me to build skill centric agent and I'm going to share some of my learnings from that journey.

s14
00:01:55.600 --> 00:01:57.280
So a quick raise of hands.

s15
00:01:57.280 --> 00:02:01.439
How many of you really build skills here?

s16
00:02:02.640 --> 00:02:04.799
So almost all of you.

s17
00:02:04.799 --> 00:02:11.920
Now lower your hands if you have built that only in context of cloud code and codeex.

s18
00:02:11.920 --> 00:02:19.120
and raise your hand if you have like built your own harness and added you know skills to that.

s19
00:02:19.280 --> 00:02:21.440
So okay so I see a few hands.

s20
00:02:21.440 --> 00:02:39.440
So reason I'm asking is when I see most of the online discourse it's about you know coding harnesses and skills in context of the coding agents and how to write great skills like so we need that but today I'm not going to talk about that

s21
00:02:39.440 --> 00:02:54.720
what I'm going to focus on is skills in the context of agentic products and how do you really add support for skills in in your own harness and how to scale it at enterprise scale.

s22
00:02:55.200 --> 00:02:58.080
So traditionally product used to look like this.

s23
00:02:58.080 --> 00:03:04.560
A surface made of screens, buttons, forms and dashboards.

s24
00:03:04.560 --> 00:03:08.319
The user navigated this UI.

s25
00:03:08.319 --> 00:03:15.840
But nowadays we are seeing more and more these kind of interfaces where agent is at the forefront.

s26
00:03:15.840 --> 00:03:26.480
The user either talks to the agent or agent behind the scene is the main decision maker helping users navigate your product.

s27
00:03:26.800 --> 00:03:35.760
Now if the agent becomes the main interface for your product then where do features live?

s28
00:03:37.040 --> 00:03:42.720
This framing of who, what, and how really helps to answer that question.

s29
00:03:42.720 --> 00:03:45.760
Prompts define who the agent is.

s30
00:03:45.760 --> 00:03:49.040
Tools define what it can connect to.

s31
00:03:49.040 --> 00:03:53.440
And skills really tell you how a task gets done.

s32
00:03:53.440 --> 00:04:00.159
And this is the great place to keep your business logic that shapes your agents behavior.

s33
00:04:00.159 --> 00:04:03.280
So skills are the new features.

s34
00:04:03.280 --> 00:04:06.080
And you can see this with this example.

s35
00:04:06.080 --> 00:04:22.160
So equity research and wealth management are two very important workflows in uh finance and these all used to be the buttons, dropdowns and screens and now they are merely skills.

s36
00:04:23.280 --> 00:04:37.600
One of the most uh underrated thing about skills especially when talking about agentric products is how it has enabled to build skills for anyone who has good understanding of the products.

s37
00:04:37.600 --> 00:04:43.919
So if skills are new features and this new features can be shipped by anyone in the company.

s38
00:04:43.919 --> 00:04:46.320
The question is what's the role of engineer?

s39
00:04:46.320 --> 00:04:53.199
Then the role of engineer is shifting from shipping features to shipping harnesses.

s40
00:04:53.199 --> 00:04:57.600
Harnesses that are smooth vehicles for your skills to run.

s41
00:04:57.600 --> 00:05:04.240
But before diving into that, let's just get to the basics and try to understand what skill is.

s42
00:05:04.240 --> 00:05:09.360
The dictionary meaning of skill is the ability to do something well.

s43
00:05:09.360 --> 00:05:14.960
And what it means is that your model can live with it without it.

s44
00:05:14.960 --> 00:05:20.400
But it obviously going to do better in presence of skill.

s45
00:05:20.400 --> 00:05:29.759
A better definition for agent skill is a standardized way to teach AI agents how to do a specific task well.

s46
00:05:29.759 --> 00:05:41.759
And a simple skill could contain just like a markdown but a very complex can have multiple references to the files and executable scripts.

s47
00:05:43.199 --> 00:05:46.240
The skill.md is the heart of your skill.

s48
00:05:46.240 --> 00:05:58.880
Name and description in the front matter are the key uh things that will help you to discover the skill and the business logic and instruction goes into the body of the skill

s49
00:05:58.880 --> 00:06:04.000
and which will also contain references to the files and scripts.

s50
00:06:04.319 --> 00:06:08.720
So let's see how to add a skill support in your harness.

s51
00:06:08.720 --> 00:06:13.840
So what do you need to do like a bare minimum skill in your harness?

s52
00:06:13.840 --> 00:06:20.960
You only need these three things like skill registry, a system prompt, and a basic file read tool.

s53
00:06:20.960 --> 00:06:29.600
If you're running scripts, then obviously you're going to need either bash or maybe a code uh running sandbox environment.

s54
00:06:29.600 --> 00:06:37.680
But this is the bare minimum requirement for adding support of skills like in your harness.

s55
00:06:37.680 --> 00:06:40.479
A simple skill registry looks like this.

s56
00:06:40.479 --> 00:06:42.319
So what is a skill registry?

s57
00:06:42.319 --> 00:06:48.000
It just like collection of skills with their name, description and path.

s58
00:06:48.000 --> 00:06:53.600
And we are going to see this with three example skills.

s59
00:06:53.600 --> 00:07:02.479
Company research skill which is supposed to do a very basic web search for a company and produce a markdown.

s60
00:07:02.479 --> 00:07:08.000
a report HTML skill that will turn the markdown into an HTML.

s61
00:07:08.000 --> 00:07:14.160
A report PDF skill that will take that markdown and turn into PDF.

s62
00:07:15.280 --> 00:07:18.720
Now how does really the agent discover the skill?

s63
00:07:18.720 --> 00:07:30.160
So you have your registry then you form a skill concat concatenating the name description path and put it in your system prompt.

s64
00:07:30.160 --> 00:07:44.319
So if you notice like we are only using the name and description path in in this system prompt and not the skill body and that's what what they call about is progressive disclosure.

s65
00:07:44.319 --> 00:07:52.240
Agent is going to read the skills and only pick the skills that it is going to read and follow the instructions from there.

s66
00:07:52.240 --> 00:07:58.400
And then you need your agentic loop that will run this system prompt.

s67
00:07:58.400 --> 00:08:12.400
Here we're keeping track of all the messages in uh in that messages array and we're going to call the model with messages and agent tools and for every turn it's either

s68
00:08:12.400 --> 00:08:24.479
looking for like making a tool call and all the tool calls get appended to the messages and if there is like no tool call we are just going to output the end of the

s69
00:08:24.479 --> 00:08:29.280
uh program and going to show the output.

s70
00:08:29.280 --> 00:08:33.839
So in this case what does like agent see in our example.

s71
00:08:33.839 --> 00:08:51.519
So it sees that the skills that are available and then the activation part is when it looks for the company research skill for take this example of if you're asking to publish a report of NVDI

s72
00:08:51.519 --> 00:09:07.519
it is going to call company research do the web searches and then use the build report skill to produce the report HTML And the output looks like something like this.

s73
00:09:07.519 --> 00:09:15.440
Now what are the learnings from some of this is the descriptions are really the routing signals.

s74
00:09:15.440 --> 00:09:25.120
And what I mean by that if you noticed I had like two different skills report HTML and report PDF.

s75
00:09:25.120 --> 00:09:28.880
But when I showed the example, you saw only HTML.

s76
00:09:28.880 --> 00:09:37.600
And the reason is I have this description saying that use the skill only when user ask for a PDF report.

s77
00:09:37.600 --> 00:09:39.200
Focus on this word PDF.

s78
00:09:39.200 --> 00:09:39.519
Right?

s79
00:09:39.519 --> 00:09:51.760
So that is the trigger word that helps agent to know which skill to pick and that's why descriptions are called routing signals.

s80
00:09:51.760 --> 00:10:04.560
And it's very important to keep your descriptions aligned to the user request and not about the skill itself.

s81
00:10:04.560 --> 00:10:18.560
It's also important to keep your descriptions distinct enough so that agent doesn't get confused and make sure don't let your skills get stale because these are the reasons why your

s82
00:10:18.560 --> 00:10:21.360
skills don't get triggered.

s83
00:10:21.360 --> 00:10:37.760
One more like very important difference between like skills when we talk in agentic product context is most of the skills are only model driven because for nontechnical users we're not adding that cognitive load to remember them

s84
00:10:37.760 --> 00:10:42.000
uh to keep the track of all the skills.

s85
00:10:44.079 --> 00:10:49.920
Another learning that had was cut by user intent and not by data model.

s86
00:10:49.920 --> 00:10:54.320
So when I started building the skill library, I had very narrow use cases.

s87
00:10:54.320 --> 00:10:58.720
So add skill for estimation analysis or add for fundamentals.

s88
00:10:58.720 --> 00:11:04.720
But I got real use cases and those use cases were not reflecting the data model.

s89
00:11:04.720 --> 00:11:14.399
Those were really about the real use cases and I had to refactor this multiple times and that is okay, right?

s90
00:11:14.399 --> 00:11:23.440
you start simply with narrower use cases and as you discover more use cases you start refactoring your skill library.

s91
00:11:23.440 --> 00:11:35.360
So in practice it means that you know instead of having an estimate analysis skill you should have earning preparation skill instead of having a skill for news and analysis

s92
00:11:35.680 --> 00:11:40.399
analyst rating skill you should have a pre-market briefing skill.

s93
00:11:41.760 --> 00:11:52.880
So we updated our uh stack to a new model and our agent start uh failing because it was not obeying the skills.

s94
00:11:52.880 --> 00:11:54.240
Nothing was changed.

s95
00:11:54.240 --> 00:11:59.680
Not a single line in the skill was changed but still it failed.

s96
00:11:59.680 --> 00:12:15.120
And when when we dig under the hood what was happening it was that this like new model was very focusing on beginning of the skill and we had very critical instruction at the end of the skill.

s97
00:12:15.120 --> 00:12:24.639
So that's why it's very important to run evals and skills without evals are really just wishful thinking.

s98
00:12:24.639 --> 00:12:35.200
Skills are not the documentation and a lot of people treat them like that and skills are really the contracts versioned to a model.

s99
00:12:35.200 --> 00:12:42.000
So whenever you're upgrading a model, make sure to rerun uh your eval.

s100
00:12:43.920 --> 00:12:51.120
So when you have like a few skills, shoving them in the system prompt really works.

s101
00:12:51.120 --> 00:12:56.160
But as soon as you start growing your skill stack, this falls apart.

s102
00:12:56.160 --> 00:13:08.000
When you have like more than 10 skill, maybe that's like a good point to think start thinking about, you know, how can you shortlist the skills that you're going to add to the system prompt.

s103
00:13:08.000 --> 00:13:20.639
And this could mean just like having embeddings and have similarity search and short lists or a smaller model that can shortlist the skills and add to the system prompt.

s104
00:13:20.639 --> 00:13:24.959
The real trouble really starts when you have hundreds of skills.

s105
00:13:24.959 --> 00:13:38.240
At that point, you really need hierarchy of skills and metadata filters and the governance in the place to keep your uh library searchable and coherent.

s106
00:13:40.720 --> 00:13:52.639
So there are like five aspect of skill library governance and these are admission, ownership, boundaries, life cycle and coherence.

s107
00:13:52.639 --> 00:14:04.480
Sounds very enterprisey right but each of this really answer like a very core question and we will get to that in a second.

s108
00:14:04.480 --> 00:14:10.880
But when you hear governance, it really doesn't need to be a red tape bottleneck.

s109
00:14:10.880 --> 00:14:14.240
It really depends on how you're implementing it.

s110
00:14:14.240 --> 00:14:19.519
How much automation is in place with proper human in the loop.

s111
00:14:19.519 --> 00:14:28.399
And the good news is that we can borrow a lot of good practices from code and apply them to the skills.

s112
00:14:28.399 --> 00:14:33.680
And these coding practices like has worked for decades.

s113
00:14:34.639 --> 00:14:37.199
So what does admission even mean?

s114
00:14:37.199 --> 00:14:44.880
What it means that should this skill even exist or it should go to an already existing skill.

s115
00:14:44.880 --> 00:14:50.880
And in practice we build automated get for the registries with human in the loop.

s116
00:14:50.880 --> 00:14:57.040
And this is very analog log to how we do the PR review process.

s117
00:14:58.800 --> 00:15:01.360
Who maintains your skill?

s118
00:15:01.360 --> 00:15:07.600
Just like how features are maintained by application teams, we need skills to be maintained by application teams.

s119
00:15:07.600 --> 00:15:14.959
And like code owners, we need to have a dedicated skill owners like named maintainers for your skills.

s120
00:15:17.199 --> 00:15:20.560
What happens to the skill over time?

s121
00:15:20.560 --> 00:15:33.920
You need to have semantic version of the skills and also whenever you're getting rid of skills you need to have deprecation warnings and make sure that your changes are reflected in the change logs.

s122
00:15:42.399 --> 00:15:47.440
So when you have large number of skills, the library should still make sense.

s123
00:15:47.440 --> 00:16:02.320
And just like features are cohesive in a good product, we need to make sure that we are conducting periodic audits and skill validation checks so that your skills would really

s124
00:16:02.320 --> 00:16:04.560
make sense.

s125
00:16:10.399 --> 00:16:28.160
Whenever you have a lot of skills, it's important to have the allow list tools in in the skills and your tools are supposed to be access control and this is very important to keep the correct boundaries around skills.

s126
00:16:30.480 --> 00:16:33.040
So what are the main takeaways?

s127
00:16:33.040 --> 00:16:38.959
The main takeaways are skills are the features in your agentic products.

s128
00:16:38.959 --> 00:16:51.440
Our role is shifting from features that is shipping features to shipping harnesses and routing mechanism doesn't get tuned as you scale.

s129
00:16:51.440 --> 00:17:00.320
It changes the mechanism itself and at enterprise scale the skill library governance is really non-negotiable.

s130
00:17:01.440 --> 00:17:02.560
That was my talk.

s131
00:17:02.560 --> 00:17:05.086
Thank you very much for listening.

s132
00:17:05.086 --> 00:17:07.086
[applause]

s133
00:17:21.461 --> 00:17:23.461
[music]