From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, Runlayer

AI Engineer · 18 min · 176 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hello everyone.
00:12I am Rafael.
00:14Today I want to talk about self-improving agents, but not in a single-player mode where only one agent gets smarter.
00:21I want to discuss how to apply this concept, combined with network effects, to create a company or even an enterprise that self-improves while you sleep.
00:34Imagine the best engineer you've ever worked with, or salesperson, or lawyer.
00:41The one who, every time he solved a really difficult problem, wrote down exactly how he did it and passed the instructions on to the team.
00:49So when they encounter a problem like this, they don't need to solve it from scratch, right?
00:56Now imagine this happening with every complex problem across your entire company, and automatically every time.
01:03This is a dream, right?
01:06But it almost never works that way.
01:11Even when you solve a very complex problem, often the knowledge, all the tuning, and all the reasoning just disappears.
01:19Oh, sorry.
01:22All of these considerations simply disappear when you move on to the next task or close the session.
01:29But I want to convince you that this is actually possible and we already have all the necessary elements for this.
01:37If this sounds interesting, let me introduce myself again.
01:40I am Rafael.
01:41I am a founding engineer at Runlayer.
01:43We create a "golden path" for implementing AI in your company.
01:47Before that, I managed Zapier agents and various AI teams, but in general, my previous work was about making agents not just flashy demos, but reliable tools in production.
02:00Before we get to the interesting parts, I want to break my topic into three separate parts and start simply with improving agents.
02:07There are many ways to do this.
02:09We can use better models, better tools for prompts, environments, etc., but today I want to focus only on skills because they are very accessible, right?
02:19At least in theory.
02:21To make sure we understand each other, for me a skill is an instruction that an agent can read when they feel that knowledge is relevant.
02:31Technically, this term is called progressive disclosure, and it means that at the very beginning, the system or client only sees a brief summary or description of the skill.
02:43But when an agent understands that the knowledge inside a skill can be useful, it can read it in its entirety, add it to the main context, and change the trajectory of its actions based on this knowledge.
02:55Why is this so important?
02:56Well, unless you've been living in isolation for the last 6–12 months, your workload has probably changed significantly, and that's because agents have gotten a lot better, or rather— models have gotten a lot better.
03:09And it's not just a feeling, it can be measured.
03:12Here you see a graph that you've probably seen too many times.
03:17For those who don't know, this is the MEETER benchmark.
03:20It measures how long agents can work independently while still maintaining a decent chance of completing the task.
03:27And the word "chance" is key here.
03:30We'll come back to this in a moment, but the trend is clear.
03:34Advanced capabilities, advanced intelligence are getting better and better.
03:38They do better with multi-step deep work.
03:41There is only one caveat: unless this advanced intelligence is taken away from you.
03:45And I am as prone to mistakes as you probably are.
03:49But there is a nuance.
03:51With great power or long-term tasks comes great responsibility.
03:56If an agent can reason for much longer, it also means that it can go down a very, very deep rabbit hole.
04:04He can spend hundreds of tool calls trying to defend a false thesis, argue for hours, and the next agent will do exactly the same thing, or perhaps even worse.
04:16Therefore, the initial trajectory is of great importance.
04:18And that's why skills are so important, because they direct the agent to the right goal.
04:24Otherwise, you will waste time, waste tokens, and most importantly, waste money.
04:31Valuable skills are those rituals that no one wants to rediscover.
04:35This knowledge is often not something exciting.
04:37They are probably deeply ingrained in your company.
04:41And what if agents were trying to discover this knowledge from scratch—for example, the sequence of scripts and function switches to create a good roadmap, or the exact wording to communicate with that picky
04:53corporate client who is about to leave you.
04:58Yes, if agents tried to figure this out from scratch, they would most likely fail.
05:03And often this failure can be very costly.
05:06This is why I believe so many AI implementations or pilots fail: you don't provide agents with the appropriate context and instructions on how to act within your company.
05:17So, skills are great, but I think they're not without their problems.
05:21There are three major issues, and the first one is that they are local and developer-centric.
05:28What I mean is: when you look at the skills ecosystem, you'll find markdown files, JSON files, CLI commands, and Git repositories.
05:38And if you're a developer, that's great.
05:41But if you’re a VP of AI or an AI implementation lead, you need to be more empathetic about how to roll out an AI strategy across the company, including non-technical
05:53people, right?
05:56And the main challenge here is how non-technical people can create and distribute these skills.
06:01Oh, I love this meme.
06:04This is a junior developer or non-technical professional who discovers the NPX skills command— the most important and popular command for installing them.
06:14This is the only command I know that installs not just a skill, but also free prompt injections, a wild script with root privileges, and someone else's API key—all in one run.
06:27Believe me, at RunLayer we scan thousands of skills every day, and some really weird things happen there.
06:35Okay, the second problem: I think we all like the situation when we are faced with a difficult problem and we already have a ready-made plan of action on how to solve it, right?
06:45But often there is simply no such plan.
06:49Because it takes time, effort, and energy to understand the problem and leave something useful for posterity.
06:59And the third problem is that not every LLM client supports the skills, right?
07:04If you're in the cloud ecosystem, then everything is probably great.
07:07If you're a developer, you'll likely be fine, even if the skills support is weird, depending on which client you're using.
07:14And again, if you think about marketing, HR, legal, finance —these departments are probably still using ChatGPT or some other LLM client where the skills may not be supported.
07:25There was a big heated discussion a while ago when these skills first emerged.
07:32Some have proclaimed: thank god MCP is finally dead, because I don't want to mess with OAuth.
07:38Now I can "hardcode " a bunch of CLI commands, write my API key right into a markdown file, and call it progress, right?
07:45But, in my opinion, everyone missed the point.
07:48Skills can convey knowledge about how to do something , while tools give us capabilities.
07:54The tools with MCP the ability to do these things, but MCP can also be used as a protocol for transferring knowledge, for transferring skills.
08:03We can use MCP as a distribution layer.
08:08So, instead of installing skills locally, you can create a remote MCP server that will become the source of truth for your entire company.
08:16Clients can connect to it and search for the skills they need.
08:20The server can enforce policies and ensure that everyone gets the best version of the skill based on the request and , for example, the assigned role.
08:29We've been doing this internally at Run:ai for the last 6 months and it's working pretty well.
08:34But this is no longer just an internal matter.
08:37We see the broader MCP ecosystem moving in this direction as well.
08:42There is a working group called “Skills over MCP,” and while the exact API is not yet final, the direction is clear.
08:49MCP can be used to provide skills and spread knowledge throughout your company.
08:57Yes, and although the official specification leans towards using MCP resources, resource support is still patchy.
09:04Therefore, we focused on tools.
09:10And if you remember the first and third problems I mentioned— developer focus and lack of customer support— then MCP as a distribution layer solves that because it is unified and works in almost every client, not just development ones.
09:28So the problem of skills dissemination among non-technical professionals is essentially solved, right?
09:33Additionally, you now have a centralized API.
09:36So as a VP of AI or AI implementation leader, you have a single place to manage and control what knowledge is shared across your company.
09:47Okay, now we know how to improve agents because we have the skills.
09:51We know we can use MCP to spread these skills to all clients and agents, but let’s talk about something more interesting: how agents can improve themselves.
10:02And every time I see a new article or framework on the topic, I always go back to the work on Voyager.
10:12Unfortunately, you can't see the video, but there is a Minecraft game playing in the background .
10:15In 2023, right after the release of GPT-4, that is, a long time ago, the question arose: how long can multiple agents or systems explore the Minecraft space?
10:28They placed the agent in Minecraft without any instructions.
10:31He set goals for himself .
10:33It just kept getting better without even changing the model weights or prompts.
10:38And in 2023, I think it was a real breakthrough.
10:42How did they do it?
10:45The main thing to pay attention to here is the skills library.
10:49So Voyager didn't just thoroughly explore Minecraft space and then discard the data it received.
10:56When an agent discovered something new, it stored a recipe in the form of code consisting of calls to the tools that led it to the discovery.
11:06So later, when the agent decided, "Hey, I need to create a workbench or a diamond pickaxe."
11:12He didn't have to invent all this knowledge from scratch.
11:16He could simply go to the skill library and get it as code —as a recipe that could be reused.
11:23And this is actually very important.
11:25The main idea I want to convey is that research should be transformed into reusable procedural knowledge.
11:35And we started doing something very similar in Run Layer.
11:40We let the agents do the real work, and when the run is successful, or when there's something interesting in it— like a few tool calls—we just take that trace and ask Frontier LLM to highlight a skill.
11:57While it's extremely easy to isolate a skill from a single run , I believe that we, like agents, learn from mistakes.
12:05True improvement occurs precisely when an agent fails.
12:09So, a launch that didn't go as planned is actually the richest source of information, because it reveals missing fuses, libraries, or some edge cases that you didn't think about beforehand.
12:21And each launch, successful or not, fuels our distillation mechanism, constantly refining our skills.
12:30Successes make skills more reliable, because if the agent achieved the goal , it means "Hey, this skill works."
12:38And if not, it makes the skill more sustainable, because now we know what exactly was missing here.
12:43And all this happens completely autonomously and asynchronously, without human intervention.
12:48The initial results were excellent.
12:52We were tasked with running a Chromium fork inside AWS Lambda using Sonnet 4.6 .
13:00It's not the most intelligent model, but it has sometimes succeeded.
13:05However, the 36% success rate was not ready for industrial use.
13:11So we enabled the agent self-improvement feature , and during one of the runs, the agent found a solution, and our skill distiller created a reusable algorithm.
13:23It was included in all future launches, and the success rate increased to 100%.
13:29So yeah, that's great.
13:31It's like a developer being more concerned with documentation.
13:36Um, success rate is the easiest to show, but a good skill is narrowing the search space.
13:42This forces the agent to Gm, it narrows the search space, and the agent stops making tool calls, exploring fewer branches.
13:53This saves tokens and reduces costs.
14:00But, putting aside economy and precision, I think there's a deeper reason why skills are worth singling out .
14:06Um, the problem is that advanced intelligence is a rented or borrowed resource.
14:11He may disappear.
14:13I think, yes, we all miss Fable.
14:15It was a great model, wasn't it?
14:17It may become outdated or it may become stupider.
14:19We complain every day on Twitter or Slack that Opus is really stupid these days, right?
14:25So while we have access to the best version of this intelligence, we should use it to discover and preserve useful routines.
14:33We should create a catalog of how work is done in our company, as a collective subconscious.
14:39And how do we do it?
14:44Well, I believe we already have all the pieces of the puzzle.
14:49Um, we have skills as a way to preserve procedures, improve agents, and manage models.
14:54We have MCP as the only distribution level for these skills.
14:59And we also have a central repository, a knowledge base for all these skills, which is constantly updated and supplemented with new data.
15:07Um, yes, we also have a continuous learning flywheel for a single agent, but what's interesting is what I promised you at the very beginning—the network effect.
15:17Um, so what happens if you multiply that by every agent and every customer in the entire company.
15:23At Runway AI, we call this the organizational flywheel of self-improvement.
15:29Um, we're grouping a number of similar launches and trying to isolate the skill.
15:34It goes through our numerous checks, like whether it contains any prompt injections or PII, um, and a lot of other things.
15:42When it's ready, it goes into our skills library and, through our MCD gateway, is made available to every customer and agent in the company.
15:51As a result, we have a knowledge base that is constantly evolving, always relevant, and continuously updated with new information and new edge cases.
16:01So once one agent solves a very complex problem, it becomes a skill that is stored in the shared organizational library.
16:13So when someone else encounters the same problem in the future, they won't have to find a solution from scratch.
16:19They don't necessarily need to know, uh , the exact sequence of those magic commands that will restart the database or roll back production.
16:27They can just, um, refer to the playbook they have.
16:32And why is this so important?
16:36Because I think 50% of us have probably already had an existential crisis because of these models, and the thoughts boil down to , uh, models becoming a commodity .
16:48Everyone here has access to them .
16:50Um, the cost of software is approaching zero.
16:55Anyone can replicate my product, um, at least on the surface.
16:59In fact, so many people work at the show today and think, " Hey, everyone's building the same thing , right?"
17:06So, what is my "moat"?
17:09Um, and I believe that this " moat" could be about creating this kind of flywheel and accumulating knowledge through it, because it's like your own hard-earned lessons, systematized and aligned with how your company works, you
17:23know?
17:24It's captured, it's accumulated, raising the bar for everyone in the company, giving them access to the best knowledge available.
17:34Um, it's also a way to become resistant to model intelligence obsolescence, and it's a great way to reduce costs without even changing the model.
17:42So, yes.
17:44This is perhaps the only thing a competitor will not be able to copy—your internal culture and identity.
17:54And yes, that's practically all.
17:56Um, thanks for coming.
17:58If this resonated with you, I am Raphael.
18:01I work at Front Layer.
18:02We are developing for you the " golden path" of using AI.
18:06I'm open to talking about skills, MCP agents, and more.
18:09And that's exactly what you build in Front Layer.
18:12Thank you.