From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, Runlayer
https://www.youtube.com/watch?v=u-o0sW9nwmk
AI Engineer · 18 min · 176 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:12] Hello everyone.
[00:12] I am Rafael.
[00:14] Today I want to talk about self-improving agents, but not in a single-player mode where only one agent gets smarter.
[00:21] I want to discuss how to apply this concept, combined with network effects, to create a company or even an enterprise that self-improves while you sleep.
[00:34] Imagine the best engineer you've ever worked with, or salesperson, or lawyer.
[00:41] The one who, every time he solved a really difficult problem, wrote down exactly how he did it and passed the instructions on to the team.
[00:49] So when they encounter a problem like this, they don't need to solve it from scratch, right?
[00:56] Now imagine this happening with every complex problem across your entire company, and automatically every time.
[01:03] This is a dream, right?
[01:06] But it almost never works that way.
[01:11] Even when you solve a very complex problem, often the knowledge, all the tuning, and all the reasoning just disappears.
[01:19] Oh, sorry.
[01:22] All of these considerations simply disappear when you move on to the next task or close the session.
[01:29] But I want to convince you that this is actually possible and we already have all the necessary elements for this.
[01:37] If this sounds interesting, let me introduce myself again.
[01:40] I am Rafael.
[01:41] I am a founding engineer at Runlayer.
[01:43] We create a "golden path" for implementing AI in your company.
[01:47] Before that, I managed Zapier agents and various AI teams, but in general, my previous work was about making agents not just flashy demos, but reliable tools in production.
[02:00] Before we get to the interesting parts, I want to break my topic into three separate parts and start simply with improving agents.
[02:07] There are many ways to do this.
[02:09] We can use better models, better tools for prompts, environments, etc., but today I want to focus only on skills because they are very accessible, right?
[02:19] At least in theory.
[02:21] To make sure we understand each other, for me a skill is an instruction that an agent can read when they feel that knowledge is relevant.
[02:31] Technically, this term is called progressive disclosure, and it means that at the very beginning, the system or client only sees a brief summary or description of the skill.
[02:43] But when an agent understands that the knowledge inside a skill can be useful, it can read it in its entirety, add it to the main context, and change the trajectory of its actions based on this knowledge.
[02:55] Why is this so important?
[02:56] Well, unless you've been living in isolation for the last 6–12 months, your workload has probably changed significantly, and that's because agents have gotten a lot better, or rather— models have gotten a lot better.
[03:09] And it's not just a feeling, it can be measured.
[03:12] Here you see a graph that you've probably seen too many times.
[03:17] For those who don't know, this is the MEETER benchmark.
[03:20] It measures how long agents can work independently while still maintaining a decent chance of completing the task.
[03:27] And the word "chance" is key here.
[03:30] We'll come back to this in a moment, but the trend is clear.
[03:34] Advanced capabilities, advanced intelligence are getting better and better.
[03:38] They do better with multi-step deep work.
[03:41] There is only one caveat: unless this advanced intelligence is taken away from you.
[03:45] And I am as prone to mistakes as you probably are.
[03:49] But there is a nuance.
[03:51] With great power or long-term tasks comes great responsibility.
[03:56] If an agent can reason for much longer, it also means that it can go down a very, very deep rabbit hole.
[04:04] He can spend hundreds of tool calls trying to defend a false thesis, argue for hours, and the next agent will do exactly the same thing, or perhaps even worse.
[04:16] Therefore, the initial trajectory is of great importance.
[04:18] And that's why skills are so important, because they direct the agent to the right goal.
[04:24] Otherwise, you will waste time, waste tokens, and most importantly, waste money.
[04:31] Valuable skills are those rituals that no one wants to rediscover.
[04:35] This knowledge is often not something exciting.
[04:37] They are probably deeply ingrained in your company.
[04:41] And what if agents were trying to discover this knowledge from scratch—for example, the sequence of scripts and function switches to create a good roadmap, or the exact wording to communicate with that picky
[04:53] corporate client who is about to leave you.
[04:58] Yes, if agents tried to figure this out from scratch, they would most likely fail.
[05:03] And often this failure can be very costly.
[05:06] This is why I believe so many AI implementations or pilots fail: you don't provide agents with the appropriate context and instructions on how to act within your company.
[05:17] So, skills are great, but I think they're not without their problems.
[05:21] There are three major issues, and the first one is that they are local and developer-centric.
[05:28] What I mean is: when you look at the skills ecosystem, you'll find markdown files, JSON files, CLI commands, and Git repositories.
[05:38] And if you're a developer, that's great.
[05:41] But if you’re a VP of AI or an AI implementation lead, you need to be more empathetic about how to roll out an AI strategy across the company, including non-technical
[05:53] people, right?
[05:56] And the main challenge here is how non-technical people can create and distribute these skills.
[06:01] Oh, I love this meme.
[06:04] This is a junior developer or non-technical professional who discovers the NPX skills command— the most important and popular command for installing them.
[06:14] This is the only command I know that installs not just a skill, but also free prompt injections, a wild script with root privileges, and someone else's API key—all in one run.
[06:27] Believe me, at RunLayer we scan thousands of skills every day, and some really weird things happen there.
[06:35] Okay, the second problem: I think we all like the situation when we are faced with a difficult problem and we already have a ready-made plan of action on how to solve it, right?
[06:45] But often there is simply no such plan.
[06:49] Because it takes time, effort, and energy to understand the problem and leave something useful for posterity.
[06:59] And the third problem is that not every LLM client supports the skills, right?
[07:04] If you're in the cloud ecosystem, then everything is probably great.
[07:07] If you're a developer, you'll likely be fine, even if the skills support is weird, depending on which client you're using.
[07:14] And again, if you think about marketing, HR, legal, finance —these departments are probably still using ChatGPT or some other LLM client where the skills may not be supported.
[07:25] There was a big heated discussion a while ago when these skills first emerged.
[07:32] Some have proclaimed: thank god MCP is finally dead, because I don't want to mess with OAuth.
[07:38] Now I can "hardcode " a bunch of CLI commands, write my API key right into a markdown file, and call it progress, right?
[07:45] But, in my opinion, everyone missed the point.
[07:48] Skills can convey knowledge about how to do something , while tools give us capabilities.
[07:54] The tools with MCP the ability to do these things, but MCP can also be used as a protocol for transferring knowledge, for transferring skills.
[08:03] We can use MCP as a distribution layer.
[08:08] So, instead of installing skills locally, you can create a remote MCP server that will become the source of truth for your entire company.
[08:16] Clients can connect to it and search for the skills they need.
[08:20] The server can enforce policies and ensure that everyone gets the best version of the skill based on the request and , for example, the assigned role.
[08:29] We've been doing this internally at Run:ai for the last 6 months and it's working pretty well.
[08:34] But this is no longer just an internal matter.
[08:37] We see the broader MCP ecosystem moving in this direction as well.
[08:42] There is a working group called “Skills over MCP,” and while the exact API is not yet final, the direction is clear.
[08:49] MCP can be used to provide skills and spread knowledge throughout your company.
[08:57] Yes, and although the official specification leans towards using MCP resources, resource support is still patchy.
[09:04] Therefore, we focused on tools.
[09:10] And if you remember the first and third problems I mentioned— developer focus and lack of customer support— then MCP as a distribution layer solves that because it is unified and works in almost every client, not just development ones.
[09:28] So the problem of skills dissemination among non-technical professionals is essentially solved, right?
[09:33] Additionally, you now have a centralized API.
[09:36] So as a VP of AI or AI implementation leader, you have a single place to manage and control what knowledge is shared across your company.
[09:47] Okay, now we know how to improve agents because we have the skills.
[09:51] We know we can use MCP to spread these skills to all clients and agents, but let’s talk about something more interesting: how agents can improve themselves.
[10:02] And every time I see a new article or framework on the topic, I always go back to the work on Voyager.
[10:12] Unfortunately, you can't see the video, but there is a Minecraft game playing in the background .
[10:15] In 2023, right after the release of GPT-4, that is, a long time ago, the question arose: how long can multiple agents or systems explore the Minecraft space?
[10:28] They placed the agent in Minecraft without any instructions.
[10:31] He set goals for himself .
[10:33] It just kept getting better without even changing the model weights or prompts.
[10:38] And in 2023, I think it was a real breakthrough.
[10:42] How did they do it?
[10:45] The main thing to pay attention to here is the skills library.
[10:49] So Voyager didn't just thoroughly explore Minecraft space and then discard the data it received.
[10:56] When an agent discovered something new, it stored a recipe in the form of code consisting of calls to the tools that led it to the discovery.
[11:06] So later, when the agent decided, "Hey, I need to create a workbench or a diamond pickaxe."
[11:12] He didn't have to invent all this knowledge from scratch.
[11:16] He could simply go to the skill library and get it as code —as a recipe that could be reused.
[11:23] And this is actually very important.
[11:25] The main idea I want to convey is that research should be transformed into reusable procedural knowledge.
[11:35] And we started doing something very similar in Run Layer.
[11:40] We let the agents do the real work, and when the run is successful, or when there's something interesting in it— like a few tool calls—we just take that trace and ask Frontier LLM to highlight a skill.
[11:57] While it's extremely easy to isolate a skill from a single run , I believe that we, like agents, learn from mistakes.
[12:05] True improvement occurs precisely when an agent fails.
[12:09] So, a launch that didn't go as planned is actually the richest source of information, because it reveals missing fuses, libraries, or some edge cases that you didn't think about beforehand.
[12:21] And each launch, successful or not, fuels our distillation mechanism, constantly refining our skills.
[12:30] Successes make skills more reliable, because if the agent achieved the goal , it means "Hey, this skill works."
[12:38] And if not, it makes the skill more sustainable, because now we know what exactly was missing here.
[12:43] And all this happens completely autonomously and asynchronously, without human intervention.
[12:48] The initial results were excellent.
[12:52] We were tasked with running a Chromium fork inside AWS Lambda using Sonnet 4.6 .
[13:00] It's not the most intelligent model, but it has sometimes succeeded.
[13:05] However, the 36% success rate was not ready for industrial use.
[13:11] So we enabled the agent self-improvement feature , and during one of the runs, the agent found a solution, and our skill distiller created a reusable algorithm.
[13:23] It was included in all future launches, and the success rate increased to 100%.
[13:29] So yeah, that's great.
[13:31] It's like a developer being more concerned with documentation.
[13:36] Um, success rate is the easiest to show, but a good skill is narrowing the search space.
[13:42] This forces the agent to Gm, it narrows the search space, and the agent stops making tool calls, exploring fewer branches.
[13:53] This saves tokens and reduces costs.
[14:00] But, putting aside economy and precision, I think there's a deeper reason why skills are worth singling out .
[14:06] Um, the problem is that advanced intelligence is a rented or borrowed resource.
[14:11] He may disappear.
[14:13] I think, yes, we all miss Fable.
[14:15] It was a great model, wasn't it?
[14:17] It may become outdated or it may become stupider.
[14:19] We complain every day on Twitter or Slack that Opus is really stupid these days, right?
[14:25] So while we have access to the best version of this intelligence, we should use it to discover and preserve useful routines.
[14:33] We should create a catalog of how work is done in our company, as a collective subconscious.
[14:39] And how do we do it?
[14:44] Well, I believe we already have all the pieces of the puzzle.
[14:49] Um, we have skills as a way to preserve procedures, improve agents, and manage models.
[14:54] We have MCP as the only distribution level for these skills.
[14:59] And we also have a central repository, a knowledge base for all these skills, which is constantly updated and supplemented with new data.
[15:07] Um, yes, we also have a continuous learning flywheel for a single agent, but what's interesting is what I promised you at the very beginning—the network effect.
[15:17] Um, so what happens if you multiply that by every agent and every customer in the entire company.
[15:23] At Runway AI, we call this the organizational flywheel of self-improvement.
[15:29] Um, we're grouping a number of similar launches and trying to isolate the skill.
[15:34] It goes through our numerous checks, like whether it contains any prompt injections or PII, um, and a lot of other things.
[15:42] When it's ready, it goes into our skills library and, through our MCD gateway, is made available to every customer and agent in the company.
[15:51] As a result, we have a knowledge base that is constantly evolving, always relevant, and continuously updated with new information and new edge cases.
[16:01] So once one agent solves a very complex problem, it becomes a skill that is stored in the shared organizational library.
[16:13] So when someone else encounters the same problem in the future, they won't have to find a solution from scratch.
[16:19] They don't necessarily need to know, uh , the exact sequence of those magic commands that will restart the database or roll back production.
[16:27] They can just, um, refer to the playbook they have.
[16:32] And why is this so important?
[16:36] Because I think 50% of us have probably already had an existential crisis because of these models, and the thoughts boil down to , uh, models becoming a commodity .
[16:48] Everyone here has access to them .
[16:50] Um, the cost of software is approaching zero.
[16:55] Anyone can replicate my product, um, at least on the surface.
[16:59] In fact, so many people work at the show today and think, " Hey, everyone's building the same thing , right?"
[17:06] So, what is my "moat"?
[17:09] Um, and I believe that this " moat" could be about creating this kind of flywheel and accumulating knowledge through it, because it's like your own hard-earned lessons, systematized and aligned with how your company works, you
[17:23] know?
[17:24] It's captured, it's accumulated, raising the bar for everyone in the company, giving them access to the best knowledge available.
[17:34] Um, it's also a way to become resistant to model intelligence obsolescence, and it's a great way to reduce costs without even changing the model.
[17:42] So, yes.
[17:44] This is perhaps the only thing a competitor will not be able to copy—your internal culture and identity.
[17:54] And yes, that's practically all.
[17:56] Um, thanks for coming.
[17:58] If this resonated with you, I am Raphael.
[18:01] I work at Front Layer.
[18:02] We are developing for you the " golden path" of using AI.
[18:06] I'm open to talking about skills, MCP agents, and more.
[18:09] And that's exactly what you build in Front Layer.
[18:12] Thank you.