WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=u-o0sW9nwmk NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/u-o0sW9nwmk.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.270 --> 00:00:12.918 Hello everyone. s2 00:00:12.918 --> 00:00:14.082 I am Rafael. s3 00:00:14.082 --> 00:00:21.953 Today I want to talk about self-improving agents, but not in a single-player mode where only one agent gets smarter. s4 00:00:21.953 --> 00:00:34.691 I want to discuss how to apply this concept, combined with network effects, to create a company or even an enterprise that self-improves while you sleep. s5 00:00:34.691 --> 00:00:41.877 Imagine the best engineer you've ever worked with, or salesperson, or lawyer. s6 00:00:41.877 --> 00:00:49.555 The one who, every time he solved a really difficult problem, wrote down exactly how he did it and passed the instructions on to the team. s7 00:00:49.555 --> 00:00:56.010 So when they encounter a problem like this, they don't need to solve it from scratch, right? s8 00:00:56.010 --> 00:01:03.835 Now imagine this happening with every complex problem across your entire company, and automatically every time. s9 00:01:03.835 --> 00:01:06.235 This is a dream, right? s10 00:01:06.235 --> 00:01:10.568 But it almost never works that way. s11 00:01:11.687 --> 00:01:19.510 Even when you solve a very complex problem, often the knowledge, all the tuning, and all the reasoning just disappears. s12 00:01:19.510 --> 00:01:22.506 Oh, sorry. s13 00:01:22.506 --> 00:01:29.512 All of these considerations simply disappear when you move on to the next task or close the session. s14 00:01:29.512 --> 00:01:37.001 But I want to convince you that this is actually possible and we already have all the necessary elements for this. s15 00:01:37.001 --> 00:01:40.432 If this sounds interesting, let me introduce myself again. s16 00:01:40.432 --> 00:01:41.467 I am Rafael. s17 00:01:41.467 --> 00:01:43.583 I am a founding engineer at Runlayer. s18 00:01:43.583 --> 00:01:47.551 We create a "golden path" for implementing AI in your company. s19 00:01:47.551 --> 00:01:59.740 Before that, I managed Zapier agents and various AI teams, but in general, my previous work was about making agents not just flashy demos, but reliable tools in production. s20 00:02:00.516 --> 00:02:07.713 Before we get to the interesting parts, I want to break my topic into three separate parts and start simply with improving agents. s21 00:02:07.713 --> 00:02:09.503 There are many ways to do this. s22 00:02:09.503 --> 00:02:19.736 We can use better models, better tools for prompts, environments, etc., but today I want to focus only on skills because they are very accessible, right? s23 00:02:19.736 --> 00:02:21.637 At least in theory. s24 00:02:21.637 --> 00:02:31.827 To make sure we understand each other, for me a skill is an instruction that an agent can read when they feel that knowledge is relevant. s25 00:02:31.827 --> 00:02:43.064 Technically, this term is called progressive disclosure, and it means that at the very beginning, the system or client only sees a brief summary or description of the skill. s26 00:02:43.064 --> 00:02:55.353 But when an agent understands that the knowledge inside a skill can be useful, it can read it in its entirety, add it to the main context, and change the trajectory of its actions based on this knowledge. s27 00:02:55.353 --> 00:02:56.808 Why is this so important? s28 00:02:56.808 --> 00:03:09.061 Well, unless you've been living in isolation for the last 6–12 months, your workload has probably changed significantly, and that's because agents have gotten a lot better, or rather— models have gotten a lot better. s29 00:03:09.061 --> 00:03:12.479 And it's not just a feeling, it can be measured. s30 00:03:12.479 --> 00:03:17.073 Here you see a graph that you've probably seen too many times. s31 00:03:17.073 --> 00:03:20.281 For those who don't know, this is the MEETER benchmark. s32 00:03:20.281 --> 00:03:27.268 It measures how long agents can work independently while still maintaining a decent chance of completing the task. s33 00:03:27.268 --> 00:03:30.281 And the word "chance" is key here. s34 00:03:30.281 --> 00:03:34.259 We'll come back to this in a moment, but the trend is clear. s35 00:03:34.259 --> 00:03:38.011 Advanced capabilities, advanced intelligence are getting better and better. s36 00:03:38.011 --> 00:03:41.134 They do better with multi-step deep work. s37 00:03:41.134 --> 00:03:45.825 There is only one caveat: unless this advanced intelligence is taken away from you. s38 00:03:45.825 --> 00:03:49.974 And I am as prone to mistakes as you probably are. s39 00:03:49.974 --> 00:03:51.809 But there is a nuance. s40 00:03:51.809 --> 00:03:56.553 With great power or long-term tasks comes great responsibility. s41 00:03:56.553 --> 00:04:04.005 If an agent can reason for much longer, it also means that it can go down a very, very deep rabbit hole. s42 00:04:04.005 --> 00:04:16.123 He can spend hundreds of tool calls trying to defend a false thesis, argue for hours, and the next agent will do exactly the same thing, or perhaps even worse. s43 00:04:16.123 --> 00:04:18.847 Therefore, the initial trajectory is of great importance. s44 00:04:18.847 --> 00:04:24.533 And that's why skills are so important, because they direct the agent to the right goal. s45 00:04:24.533 --> 00:04:31.502 Otherwise, you will waste time, waste tokens, and most importantly, waste money. s46 00:04:31.502 --> 00:04:35.349 Valuable skills are those rituals that no one wants to rediscover. s47 00:04:35.349 --> 00:04:37.965 This knowledge is often not something exciting. s48 00:04:37.965 --> 00:04:41.261 They are probably deeply ingrained in your company. s49 00:04:41.261 --> 00:04:53.649 And what if agents were trying to discover this knowledge from scratch—for example, the sequence of scripts and function switches to create a good roadmap, or the exact wording to communicate with that picky s50 00:04:53.649 --> 00:04:57.524 corporate client who is about to leave you. s51 00:04:58.192 --> 00:05:03.827 Yes, if agents tried to figure this out from scratch, they would most likely fail. s52 00:05:03.827 --> 00:05:06.619 And often this failure can be very costly. s53 00:05:06.619 --> 00:05:17.581 This is why I believe so many AI implementations or pilots fail: you don't provide agents with the appropriate context and instructions on how to act within your company. s54 00:05:17.581 --> 00:05:21.636 So, skills are great, but I think they're not without their problems. s55 00:05:21.636 --> 00:05:28.728 There are three major issues, and the first one is that they are local and developer-centric. s56 00:05:28.728 --> 00:05:38.589 What I mean is: when you look at the skills ecosystem, you'll find markdown files, JSON files, CLI commands, and Git repositories. s57 00:05:38.589 --> 00:05:41.237 And if you're a developer, that's great. s58 00:05:41.237 --> 00:05:53.285 But if you’re a VP of AI or an AI implementation lead, you need to be more empathetic about how to roll out an AI strategy across the company, including non-technical s59 00:05:53.285 --> 00:05:56.110 people, right? s60 00:05:56.110 --> 00:06:01.830 And the main challenge here is how non-technical people can create and distribute these skills. s61 00:06:01.830 --> 00:06:04.681 Oh, I love this meme. s62 00:06:04.681 --> 00:06:14.633 This is a junior developer or non-technical professional who discovers the NPX skills command— the most important and popular command for installing them. s63 00:06:14.633 --> 00:06:27.844 This is the only command I know that installs not just a skill, but also free prompt injections, a wild script with root privileges, and someone else's API key—all in one run. s64 00:06:27.844 --> 00:06:35.267 Believe me, at RunLayer we scan thousands of skills every day, and some really weird things happen there. s65 00:06:35.267 --> 00:06:45.757 Okay, the second problem: I think we all like the situation when we are faced with a difficult problem and we already have a ready-made plan of action on how to solve it, right? s66 00:06:45.757 --> 00:06:49.070 But often there is simply no such plan. s67 00:06:49.070 --> 00:06:58.128 Because it takes time, effort, and energy to understand the problem and leave something useful for posterity. s68 00:06:59.323 --> 00:07:04.028 And the third problem is that not every LLM client supports the skills, right? s69 00:07:04.028 --> 00:07:07.176 If you're in the cloud ecosystem, then everything is probably great. s70 00:07:07.176 --> 00:07:14.193 If you're a developer, you'll likely be fine, even if the skills support is weird, depending on which client you're using. s71 00:07:14.193 --> 00:07:25.548 And again, if you think about marketing, HR, legal, finance —these departments are probably still using ChatGPT or some other LLM client where the skills may not be supported. s72 00:07:25.548 --> 00:07:32.490 There was a big heated discussion a while ago when these skills first emerged. s73 00:07:32.490 --> 00:07:38.387 Some have proclaimed: thank god MCP is finally dead, because I don't want to mess with OAuth. s74 00:07:38.387 --> 00:07:45.938 Now I can "hardcode " a bunch of CLI commands, write my API key right into a markdown file, and call it progress, right? s75 00:07:45.938 --> 00:07:48.469 But, in my opinion, everyone missed the point. s76 00:07:48.469 --> 00:07:54.619 Skills can convey knowledge about how to do something , while tools give us capabilities. s77 00:07:54.619 --> 00:08:03.988 The tools with MCP the ability to do these things, but MCP can also be used as a protocol for transferring knowledge, for transferring skills. s78 00:08:03.988 --> 00:08:08.281 We can use MCP as a distribution layer. s79 00:08:08.281 --> 00:08:16.203 So, instead of installing skills locally, you can create a remote MCP server that will become the source of truth for your entire company. s80 00:08:16.203 --> 00:08:20.132 Clients can connect to it and search for the skills they need. s81 00:08:20.132 --> 00:08:29.318 The server can enforce policies and ensure that everyone gets the best version of the skill based on the request and , for example, the assigned role. s82 00:08:29.318 --> 00:08:34.781 We've been doing this internally at Run:ai for the last 6 months and it's working pretty well. s83 00:08:34.781 --> 00:08:37.810 But this is no longer just an internal matter. s84 00:08:37.810 --> 00:08:42.441 We see the broader MCP ecosystem moving in this direction as well. s85 00:08:42.441 --> 00:08:49.588 There is a working group called “Skills over MCP,” and while the exact API is not yet final, the direction is clear. s86 00:08:49.588 --> 00:08:57.597 MCP can be used to provide skills and spread knowledge throughout your company. s87 00:08:57.614 --> 00:09:04.835 Yes, and although the official specification leans towards using MCP resources, resource support is still patchy. s88 00:09:04.835 --> 00:09:08.220 Therefore, we focused on tools. s89 00:09:10.814 --> 00:09:27.552 And if you remember the first and third problems I mentioned— developer focus and lack of customer support— then MCP as a distribution layer solves that because it is unified and works in almost every client, not just development ones. s90 00:09:28.237 --> 00:09:33.459 So the problem of skills dissemination among non-technical professionals is essentially solved, right? s91 00:09:33.459 --> 00:09:36.247 Additionally, you now have a centralized API. s92 00:09:36.247 --> 00:09:47.116 So as a VP of AI or AI implementation leader, you have a single place to manage and control what knowledge is shared across your company. s93 00:09:47.560 --> 00:09:51.981 Okay, now we know how to improve agents because we have the skills. s94 00:09:51.981 --> 00:10:02.924 We know we can use MCP to spread these skills to all clients and agents, but let’s talk about something more interesting: how agents can improve themselves. s95 00:10:02.924 --> 00:10:12.017 And every time I see a new article or framework on the topic, I always go back to the work on Voyager. s96 00:10:12.017 --> 00:10:15.998 Unfortunately, you can't see the video, but there is a Minecraft game playing in the background . s97 00:10:15.998 --> 00:10:28.298 In 2023, right after the release of GPT-4, that is, a long time ago, the question arose: how long can multiple agents or systems explore the Minecraft space? s98 00:10:28.298 --> 00:10:31.815 They placed the agent in Minecraft without any instructions. s99 00:10:31.815 --> 00:10:33.860 He set goals for himself . s100 00:10:33.860 --> 00:10:38.494 It just kept getting better without even changing the model weights or prompts. s101 00:10:38.494 --> 00:10:42.889 And in 2023, I think it was a real breakthrough. s102 00:10:42.889 --> 00:10:45.288 How did they do it? s103 00:10:45.288 --> 00:10:49.586 The main thing to pay attention to here is the skills library. s104 00:10:49.586 --> 00:10:56.477 So Voyager didn't just thoroughly explore Minecraft space and then discard the data it received. s105 00:10:56.477 --> 00:11:06.478 When an agent discovered something new, it stored a recipe in the form of code consisting of calls to the tools that led it to the discovery. s106 00:11:06.969 --> 00:11:12.926 So later, when the agent decided, "Hey, I need to create a workbench or a diamond pickaxe." s107 00:11:12.926 --> 00:11:16.518 He didn't have to invent all this knowledge from scratch. s108 00:11:16.518 --> 00:11:23.270 He could simply go to the skill library and get it as code —as a recipe that could be reused. s109 00:11:23.270 --> 00:11:25.553 And this is actually very important. s110 00:11:25.553 --> 00:11:35.581 The main idea I want to convey is that research should be transformed into reusable procedural knowledge. s111 00:11:35.581 --> 00:11:40.384 And we started doing something very similar in Run Layer. s112 00:11:40.384 --> 00:11:55.418 We let the agents do the real work, and when the run is successful, or when there's something interesting in it— like a few tool calls—we just take that trace and ask Frontier LLM to highlight a skill. s113 00:11:57.355 --> 00:12:05.375 While it's extremely easy to isolate a skill from a single run , I believe that we, like agents, learn from mistakes. s114 00:12:05.375 --> 00:12:09.452 True improvement occurs precisely when an agent fails. s115 00:12:09.452 --> 00:12:21.837 So, a launch that didn't go as planned is actually the richest source of information, because it reveals missing fuses, libraries, or some edge cases that you didn't think about beforehand. s116 00:12:21.837 --> 00:12:30.738 And each launch, successful or not, fuels our distillation mechanism, constantly refining our skills. s117 00:12:30.738 --> 00:12:38.421 Successes make skills more reliable, because if the agent achieved the goal , it means "Hey, this skill works." s118 00:12:38.421 --> 00:12:43.867 And if not, it makes the skill more sustainable, because now we know what exactly was missing here. s119 00:12:43.867 --> 00:12:48.727 And all this happens completely autonomously and asynchronously, without human intervention. s120 00:12:48.727 --> 00:12:52.802 The initial results were excellent. s121 00:12:52.802 --> 00:13:00.775 We were tasked with running a Chromium fork inside AWS Lambda using Sonnet 4.6 . s122 00:13:00.775 --> 00:13:05.376 It's not the most intelligent model, but it has sometimes succeeded. s123 00:13:05.376 --> 00:13:11.119 However, the 36% success rate was not ready for industrial use. s124 00:13:11.119 --> 00:13:23.468 So we enabled the agent self-improvement feature , and during one of the runs, the agent found a solution, and our skill distiller created a reusable algorithm. s125 00:13:23.468 --> 00:13:29.975 It was included in all future launches, and the success rate increased to 100%. s126 00:13:29.975 --> 00:13:31.625 So yeah, that's great. s127 00:13:31.625 --> 00:13:35.957 It's like a developer being more concerned with documentation. s128 00:13:36.559 --> 00:13:42.186 Um, success rate is the easiest to show, but a good skill is narrowing the search space. s129 00:13:42.186 --> 00:13:53.502 This forces the agent to Gm, it narrows the search space, and the agent stops making tool calls, exploring fewer branches. s130 00:13:53.502 --> 00:13:58.232 This saves tokens and reduces costs. s131 00:14:00.259 --> 00:14:06.067 But, putting aside economy and precision, I think there's a deeper reason why skills are worth singling out . s132 00:14:06.067 --> 00:14:11.898 Um, the problem is that advanced intelligence is a rented or borrowed resource. s133 00:14:11.898 --> 00:14:13.048 He may disappear. s134 00:14:13.048 --> 00:14:15.156 I think, yes, we all miss Fable. s135 00:14:15.156 --> 00:14:17.049 It was a great model, wasn't it? s136 00:14:17.049 --> 00:14:19.437 It may become outdated or it may become stupider. s137 00:14:19.437 --> 00:14:25.045 We complain every day on Twitter or Slack that Opus is really stupid these days, right? s138 00:14:25.045 --> 00:14:33.139 So while we have access to the best version of this intelligence, we should use it to discover and preserve useful routines. s139 00:14:33.139 --> 00:14:39.907 We should create a catalog of how work is done in our company, as a collective subconscious. s140 00:14:39.907 --> 00:14:44.216 And how do we do it? s141 00:14:44.216 --> 00:14:49.351 Well, I believe we already have all the pieces of the puzzle. s142 00:14:49.351 --> 00:14:54.816 Um, we have skills as a way to preserve procedures, improve agents, and manage models. s143 00:14:54.816 --> 00:14:59.942 We have MCP as the only distribution level for these skills. s144 00:14:59.942 --> 00:15:07.103 And we also have a central repository, a knowledge base for all these skills, which is constantly updated and supplemented with new data. s145 00:15:07.103 --> 00:15:17.874 Um, yes, we also have a continuous learning flywheel for a single agent, but what's interesting is what I promised you at the very beginning—the network effect. s146 00:15:17.874 --> 00:15:23.944 Um, so what happens if you multiply that by every agent and every customer in the entire company. s147 00:15:23.944 --> 00:15:29.260 At Runway AI, we call this the organizational flywheel of self-improvement. s148 00:15:29.260 --> 00:15:34.064 Um, we're grouping a number of similar launches and trying to isolate the skill. s149 00:15:34.064 --> 00:15:42.757 It goes through our numerous checks, like whether it contains any prompt injections or PII, um, and a lot of other things. s150 00:15:42.757 --> 00:15:51.999 When it's ready, it goes into our skills library and, through our MCD gateway, is made available to every customer and agent in the company. s151 00:15:51.999 --> 00:16:01.667 As a result, we have a knowledge base that is constantly evolving, always relevant, and continuously updated with new information and new edge cases. s152 00:16:01.667 --> 00:16:12.403 So once one agent solves a very complex problem, it becomes a skill that is stored in the shared organizational library. s153 00:16:13.466 --> 00:16:19.862 So when someone else encounters the same problem in the future, they won't have to find a solution from scratch. s154 00:16:19.862 --> 00:16:27.920 They don't necessarily need to know, uh , the exact sequence of those magic commands that will restart the database or roll back production. s155 00:16:27.920 --> 00:16:32.682 They can just, um, refer to the playbook they have. s156 00:16:32.682 --> 00:16:36.701 And why is this so important? s157 00:16:36.701 --> 00:16:48.613 Because I think 50% of us have probably already had an existential crisis because of these models, and the thoughts boil down to , uh, models becoming a commodity . s158 00:16:48.724 --> 00:16:50.297 Everyone here has access to them . s159 00:16:50.297 --> 00:16:55.058 Um, the cost of software is approaching zero. s160 00:16:55.058 --> 00:16:59.038 Anyone can replicate my product, um, at least on the surface. s161 00:16:59.038 --> 00:17:06.146 In fact, so many people work at the show today and think, " Hey, everyone's building the same thing , right?" s162 00:17:06.146 --> 00:17:09.534 So, what is my "moat"? s163 00:17:09.656 --> 00:17:23.367 Um, and I believe that this " moat" could be about creating this kind of flywheel and accumulating knowledge through it, because it's like your own hard-earned lessons, systematized and aligned with how your company works, you s164 00:17:23.367 --> 00:17:24.559 know? s165 00:17:24.559 --> 00:17:34.984 It's captured, it's accumulated, raising the bar for everyone in the company, giving them access to the best knowledge available. s166 00:17:34.984 --> 00:17:42.859 Um, it's also a way to become resistant to model intelligence obsolescence, and it's a great way to reduce costs without even changing the model. s167 00:17:42.859 --> 00:17:44.403 So, yes. s168 00:17:44.403 --> 00:17:54.179 This is perhaps the only thing a competitor will not be able to copy—your internal culture and identity. s169 00:17:54.327 --> 00:17:56.761 And yes, that's practically all. s170 00:17:56.761 --> 00:17:58.699 Um, thanks for coming. s171 00:17:58.699 --> 00:18:01.209 If this resonated with you, I am Raphael. s172 00:18:01.209 --> 00:18:02.916 I work at Front Layer. s173 00:18:02.916 --> 00:18:06.223 We are developing for you the " golden path" of using AI. s174 00:18:06.223 --> 00:18:09.598 I'm open to talking about skills, MCP agents, and more. s175 00:18:09.598 --> 00:18:12.516 And that's exactly what you build in Front Layer. s176 00:18:12.516 --> 00:18:14.974 Thank you.