Every Harness Will Become A Claw — Sam Bhagwat, Mastra

AI Engineer · 15 min · 183 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12I'll get us started.
00:14Um, so long day of talks and uh, how are you all feeling?
00:20Cool.
00:22Yeah, good to see we still have some energy.
00:24You know, I know there's a lot of like evening events.
00:26Um, we've heard a lot about the present and I'm going to talk about the future.
00:30Um, so my talk is called every harness will become a claw.
00:34Um, here's a little bit about me.
00:37Um, I am the co-founder CEO of Mastra.
00:40We are a Typescript agent framework.
00:42Um, I am also the author of a book that you may have gotten a copy of either outside or at a previous event.
00:48Um, we have seen a lot of agents running in production um, over the last 18 months.
00:54And I'm not I'm saying that as kind of context for and stage setting for the thoughts and ideas that I'm about to share right now.
01:03Um and uh the thing that I'm going to say is is welcome to the harness era.
01:10What do I mean by the harness era?
01:12Um well u let's just talk about the types of harnesses that we see right now.
01:16We see local harnesses.
01:18Um we use them every day daily driving our our coding, right?
01:22Um we see cloud harnesses.
01:24Um these are both products that we can purchase as well as if we work and some of these uh companies that have built their own internal coding agents that live on Slack.
01:33Um and then of course we have the your friendly local uh open source frameworks that have some of these primitives and give you the the tools that you need to build your own.
01:42Um and that's where we that's where we fit in.
01:45Um now let's talk about where we are sort of collectively as an industry um and how things have evolved over the last we'll say year to to 18 months.
01:55Um last year at at AI engineer we were talking a lot about agents.
01:59We're talking about the agent loop.
02:00We were talking about agents versus workflows.
02:03Um so so I want to you know there's as we're thinking about um the agentic spectrum I often compare it to uh self-driving as a spectrum right there are different levels of self-driving autonomy whether that's like lane assist whether that's Tesla S FSD
02:18whether that's I I'm sitting in the back of my Whimo and there's nobody behind the steering wheel right um there are various aspects of the agentic spectrum between LLMs
02:28agents harnesses and claws and I'm going to talk about what we've seen and where we're going.
02:35What makes an agent different than an LM?
02:37Hopefully, we mostly know this, but just as a quick refresher, right?
02:40It's the agent loop, its tool calls, its memory, it's the ability to retry failed tasks, it's context engineering.
02:47Um, Dex is a close friend and an inspiration for this uh one of the inspirations for this talk.
02:52Um, and it's agent state, right?
02:54These are some things that like hey I'm running an agent in a loop and I can't just do this with a oneshot call uh to to an LLM right I didn't
03:04I've tried to make these qualities I don't not sure what the quality of taking actions is active or something so I just put action but um you know qualities here are starting to emerge when we move from
03:15an agent to a harness durability and doggedness um a friend of mine was referring to an agent that he was using and he called it dogged which I really like and I'm taking that for this talk, right?
03:25So, durability just the sheer quality of like being able to run not for minutes but for hours or days.
03:32Um, you know, what what what encompasses this?
03:35Well, sometimes it's like, hey, I you know, I uh lost a connection in the middle of the turn and uh you know, but I persisted the stream and so now I can resume from the place where I started, right?
03:45There's planning mode.
03:46We all see this in cloud code.
03:48Um parallel sub aents being able to fan out multiple tasks at the same time.
03:53Uh we have more affordances with a TUI and slash commands.
03:57We have skills.
03:58Um we don't have to define all our agents up front, but we can dynamically create them on the fly.
04:03This is, you know, very powerful.
04:05Um you can sp the the harness can spin up background bash tasks, right?
04:11Um it will autocompact when it runs out of the context window.
04:14Uh you know, these are all things we'll see when we use cloud code or codecs, right?
04:20Um it persists.
04:21it will persist threads, right?
04:23You can resume a thread once that's you've like disconnected from later.
04:27Um, you can cue, you can steer, you can interrupt.
04:29You're not just blocked waiting on the LM.
04:32Hey, I take a turn and then you take a turn.
04:34I'm playing playing Civilization here and I can't take a turn until all the other civilizations are playing.
04:38No, I'm playing Starcraft.
04:39I'm playing Age of Empires.
04:40And um, right.
04:41Uh, you know, session long um tool approval.
04:45So it's not just like yeah I approved this specific tool call but yeah you can run all instances of rmrf for slash right that you see in the session
04:54even though the first one will probably wipe your machine.
04:56Um okay so there there's like you know there's a um there's a few steps here and I'm I'm I'm about halfway through these and then afterwards we're going to talk about what it means and this is kind of a
05:10in between step.
05:11I think this is something we've seen over the last really 3 months.
05:16Um, and I think we're all still starting to grapple with what it means, which is this movement from a local harness to a cloud harness where the harness is always on.
05:26What do I mean by a harness that is always on?
05:29Well, you might be talking to it in Slack.
05:33Maybe you're talking to it in Slack along with your colleagues, right?
05:36Um maybe you're each giving it instructions and has to figure out how to parse that and use user metadata.
05:43Uh maybe you have a mobile app.
05:44I was just uh uh you know maybe you have a mobile app.
05:48Maybe it tunnels to your local um to to your local machine.
05:51Some of these uh some harness mobile apps do this.
05:55Um often like cool.
05:58How is this running?
05:58Well, it's probably running in a cloud sandbox because it's maybe it's running locally in your machine.
06:03and you're tunneling into it, but maybe it's just running in a cloud in the cloud somewhere and it's got a bunch of sandboxes which enables more parallelism.
06:09You can get more um parallel sub aents beyond what you can do on your machine.
06:14This is always a trade-off and always something you get with distributed systems, right?
06:18You can do more in the cloud than you can do locally.
06:20You have more resources.
06:21It requires a different architecture.
06:22It's more powerful.
06:24Um and then lastly, you're not creating code, you know, on your just on your local machine or maybe even in a git work tree.
06:30um you're you're probably creating, you know, if you're writing code, you're probably creating a PR that that pushes right to to GitHub.
06:37Um so so you know, there's a shift, right, from from local harnesses to these always on kind of like cloud harnesses.
06:44We're still in the middle of this.
06:46You may have you may only be working with a local harness.
06:50You may have started to see cloud harnesses pop up in your your organization, right?
06:54You may be figuring out how to use them.
06:56Your teams may be figuring out how to use them.
06:58Um, and then I want to talk about what the harness to claw transition is, which is imbuing these agents, imbuing these harnesses with initiative and and learning, right?
07:10What is initiative?
07:11Well, um, if you've used, let's say, a a personal assistant uh, agent, right?
07:17And that agent texts you and says, "Hey, I saw an urgent email come in.
07:22Is that email actually urgent?
07:23Was someone like, you know, spamming you?"
07:25you know, but like like the agent is listening um to external feed services.
07:30It has a heartbeat which means it wakes up every you know defined amount of time and um and does something right.
07:38Uh again like channels some uh you might be able to text it, WhatsApp it, telegram it where whatever you want.
07:44Um you you may persist the memory memory in a more accessible later place than just simple sort of like file storage, right?
07:54um you you might it might have a a Damon, it might have a a gateway uh for for sending in and receiving incoming outcoming requests.
08:02Uh it often will do continual learning, right?
08:06So this concept that you know the agent the harness runs and then you know based on the traces that it generates it it sort of autoimproves itself and there's different ways of doing this.
08:18you see um skill automatic skill generation for example is a common one.
08:23Um it could modify the code driving this as well.
08:26Um we haven't figured out what the right way of doing it is yet.
08:30We're still exploring you know the industry is still exploring options.
08:34Um now the reason that and and maybe this is like our unique vantage point here but you know for the last three months as a framework we've just seen this as a f as the future.
08:46And so we furiously looked at the the you know the features that you know openclaw have that Hermes agent have and say and and we we've said like look you know a lot of people a lot of folks want these features but they want them with power and control.
08:59They don't want to just put a you know a claw on a box right they want to have more.
09:03And so, you know, we we've been thinking about this because we we you know, my I'm not doing my job well if I'm not giving everybody the tools that they need to build agents, to build harnesses as with the maximum power,
09:17right?
09:18Um so, so hopefully like again we hopefully I've walked a little bit through the step transition with actions, durability, doggedness, always on initiative, learning.
09:29Again, I I think I failed in like making them all the right tense phrase and making them all qualities, but I hope you get the idea here, right?
09:36Um, we're ascending on the agentic spectrum.
09:39Um, and what was a simple LLM 18 or 24 months ago is a lot more powerful.
09:44So, I've called this without sort of asking consent from Pete, but I've called this Steinberger's law, which is I I believe every harness will expand until it becomes a claw.
09:56and and and that's a little bit um technological, that's a little bit economic, that's a little bit psychological.
10:03So, let me walk you through the reasoning here.
10:05Um the first thing that I've observed um that we've all observed um as a is that harnesses tend to expand.
10:14And they expand because we want them to expand.
10:17Um we want to DM them in Slack.
10:20We want to text them and like start overnight uh tasks before bedtime.
10:25We want this dopamine casino that we get when we put in tokens and get out code, right?
10:31Um or or whatever other actions, you know, um agent agents are bigger than just coding agents, but um we want our own dopamine casino.
10:40And this this image is thanks to uh to Dex Horty.
10:43Um but I see something else in our future.
10:49um which is that and and it's something that like I don't think we we sort of talk about as much uh but after this after this phase where where we're sort of making everything more and more powerful
11:04um there will be a shakeout um and and let me walk you through sort of uh through my reasoning here which is that in the 2010s we had these platforms we had Android, we had iOS.
11:20And all of a sudden, there were all these things we could do on our phones that we previously weren't able to do.
11:26We could get directions, we could hail rides, um, we could send payments.
11:31Um, we could play music.
11:33Um, other ones emerged over the course of the decade.
11:36We could watch short form video.
11:38Um, we could [clears throat] watch long form documentaries.
11:41We could browse the internet.
11:42You know, some were kind of ported over from the desktop.
11:44We could browse the internet.
11:46Um, again, you know, some, you know, we could order food, right?
11:49Um, we could put book accommodations, but but if you look at most of these kinds of categories, and there are quite a few categories, there really only like one or two, you know, logos here that we use, you know, okay, how many
12:03maybe, you know, we use Uber and we we use Lyft, but like do anyone use another rideing app here, you know, like and and so when you talk to people that are smart about like consumer
12:17behavior, the the The reason they say that this is is because you only really have space in your brain for like a limited number of things.
12:28Like if if think about something like Thumbtac.
12:31So uh Thumbtac didn't really serve a very high economic value.
12:36Like Airbnb like we only use it Airbnb occasionally, but when we use it, we like really want it.
12:40We really need it.
12:41You know, it's really valuable to us.
12:43Thumbtac like a little bit less so, right?
12:45Um and then it's also like not frequent, right?
12:48like maybe maybe like you know something like Door Dash or Uber people can use multiple times a day, right?
12:54So there's there's sort of like it either has to be very economically valuable or has to be very frequent.
12:59And if it's neither one of the two, um we just forget about it, right?
13:03It's like that, you know, college friend that like we haven't really talked to in years.
13:08It's not cuz like they weren't important at one point in our life, but like there's just nothing that maybe they moved to a new city or we moved to a new city or our lives, our friend groups, our careers diverged
13:18and all of a sudden like, you know, maybe we're calling them once a once a year or once every other year or or whatever.
13:23And you know, there's just nothing that makes them pertinent and brings them up in our our brains.
13:30And so right now we're like really excited because there's all this energy and excitement and we're all excited to these harnesses that we're like, you know, putting in tokens and getting out like useful things that we all love.
13:40Um and and I think that in the notsodistant future there will be this very real shakeout and and these categories will kind of emerge and we'll realize that we only have space in our lives
13:51for so many of these claws.
13:54They're very powerful.
13:55We we love them very much.
13:57Um and so I would I would think about um what what what does that mean for you?
14:04So the the first thing is um the first thing is don't get this is the reason that like events are are are important that like staying up with like if the rate of change increases
14:163 to 4x that means you know we need to figure out what's going on even more frequently.
14:21That's why we're all here.
14:23Um, but if you're building an agent, make sure that it has the capabilities that your users need because if it doesn't and if there's newer things that come out, like they may just,
14:32you know, pick pick something that's more powerful because that that's happening very quickly.
14:36Um, and then keep in mind that if you if you if you aren't if you're the thing you're working on, um, if even if you climb up to the top of the hill,
14:46keep in mind there's going to be another wave of this sort of these like this this shakeout coming.
14:52and you know probably sometime in the later 2020s.
14:54Uh so um that I'm Sam um I'm the uh I'm the co-founder of Monsterra the the TypeScript agent framework.
15:03I'm the author of Principles of Building AI agents.
15:05Hopefully you can get a copy of the book outside or I've got a few here.
15:09Um please stop by, say hi.
15:11Um it's great to see all of you.
15:13Thank you all for coming out.
15:14It's a real pleasure.
15:15Um enjoy the rest of the conference.
15:32[music]