We're the bottleneck, but we don't have to be — Ido Salomon, AgentCraft

AI Engineer · 14 min · 140 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:21Okay.
00:21Hi everyone.
00:22Uh I'm Solomon.
00:23I am the creator of ATMCraft and I also created MCPUI which became MCP apps and MCP steering committee.
00:31I've done a bunch of other stuff.
00:33Uh but we're here to talk about why we're the bottleneck and why we don't have to be.
00:40So, uh as you all know and use every day, agents are amazing.
00:44There's so much they can do uh from generating cat memes to B2B SAS apps.
00:51So if they are so amazing, we have to ask why aren't we all unstoppable?
00:55Why doesn't everyone have an army of agents that just does whatever they want?
01:00So it should be relatively easy.
01:02I mean you just have a bunch of monitors and you spin up 25 cloud code agents and you're done.
01:09So the reality is that it's just not that simple to do something like that.
01:14you need to a really want to use agents and secondly once you actually spin them up each one of these agents requires you to steer them direct them review them
01:26and when it's at scale especially it's just exhausting we need to find the skills to allow us to do it without burning out so in reality we are the bottleneck
01:38and the the cool thing here is that while we are the bottleneck The skills that we need to do these things to uh find ourselves not burning out every time we use an agent
01:51are with us all along.
01:52Uh if you try to compare it to other stuff that you've been doing outside of programming, for example, playing video games, it's not really that different.
02:02Like if if you're playing Warcraft, for example, or Sims, and you have a bunch of different uh units running around, you need to understand what they're doing and supervise all of it.
02:11How different is that from from actually inspecting agents?
02:15So I took this concept uh too far and I created agent uh which is a game inspired orchestrator uh that tries to bring all that good stuff together into work.
02:30So let's take uh a quick look of what that actually looks like and try and understand the journey that we can go through to really raise the ceiling of how humans and agents can collaborate.
02:41So, uh, this is AgentCraft.
02:43Uh, there's really a lot to unpack here.
02:46Uh, so we'll just go through the basics really quick.
02:49Um, what you see here is an agent.
02:52Uh, it's an actual representation of a cloud code or openclaw, whatever agent that you can imagine.
02:59Uh, it can be detected on your device or you can actually spawn new ones directly from a craft.
03:06Uh, so like I said, you have like codecs and uh, open code etc. at your command.
03:13Uh, and these agents can do everything.
03:16They're just normal agents.
03:17Uh, so I can just prompt them in this nice side panel that also has voice.
03:22So there's multimodality here.
03:24Uh, and just do something for me.
03:27Prompt.
03:28Prompt.
03:29And now these agents can do whatever.
03:33Uh, and if you look around you would see that the map has a a bunch of different elements.
03:37So you have buildings.
03:38These buildings represent functionalities.
03:40So for example, if you can manage your plugins or skills, uh you have like integrated terminal, integrated git, uh you can do in your all of your work from within this space.
03:52So now that we have this and we understand the basics, let's look at what we can do uh to raise the ceiling.
03:59So the first step is obviously visibility.
04:01I need to understand what's going on.
04:03What are my agents doing?
04:04So, I have this nice side panel and it shows me the different agents and what their task is and what the last thing they did is and what they're currently doing.
04:12Everything I need to quickly understand who needs my attention and who doesn't.
04:17The second part of it is we're not just limited to this.
04:20We have a map.
04:22So, I can take let's say my file system, project it onto the map, and now I can see exactly what my agents are doing.
04:30Not just by looking at a list of stuff.
04:32I can actually see okay this agent is now working on this directory or this file and these files are represented as runes on the map.
04:39So I can really look at each one and see exactly what each agent did and when.
04:45And now I can take this nice information because we are the orchestrator uh and use that to create linkages and now I can create cool stuff like heat maps.
04:56So I can really easily see what's going on.
04:59And the the other aspect is that now that we have this visibility, that's not really enough.
05:06We need to be able to react to what's going on.
05:08If someone needs my attention, I need to be able to quickly jump to it.
05:12So much like video games or RTS games or Civilization, whatever, you would click uh spacebar and it would just take you to the nearest thing that actually needed your attention.
05:22So you can really easily understand what's going on.
05:25and jump from one thing to the other and answer questions or approve plans, etc. That was nice.
05:34I mean, it's a nice step just to have that visibility, but it's not really enough.
05:40It's pretty exhausting to keep looking at what the agents are doing in the granularity of what files are they using and what tools are they using at the moment.
05:49So, we need something else.
05:51Uh and the reason is that we have like two two main problems here.
05:55Uh one is that there's just too much to hold in our heads.
05:58Even if I can understand what's going on and I can see everything, I can't really do like 20 things at the same time.
06:03I can't even think of the next thing I want to do if I have this, you know, huge coding project or whatever.
06:10So I need to find some way to fix that.
06:12Uh so we can I mean we can tell the agents uh you know go through my codebase, find out what the next things I need to do are, create these tasks for me and I just click a button, I accept the quest and there's some agent that does it for me
06:26which is fun.
06:27Uh it's cool.
06:28Uh but the next problem here is that okay now I have 20 stuff going at the same time and I have to babysit each and every one and that is completely exhausting.
06:38So you could ask okay we have agents agents are amazing as we've established so I can just go ahead and tell the agent to do it for me now it's called software factories but at the time it was just doing stuff autonomously
06:53uh so you can give it like a general sense of what you want it would spawn an orchestrator and break it down for you and run it all in well in this case it's a local
07:02uh container so everything is just done in isolation and I don't need to do any babysitting the agent does it for me.
07:10On top of that, you have loops, which everyone really likes right now.
07:14So, you can do cool things like just tell it to, I don't know, scan Twitter for cool stuff you want to build or scan your GitHub, and it will just create these nice
07:23uh features for you in the background without you needing to do too much up front.
07:29The bad thing is that I mean this is nice uh but what do I do when I have again I don't know I would say 20 but even five things to review
07:39at the same time.
07:41Uh it's very very hard to get that out of the way.
07:45So what I did is create uh this like review kit.
07:49So I get the entire changes I need and I get uh what files changed if you want to go like file by file or you can get just get visual evidence.
07:57So you can actually see videos of what changed or photos of what changed so you can really quickly understand what you want or even run a few uh instances in parallel and just pick the best implementation
08:08out of these.
08:11So now that we have this we have uh visibility and we have autonomy uh it's pretty cool like we reached a pretty good place in raising the ceiling of what we can do with agents
08:21but it's still pretty exhausting.
08:24I mean it's it was fun.
08:26Uh, but now I have to ask myself, do I really need to be the only one doing this?
08:31It's pretty lonely to just talk to agents all day and trying to and you know, if you look at uh what a team does, for example, if it's just me, that's fine.
08:40But if I have a team, I might not even be the right person to review what's going on.
08:45So, I need something else.
08:47Uh, and that something is collaboration.
08:50We can just have other people join us.
08:54So the way that looks in agent craft is that I can create these uh you can call it you know war halls war room whatever uh host them it's like completely local but you can do like tunnels
09:07and people can join you h so I can create a room and let's say I have a product designer in my team in this case uh it's my wife
09:14and I can approve uh her otherwise she'll be mad uh so I can approve her and she does her thing she designs whatever and I as an engineer can not only see that that's that's her
09:28um I can not only see that on my map uh I can actually take that and follow off off of it
09:35prompt prompt
09:36yeah that's just like her um so I can take that design follow up on it implement it while she keeps working uh and the cool thing here is that not only can I hand off or fork and kind of have this shared workspace so I'm not really bound anymore or to Git.
09:54I can use Git, but I'm not really bound to something.
09:56Everyone is working together.
09:57Uh you can see that there's also some kind of chat between not only between the people within the room, but also between agents.
10:06So we can have this really nice notice board of exactly what each person is doing, what each agent doing.
10:11They're all aware of each other and they can collaborate much more closely uh than you normally would be able to do.
10:17Uh so for example, here I can say, okay, I took off this design task.
10:22Uh so now we have oh by the way this is also from devices you can have like mobile and and if you like telegram you can use telegram whatever floats your boat.
10:34So we have visibility and we have autonomy and we have collaboration so we managed to raise the ceiling.
10:40Uh it's now possible especially if you're power users to do really cool stuff and use more agents and do more things.
10:48But the interesting thing here was, you know, if I looked at the feedback I got, it's not just that power users were able to do more.
10:55The cooler thing for me was that people that had even nothing to do with agents or nothing to do with productivity, uh, come and say, uh, okay, my kids really loved it and they use it all the time to to orchestrate agents, which they didn't do until now.
11:10uh or you know someone that flunked out of college because he played Starcraft and now you can suddenly do really productive stuff with it.
11:18Uh and a bunch of other examples of kind of bringing communities that so far were kind of behind into this new world of productivity.
11:27So we need to raise the ceiling but we also need to uh lower the floor.
11:34We need to have some kind of way to take no probably around 90% of the people in the world and bring them into this hygienic future without having them burn out.
11:45So again, I took it too far.
11:47Um, and there's this uh currently named TBD um potentially loopers.
11:54Uh the idea here is okay why don't why do we need something like you know Warcraft which is potentially very complicated uh especially or might be even daunting if you're not into the granularity of what files are there are or how uh complex the UI is and kind of try to find
12:10a lower denominator of what makes stuff accessible especially now that agents are much more capable so instead of granularity or having like exactly what files they're working on maybe I can have it more of a mobile
12:24game level uh have projects that they keep returning to and use loops and other stuff that rely on agents being more autonomous to have like a very basic level of I prompt I get the result I don't really care perhaps about the way I can see it if I want to and I have this really nice interactive
12:43and cool experience that makes me want to come back to this and obviously I mean this is one visualization uh it can really resonate with some people but might not might not present to others because we're not all the same and
12:58what may appear like amazing and easy to one person can appear very difficult to another.
13:03Uh but here uh I'm trying to look at a way to make this more customizable.
13:10Try and understand what it is that you as a person need in order to understand and want to use these tools.
13:17So this is a very very early draft.
13:21Um but if you want to use it, please let me know.
13:24So we're the bottleneck.
13:25I think we've established that uh pretty well until now.
13:28But the skills that we need to stop being the bottleneck and enable this new future are already with us.
13:35So we are the bottleneck, but we don't have to be.
13:39And I hope that you all can take what's relevant to you and your life and apply it forward.
13:46So I would ask you for two things.
13:48One, agent is already out there.
13:50Like you can just npx install agent as a website uh and do your thing.
13:55Uh this new thing that I just showed you uh potentially loopers uh is still experimental.
14:02I'm still trying to understand the interest and what people are actually looking for to uh get the floor that they want or the ceiling that they want.
14:11So, please uh go here and express your interest and tell me what you're looking for so I can know what to do with this.
14:18And thank you.
14:33I got it.