Orchestras, Not Factories: How the Fastest Builders Work — Charlie Holtz, Conductor

AI Engineer · 17 min · 198 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:25Hello.
00:26I was at a wedding this weekend and it was in New York and there were a bunch of trendy people there and uh I I told I was talking to someone who I'd never met before and
00:38uh I told him I was prepping for this presentation and he was like, "Oh, that's cool.
00:41Like, what what's the conference about?"
00:43And I said it was about AI engineering.
00:46And I could just see his eyes glaze over and he started like looking behind me to find the next person to talk to.
00:52And I'm it's like so cool to actually be in a room full of people who actually want to hear about this stuff.
00:57So, I I I I'm very excited to be here.
01:00Um I am the co-founder of Conductor.
01:02Um has anyone here used Conductor?
01:05Okay, nice.
01:06Nice.
01:07Okay, cool.
01:07So, uh uh for those who don't know, uh Conductor is a desktop app for managing a team of coding agents all at the same time.
01:15So, instead of having a bunch of terminal windows for your Cloud Codes, your Code Accesses, or your uh whatever uh coding agent, you have one interface to manage them all.
01:24And one of the really cool things about building Conductor has been that I've seen a lot of the best builders up close.
01:31I've like watched their workflow, I've seen how they work, I've seen the things they do do and the things that they avoid doing.
01:37And so, I thought I would compile a bunch of the principles um that I've seen the best engineers use um and give them to you all.
01:44Um so, here's what Conductor looks like.
01:48Uh can you can you guys see this?
01:50Okay, nice.
01:51Okay.
01:51So, here's what Conductor looks like.
01:53Um and here are my principles for being the fastest builder in your organization.
02:00So, let's start with number one, stay near the frontier.
02:06Staying near the frontier means you are always trying the latest things, basically the day they come out.
02:11Uh it means that when uh uh ultra code comes out, you're trying it.
02:16It means that when uh uh slash go comes out, you're giving it a go.
02:21Um it's really important to stay near the frontier um for a few reasons.
02:25Um if you are doing your own startup, then staying near the frontier means that you will um come up with lots of new ideas for what you should actually be building.
02:33Um and this literally happened to us.
02:34We were building a totally different app called Chorus.
02:37Um but we started using We were such power users of cloud code back in February of last year that we uh started building our whole workflow around cloud code, and we started cloning our repo five times, and then we discovered work trees, and then bit by bit we had built conductor as an internal tool.
02:54And we couldn't have figured that out if we uh weren't staying near the frontier.
02:58And if you're not doing a startup, you should be the person at your company who always knows what the latest uh the latest workflows are.
03:06Um it used to be that you could just kind of like use your social graph and the information that was important about the best workflows to use would trickle down to you.
03:15But things just like move way too fast now.
03:17You're You're always going to be three to six months behind if you do that.
03:20Um so, it's a very important to stay near the frontier.
03:23And there's an important word here, near.
03:26Um there is a danger if you are at the frontier, um you can you can do what uh I call mid-whip miming, where you're spending all of your time working on your workflow and not doing actual work.
03:38And so, internally we've come up uh with a heuristic for this.
03:41Um we call it don't beat the market.
03:43Don't try and beat the market.
03:45Um uh the concept here, like the heuristic you should use when you're you're to decide if you're near the frontier or at the frontier and uh too deep in into
03:56the latest trends is you should ask yourself why isn't this workflow the default?
04:02So for example when Ralph loops were becoming a big thing you should ask yourself like should I spend a ton of time optimizing my workflow to work with Ralph loops
04:16because if Ralph loops work for everyone like if they are the default then you probably should just wait for Anthropic or Open AI or ever to build the workflow into the into the default harness.
04:29You can think of this is sort of like an efficient market hypothesis where unless you have like real alpha you shouldn't be optimizing your workflow too much.
04:39And what I mean by real alpha is like some kind of information about either your users or your code base that the models might not know about.
04:48So an example for us is we it's we're we're a chat app.
04:51We have to render really long chats really quickly and performance is is important to us and so we need to spend a lot of time optimizing our react queries
05:00to render the chats quickly and we're willing to make sacrifices in other parts of our code base to make that happen.
05:07Um So if you have some kind of alpha like some kind of information about the app you're building that the models might not know about then you should put time into the workflow.
05:16Otherwise don't don't midwit meme.
05:18Don't don't be the person who has an amazing Emacs setup but like doesn't actually get stuff done.
05:24Okay three.
05:26Create slop free zones.
05:27So at Conductor we have this term we call a slop free zone.
05:32And a slop free zone is a part of the code base or a part of the app that requires really strict human review.
05:40And we're actually I think a little bit unusual in this way.
05:43I think a lot of people assume that we are uh pure token maxers and we are like ripping through like 30,000 line PRs, but we're actually not.
05:51Um we're actually quite careful with certain parts of our codebase, and then very loose with other parts of our codebase.
05:57Um the reason this is important is because if you are not careful about your slot free zones, your your code your codebase can get in a really tricky spot.
06:06Um and this actually happened to us.
06:07We have We've had to rewrite our whole app like a couple of times because we weren't careful about slot free zones.
06:13Um specifically, we have a uh migrations file.
06:18And uh in our CI uh any change to migrations file requires the AI uh a human to review it.
06:26Um we also assume that anything written in Slack is slot free.
06:29It's It's not written by the AI.
06:30It's written by a human.
06:32All of our docs, um all our cloud MDs, like all our skills, we put a ton of time into making them good.
06:39Um and this is also something I've seen with all the the the best builders up close.
06:43Like they put an unusual amount of time into the cloud MD or their skill files.
06:48Um and I I think like another way that I've thought about this is like if you had a new intern that was joining your company, and you had the opportunity
06:58to like whisper something in their ear every time they started working.
07:03Like every day, anytime they sat down, you could like whisper something in their ear.
07:06You would probably put a lot of thought into like what it is that you're whispering in their ear.
07:10And this is what the cloud MD or agent's MD is.
07:13It's like information that gets loaded into the agent's context every time they start working.
07:17Um and so you probably want to put a lot of thought into into those.
07:22Um okay, four.
07:25Feed the beast.
07:26Uh at Conductor, we have a uh internal tool we call the Conductor Internal Agent.
07:34Um and it is uh we also also known as the CIA.
07:38And the CIA is basically like the the uh centralized database of everything that's happening in the organization.
07:46So, anytime a new Slack message gets sent, the CIA picks it up.
07:50The CIA agent will see that a new message sent Slack, it will pick it up, and it will save it to a Postgres uh uh table.
07:56Anytime a user has a bug request in Discord, the same thing happens.
08:00Anytime we have a meeting, we are recording it, uh and it goes into the CIA.
08:05Um we call this feed the beast because you you want to for if your agents to be effective uh in your company, you want them to have as much information and as much context as they can have about the way you guys specifically work.
08:18And the best way to do that is by having a centralized place for all of the information to go.
08:25Uh I think this this tweet sums it up pretty well.
08:28Uh uh It's really effective to just put everything in a database and then give your agent a SQL tool and let it handle uh uh handle the rest.
08:40Okay.
08:41Next is free-range agents.
08:46Give your agents a lot of space to play.
08:49Give them Give them a sandbox where they that won't get killed, where they can explore your code base, where they can work on really hard tasks, where they can they know that they're not going to get shut down when you close your laptop lid.
09:01Uh Give them Give them opportunities to create more of themselves.
09:06Um Give them ways uh to collaborate with other agents and other humans.
09:11Um I think what's really in- interesting about free-range agents and like this this concept and why this is important is the models are getting better, and they are able to run for much longer,
09:23and there're going to be many more of them.
09:25And so, if they are confined to your laptop, then the they're not going to be nearly as effective as uh as if they are free roaming.
09:34Um The other thing that's important here is that once you give them a a sandbox to play in that isn't confined to your laptop, there's a bunch of really cool stuff that you can build on top.
09:44Um and we have built some of those things into Conductor and I I'll give you a quick glimpse into into some of those cool things.
09:52So, this is Conductor.
09:58I'm going to make it a bit bigger.
10:00Um and this is actually a new version of Conductor that is uh coming out this week and it is centered around collaboration in the cloud.
10:10And so, the the the thing I said was you need a a sandbox for agents to play.
10:14You need a free-range agent.
10:15And so, you'll notice that each workspace has a little cloud icon at the top and I can click it and get information about the sandbox that the agent is running in.
10:25Um and what's what's awesome about this is I can close my laptop and the agents are going to keep running.
10:30Up up until basically this week, every uh every task in Conductor was built on a get work tree, but now they're in a cloud sandbox.
10:37They are free-range agents.
10:38But what's also really cool um that you can that we can build on top of uh cloud is collaboration.
10:46So, you'll see here, I'll make this even bigger.
10:49That's me.
10:51And here are a list of the things that I am working on.
10:53And you can see that I am in the Conductor org.
10:56But if I scroll down, I can see what Cadence is working on.
11:01I can see what Lewis is working on.
11:04I can see what Tywan is working on and I can I can I can see that I can see Jackson's face pop up there.
11:09I can click in and see what he's working on in real time.
11:12Um and I think collaboration is the the one one of the most important new concepts uh in these tools that no one is really talking about right now.
11:21Um collaboration is important because uh not only as not only it is it true that all great things are built with teams of people.
11:30Like they're not built by individuals.
11:31They're they're built by teams.
11:32Um but also as the models get better, um as we've seen this with the two days of Fable, you can get a lot more ambitious with the kinds of things you're building.
11:40And if you're getting more ambitious, you're going to need more people and more agents to work on those things.
11:45So, I'm going to go into an uh a workspace that Caden is working on.
11:51Uh this one, and I'm going to say um I can review the changes that he's made.
11:57Um I'll make this a little smaller, and this looks fine, but I'm just going to say uh can we actually use tabs, not spaces?
12:10And Caden should be able to see that message happen in real time, and he can actually chat in the workspace as well.
12:19So, we can see here that he's typing.
12:22So, I see Caden is typing.
12:24Let's see what he says.
12:31Seems like he's typing a lot.
12:37Okay, maybe he stopped typing.
12:45I'll give him a second to take a look at it.
12:47But, the point is we can now have collaborative workspaces that are shared in real time with people on our team.
12:54Um Okay, he's typing again.
13:01Okay, come back.
13:02The agents have escaped.
13:04All right.
13:05So, I'm really excited about this, and we're rolling this out to all conductor users this week.
13:11Um I think I think collaboration is going to be one of the most important one of the most important new interface changes this year.
13:18Um the other really cool thing about cloud is that um and giving it the agents the free-range sandbox is that we can give the agents APIs to spawn themselves.
13:27So, I have here a uh I'll bring up my open claw.
13:32Uh this is my open claw called Lord Crandon.
13:36And you can see that it has I don't know how how well you can see this text, but it has access to a conductor API.
13:44And so, from my phone or from my Telegram or from really or from Slack or wherever I am, I can say, "Hi, can you create a new workspace for me that makes
13:56Yeah, makes all the buttons blue."
14:00And so, I'll I'll text that to Lord Crandon, and Lord Crandon has access to the conductor API and so can kick off work itself.
14:07Um So, let's see what it does here.
14:15Okay, so just created the workspace.
14:17The agent is on it.
14:19And then I can go into my conductor while I'm out and about, but it's still setting up the workspace, and it will do work for me while I am gone.
14:29So, I'm pretty excited about all the all the cool things you can build on top of of free range agents.
14:38Okay.
14:40The final principle that I want to talk about today and the title of this talk is orchestras, not factories.
14:48The whole like talk track today is about software factories.
14:52And I honestly kind of hate the term.
14:55I think it's the wrong way of thinking about these new tools that are emerging.
15:00I think like when I think of a factory, I think of automation.
15:05And I think of like there's a lot of amazing things about automation and like it makes our lives more efficient, and we can like create more of everything.
15:13But I don't want the future to be built around factories.
15:17I want the future to I want to feel like a human.
15:20I want to like be in the flow.
15:22I want to be like in front of an orchestra like waving my baton and we're and like I wave it this way and this this team of agents starts working and then this intermingling of humans and agents starts working as I go here.
15:34And I can when I want to I can zoom in on the details, but most of the time I can zoom out.
15:38And I don't think the future should be we are like managing swarms of agents and we are like factory line managers like pushing buttons getting the agents to like pump out the next feature.
15:50Like we can we can we tried this like 10 years ago with the term feature feature factories and it just doesn't work.
15:56Like I want my software to feel human and crafted.
15:59I want to feel like a human at the center of it all.
16:02And I think because we're all building these tools we actually have a responsibility to make the tools great for humans.
16:09I think it's really important to like use the words that make us feel excited and like feel feel capable and feel like we're in the flow and having fun.
16:18And so I don't think the future is is is something like this.
16:24I don't want to be I don't want to be in in my dark factory.
16:27I don't want to be a line manager.
16:29I want to feel like this.
16:31I want to be in the flow.
16:32I want to be having fun.
16:33I want to be crafting things.
16:35I want to feel like I'm Steve Jobs designing the Mac with a team of amazing humans and AI agents all in the same place.
16:42Like I want to feel like I'm in an orchestra.
16:46So here are my principles for for being the best builder in your organization.
16:52Stay near the frontier.
16:54Don't try and beat the market.
16:55Create slop free zones.
16:57Feed the beast.
16:58Free range agents and think about orchestras not factories.
17:05Came up with this handy acronym for for remembering it.
17:07Stickfo.
17:08All right.
17:09So thanks thanks a ton for for having me.
17:11I'll be around.
17:12Feel free to ask questions and I'll see you on the internet.