Software Engineering Is Becoming Factory Engineering — Zach Lloyd, Warp

AI Engineer · 20 min · 266 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Okay.
00:13Hello everyone.
00:14Uh, I'm excited to be here.
00:16Uh, my name is Zach Lloyd.
00:17Um, today I'm going to be talking about self-improving software factories, the new open-source model, and basically what I think is happening to development.
00:27A little bit about me just to begin.
00:30So I am a former uh principal engineer from Google.
00:33Used to lead engineering on the Google Doc Suite.
00:36I've been an engineer now for over 20 years.
00:40Long time.
00:41I am still uh shipping frequently, but I haven't written a line of code in the last six months.
00:47Uh and I'm the founder of a company called Warp.
00:50Uh Warp, if you're not familiar, is a open-source agentic development environment.
00:55Uh you may know us as a terminal.
00:58That is how the company started.
00:59We're basically a terminal that has agents built in.
01:03Uh we open sourced it a couple months ago and I'm going to talk a little bit about that experience and the motivation for it.
01:09It's popular open source project over 60,000 GitHub stars.
01:12We've had a couple hundred people contributing.
01:15We have over 800,000 active developers who are using warp.
01:19And increasingly we are focused not just on the terminal aspect and the interactive aspect of development but more so on how do you automate development.
01:30I'm going to talk mostly about that.
01:33So the thesis that I have uh is that the discipline of software engineering is going to become something more like factory engineering.
01:44And I'll explain what I mean by this in a minute but just keep that in mind.
01:47That's that's what I think is going to happen.
01:49Um, if you look at development over the past couple years, it's I mean it's crazy how it's changed.
01:57We've gone from a world of chat and AI autocomplete cursor co-pilot to the phase that we're in now, which I consider to be mostly interactive agents.
02:09So you're sort of sitting at your computer and you are telling cloud code to do something, you're telling warp to do something.
02:16And I believe what's going to happen over the next six months, a year, hard to predict the pace, is that we're going to move much more towards a world of automation.
02:25But before I get into that, just quick show of hands.
02:28How many folks in here are building with agents?
02:31Every 100% makes sense.
02:33Uh how many how many folks are building typically with multiple agents at one time?
02:40So again, almost everyone.
02:42How many people are running an agent right now?
02:44I'm not offended.
02:44I would.
02:45Okay, that's totally cool.
02:47I would be doing it too.
02:48Uh, how many folks are running agents in the cloud out of curiosity?
02:53So, that looks like less than half, but still significant.
02:59Uh, and how many folks have have set up a system internally to automate the whole software development life cycle?
03:06So, everything from like triaging, specking, implementing, reviewing.
03:11So, I see some hands.
03:14So some people are doing this.
03:15So this is what's going to happen.
03:17Uh every project of significant size I believe is going to have something like this.
03:23Um and it's going to look kind of like this big loop.
03:27And everyone is talking about loops.
03:29There's nothing that complicated about loops.
03:31This loop says a cloud software factory.
03:33This loop could literally just say like the software development life cycle.
03:36It's the same thing.
03:37Um but just to go through this loop, it's like ideas are going to come in at the top.
03:43Agents are going to do triage.
03:45If something is complicated, they will write a spec.
03:47Uh these little blue boxes are where humans step in.
03:50Humans will review the spec.
03:52Uh agents will do the implementation.
03:54A human and agent will review the code.
03:56Agents will verify.
03:57Human will review the product.
03:59You ship and then you monitor and round you go.
04:03Uh and this is what software development for better or worse I think is going to end up looking like.
04:07So I repeat the thesis which is that if this is what software engineering is going to look like um software engineers are going to be the ones who end up building and managing these factories.
04:19[snorts]
04:20Now I promised at the beginning and I put in the title of the talk that I was going to talk about open source and so I want to do that for a few minutes.
04:27I'm going to take a quick digression.
04:29Um, I bring up open source because one of the main reasons that Warp opens sourced was to build uh build a public factory.
04:38Um, and so this is a picture of this website we've built called build.warp.dev which shows all of the issues that are flowing through our system and what state they're in,
04:50what agents are working on them, what contributors are working on them.
04:54And it's kind of like a protoactory done at scale.
04:58It's not working perfectly, but it is working.
05:01And one of the reasons we open sourced was to try to build this.
05:05Um, just in general, I think it's interesting to talk about open source in the time of agentic development.
05:12This is a really stupid graph, but it's like it just it you get it.
05:16It's like it's becoming much cheaper to build software.
05:20Um, a correlary of that is that it's becoming trivial to clone software.
05:27And so if you are in the software business, and I don't know how many folks in this room are in the software business per se, but it's very hard to build a software business if it's free to build software.
05:38It's hard to capture the value, especially if a competitor can clone.
05:43And so my big takeaway or big tip for everyone in here is that the first thing you should do is patent your code.
05:49I'm kidding.
05:50Don't This is a complete joke.
05:52[laughter]
05:52Don't don't don't do this.
05:55Uh my my first tip is obviously you need to have a great product.
06:00Uh this has always been the case.
06:02Um but I would say a great product probably was never enough.
06:06But even now more than ever, if you think that you're going to build a great software business just by building and shipping a great product, you're probably not going to succeed.
06:15Uh you need advantages beyond the product.
06:18And so those advantages could look like distribution ecosystem.
06:23It could be that you have a great brand or a data moat.
06:26You might have capital.
06:28Um, but if you're a startup, again, I'm coming from the startup world here, you just don't have these advantages.
06:34And so, you you still want to break through.
06:38Uh, and one of the ways that I suggest doing this is by building in the open.
06:42And so, to be clear, it took Warp five years of building closed to sort of make the leap into building in the open.
06:49And I'll explain why.
06:51But if you build in the open, um it helps build your ecosystem.
06:55It it can take you from being like hated on hacker news to like tolerated.
06:59Uh it can burnish your brand.
07:01It creates community.
07:02And so there's all these advantages to it.
07:05Uh and some of the things that I think have traditionally been a pain um can now be managed.
07:11And so like the traditional, you know, pain of open source might be something like you get a lot of noisy issues, you get sloppy PRs, you can end up in code review hell, you can end up having to spend a lot of time
07:25verifying changes.
07:27And so the solution, it's a kind of a long-winded way of getting to this for open source or at least for Warp in the case of open source.
07:35The thing that made us finally decide to do this was that we built a whole set of automations, really a software factory around managing the open-source project.
07:46And so, like I said, this is what I think the future is going to look like.
07:50I'm going to drill into it a bit just to get a little bit more technical for folks who want to try to build something like this for their own projects.
07:58So, what are the components of an effective software factory?
08:01uh it's really not that complicated to start or at a high level.
08:06You need a set of automations.
08:08You need a way of providing context and skills.
08:12You need a way of bringing humans in at the correct time.
08:16So sort of like when things get stuck on the factory.
08:19And then a really important thing is you need some set of self-improvement capabilities.
08:23So think of this as loops.
08:26And if you do this right, uh, in the open source world, you can get a, you know, world where agents are helping contributors contribute.
08:33They're helping maintainers maintain.
08:36And I want to emphasize there's nothing special about open source here.
08:40I think every sizable project can benefit from this approach.
08:43And I I predict that every every company, every open source project will have at its core a software factory.
08:52kind of like the way that CI/CD became just like, oh, of course you have that.
08:57Uh maybe I don't know when that happened, 10 years ago.
09:00Let's tour the t uh the factory floor for a second here.
09:03So, you're not going to look at this.
09:05This is too much.
09:06Uh the the point of this slide is not to have you read the workflow.
09:10It's that the factory floor is basically a graph um of steps where you are defining like okay, how does software get built for my product?
09:21And it looks pretty similar for every product.
09:23Things come in, they flow through.
09:25Um, they get stuck at certain points.
09:28Um, and you know, broadly speaking, just to back out a second.
09:32So there's the inputs.
09:34The inputs are really ideas.
09:36Um, the inputs could be coming from your team, they could be coming from your users.
09:41Um, the inputs themselves tend to come in through certain channels that you should think about as like your task tracker is an obvious one or Slack, your teams, like your communication channels.
09:54They could come directly from like your terminal or IDE.
09:57They could come from your monitoring systems, but there's some set of inputs that bring work into the factory.
10:03There's triage.
10:04This is a really important step.
10:06So again, I boil it down to something very simple, but like you want an agent that is looking at issues as they come in and just saying, you know, if this is easy and this is unambiguous,
10:18just implement it and this is how you can actually get going with a factory.
10:22If an issue is hard, uh I recommend having uh an agent that produces specs.
10:28Folks in here using spec driven development, show hands, people follow this.
10:32Okay, you can do this many different ways, but I think it's very effective.
10:36The way that uh we do it at Warp that I recommend is having an agent write what we call a product spec and a tech spec.
10:44Product spec describes the product invariance that you're building towards.
10:48Techsp spec describes the architecture and the shape of the code.
10:54Then you have an implementation agent.
10:55This is basically a coding agent that runs somewhere in the cloud.
10:59It makes a diff.
11:00You can use all sorts of coding agents for this.
11:03You have review.
11:05This is in many ways the most painful part.
11:08Like I expect that people are a little bit tired of reviewing a gentic slop.
11:13Uh I would have an agent do code review first and then it becomes over time like a riskmanagement exercise of like when do you bring in humans to do code review.
11:24But you want to have a step in here where humans can do it.
11:28This is a very important step uh for certain types of apps.
11:31The verification step.
11:33So this would be things like computer use.
11:35Uh if you're building a a sort of UI, having the computer actually use the code that the agent produced and producing videos and screenshots.
11:43CI/CD still use it obviously.
11:46Uh and then monitoring.
11:48So agents don't stop in your factory when code is shipped.
11:52They should observe what's been shipped.
11:54Is it crashing?
11:55Is it being used?
11:57and round and round you go because you take the output of this monitoring step and you feed it back into the top of the factory.
12:04Now you could try and build this and um I went to a talk earlier that my friend Adam gave where Uber has built an internal version of this um and it's pretty cool.
12:15I would say for most, you know, most organizations, it really depends where you are.
12:20You'll be able to build a simple version of this easily, but to build a thing that actually scales is probably like you should probably be focusing on your own product, not building this infrastructure
12:31because there's a lot of stuff that you end up wanting.
12:33You're not again not supposed to read this.
12:35Uh it's just a lot of stuff.
12:36If you do build it um or if you buy it, you'll end up with something that looks kind of like this, which is um you're going to have a bunch of ways of getting work into your factory.
12:48That's what's at the top here.
12:50You're going to have a sort of control plane for figuring out how work gets distributed across your factory floor.
12:56You're going to have the actual place where the work happens.
12:59And so that's going to be cloud sandboxes.
13:02It's going to be figuring out what agent to run.
13:05So, what's the harness?
13:07What's the model?
13:08And then finally, I think this is a really important thing.
13:10You're going to want to set up some kind of data plane that sits below your factory.
13:15And so, that's something that lets agents remember what they've done, learn, um, improve over time.
13:24The factory is not just like a product, it's also a mindset.
13:28And this brings me back to the thesis I had at the beginning.
13:33You need to measure and improve.
13:36So factory, this is where the I don't know, you can stretch this metaphor as far as you want, but like you should be thinking of efficiency.
13:44And so that means like how much software did you ship?
13:48How much did it cost in terms of human time and token time and you're going to want to measure this and try to improve it over time.
13:57A key part of this is creating loops.
14:00So, uh, loops are, again, they sound complicated.
14:04They're not that complicated.
14:05Loops are basically ways of, uh, having agents improve, um, by like observing what they're doing, where they're failing.
14:14Um, and so, a common kind of loop that you're going to want to put in your factory is like a skill loop.
14:19That means you're going to have your factory agents that are running skills, and then you'll have observer agents that are seeing how those skills are being applied, looking for issues, and trying to improve the skills.
14:29So for instance, if you had a code review agent uh and it was leaving comments and a senior engine engineering or team was going and correcting those comments, you'd want an observer agent that would look at that
14:41and basically uh improve the code review agent for the next run.
14:48This is one thought just to leave folks with like what is where does this leave engineers?
14:52Um, I think you're going to have to get into this mindset, and I'm trying very hard to get our team into this mindset.
14:59It's not always easy that you're not just building the product, but you're building the thing that builds the product.
15:07And that's like it's just different.
15:10It's more like process engineering or manufacturing or something like that.
15:15Um, and you could think like, okay, maybe that's a bummer.
15:18Like, is that a bummer?
15:19is that uh you know and it depends like it depends what joy you get out of software engineering.
15:26If your joy is in writing the code, I think you're everyone in here who is a software engineer is going to be writing less code.
15:33But if your joy is in shipping product, like it's never been a better time.
15:38And this is actually where I find my joy.
15:40It's like I like building and shipping the thing.
15:43So everyone in here is going to code less, but they're going to ship more.
15:46And that's going to be a trade-off.
15:49But if you approach it like you're a a factory engineer, I think you can see like that there's still a really cool set of engineering challenges.
16:00Um you could almost think of it as like meta-engineering like how do you engineer your system of agents to be the best possible at engineering that I think is a very compelling and interesting set of challenges to solve.
16:12So that's it.
16:13Um uh for folks who were interested uh this if you follow the link on this QR code I've set up um a open source GitHub repo where anyone who wants to try building their own factory agents can do it.
16:28This uses Warps uh agent platform as part of it.
16:32But you honestly you don't have to use it.
16:33I'm not trying to like push into our product.
16:35But this should give you a good sense of like, okay, if you want to set up uh an agent that does uh triage or an agent that does uh specw writing, how do you actually do that?
16:46How do you get from like the theory of uh working with a factory to actually putting it into practice?
16:53Um I don't know if we have the capability to do questions in here.
16:56I saved a few minutes for questions if anyone has questions.
16:59Otherwise, I will I will wrap up.
17:01Yes.
17:04what you said you don't want to build this also build
17:10it's a great question so I said something that's almost contradictory I think um you should the way you should think of it is like you everyone's going to deploy some sort of factory but then the tuning of the factory
17:23the like are these the right skills for my domain is this factory building my product in the right way I still think there's a bunch of interesting engineering challenges
17:33And for some places you can build this, but again I I think I think that the you should probably be focusing on building the core product for your company for the most part, but there's a bunch of like tuning and like
17:45uh figuring out how to make the factory work for your product that matters.
17:49That's a great question.
17:50Yes,
17:51thank you so much.
17:53So if you were a college student right now,
17:55yes,
17:56graduating and entering the workforce, where would you spend time?
18:00Yeah.
18:00So the the question in case people couldn't hear was like if I was a college student graduating and entering the workforce right now.
18:07So I think that the most important skills in this new world are adaptability.
18:12I think that that's critical thinking.
18:15It's like this the speed at which you can learn.
18:18I do think I don't know if you're a computer science student, but I still think there's a ton of value in understanding like the underlying systems and architecture and being able to reason and understand
18:29the code, understand the specs that agents are written, sorry, are writing.
18:34So, I I would focus on on those skills.
18:37Um, and like I don't know, we're hiring more people than we've ever hired.
18:42There's a lot of kind of like misdirection around like you know people not being hired because of AI.
18:48That's not the experience we've had so far.
18:50And what I'm looking for are like really adaptable product focused thinkers who can like basically be great problem solving problem solvers uh even as the underlying technology changes.
19:03Yeah, I think I have time for one more question.
19:05Yes.
19:12What about the like so the question was what about the the product taste and like the how do you actually build something useful I think is probably the right like maybe the framing and like
19:22uh where do the ideas come from and so I think that um the problem with the factory metaphor even though I'm like leaning into it because I think that's like that there's something to it is that it can kind of sound like uh mechanizing or dehumanizing
19:36um and I still think underlying ing all of this, the only thing that matters is like are you building something useful?
19:42And if you have like a a factory that is like churning out that no one cares about, it's like what's the point?
19:49And I think that human taste, human input, human product sense, um humans like guiding at those touch points where you can't automate stuff is absolutely like essential.
20:02And like that's like what I do like I'm trying to figure out what what do customers want, what do people want, what's going to be valuable to them.
20:08So I think that's an absolutely key point to it.
20:11I think I'm at time so I I have to go.
20:13I hope folks uh enjoyed this chat.
20:16I'm really grateful for being invited to speak.
20:18So thank you all very much.
20:34[music]