How Forward Deployed Engineering is done at Factory — Eno Reyes

AI Engineer · 21 min · 194 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12This is the forward deployed engineering track in case you're in the wrong room.
00:16Um as you already know, forward deployed engineering is one of the hottest topics in AI.
00:20The most important companies on the planet are building out massive FTE teams.
00:24So, think OpenAI, Anthropic, Google DeepMind, you get the idea.
00:28Forward deployed engineering was pioneered by Palantir many years ago to embed really strong software engineers directly into their customers' orgs uh to implement customize their platforms around the nuances
00:38of the real world.
00:40So, today we brought in some amazing speakers from Anthropic, Cursor, Factory, Ramp, Decagon, and many more uh to talk about the current state of forward deployed engineering, how it works at their companies,
00:51and where it's going.
00:52Our first speaker Our first speaker is Eno Reyes.
00:56He's the co-founder and CTO at Factory, which is building autonomous software engineering agents for enterprise teams.
01:01Previously, he worked in machine learning and software engineering roles at Hugging Face and Microsoft.
01:06Let's give it up for Eno.
01:08[applause]
01:11Yeah, hey everyone.
01:12Excited to chat today.
01:14Um and you know, basically I I my hope is that at the end of this you guys get a sense of some of the work that we're doing on behalf of our customers and with our customers,
01:24and the role of what we call a deployed engineer should hopefully be a little bit clearer since I think that there are honestly tons of different models um for
01:34uh for how this should actually operate inside of an org.
01:38Um and so I think that when when we start I I I do think that there are some nuances in sort of like the Palantir era playbook.
01:47Um and I generally the way that forward deployed goes is I I see that there are lots of different takes on sort of where forward deployed sits within the org, how much it interfaces with the actual product team or the engineering team, how much work is done on behalf of the customers versus
02:04with them, and how much work is done on code itself or basically like in the software system versus with the humans and sort of strategizing, right?
02:13And so, um generally, I think the there's um in this older model, a lot of the way that software needed to be built was you needed to go and access that codebase.
02:23You needed to integrate directly in to data streams or software or products that basically you could only access behind the curtain of the customer.
02:32And so, if you were building something that was heavily integrated into their environment, yeah, you kind of had the need to send and sort of parachute in individuals into the org.
02:41Um but really that has transformed over time into a role that sort of forks out, and you see a lot of people who are sort of quote-unquote forward deployed engineers
02:53or deployed engineers or applied AI engineers, and it's always it's always a little bit unclear.
02:59Are they doing maybe professional services work on behalf of their customer?
03:03Are they transforming like the product around an individual customer?
03:08Are they just building entirely net new things in the customer's environment, maybe on top of your product?
03:14Uh and I think that the the at least at Factory, we definitely do not want to be doing professional services work on behalf of a customer.
03:23So, if a customer says, "I want to do a uh modernization of a codebase, uh and it's you know, I just got quoted from all of the big consulting firms, it's going to cost this much.
03:33Could you do this consulting work for us?"
03:36Uh our goal is not to go and actually do that migration on their behalf, even if we happen to be using our product, right?
03:44Um and that is because we don't think that that actually makes our product that much better.
03:48Uh and ultimately, that is a great way way get I'd say a decent amount of revenue, but I don't think that that's the way that you can scale a business out
03:55enormously, right?
03:56And so, what we've done is we've instead said, we need deployed engineers to be the tip of the spear of the product.
04:04And I'm going to do this and then go back.
04:06But but really when we say the tip of the spear, what we mean is that deployed engineers are basically the stream of information from our largest and most critical customers
04:18of the engineering leadership in that org.
04:20The on-the-ground tactical engineers, their thought process about how software development and AI is actually happening at that org, and then flowing all of that information back into our product
04:31to then rapidly adjust our product in order to then fit into the customer's environment better, right?
04:37And so, factory really should be when it gets deployed, and I'll talk about what factory is in a second, but we want that to be effectively self-assembled inside of our customer's environment, right?
04:47And then there's a lot of work that goes into understanding that customer's environment, what the flows that happen, and ultimately the ROI story.
04:57And what factory really is to our customers is a set of building blocks for building a software software factory, right?
05:05And so, when we say software factory, what we mean is there's this implicit process that every organization in the world sits on top of, where signals from the outside world flow in on one side, and those signals could be a lot of different things.
05:19It could be customer conversations, it could be bug reports, it could be internal Slack or Teams conversations, it could be an executive saying, "We're going to build this thing," right?
05:28All of these are signals.
05:30Some of them have higher weight than others, and those signals flow in, and humans implicitly or explicitly then choose to then prioritize, triage, and build plans around those signals.
05:42Those plans are then converted, typically by software developers, into changes into some source of truth, a code base, an engineering system.
05:51Um and as those changes are actually executed on, they flow through a validation stage where people maybe review the code, they QA, they assess the security implications, they uh pass it through automated validation like SAST tools,
06:06linters, type checkers, and ultimately when everything passes, they then ship and deploy.
06:11And what do you do with deployed monitored software?
06:14Well, it it generates more signals, right?
06:16So, this implicit feedback loop is instrumented very poorly, to be honest, at most organizations.
06:22And if you're able to take AI and actually transform each of these stages of the pipeline and build an understanding of what the workflow looks like at your org from each stage to each stage,
06:33then you actually can get to the point where you have a a flow through from signal to deploy that has no human intervention.
06:41Now, importantly, that does not mean that humans are not a part of engineering this system, right?
06:46But it is that the flow of signal to deploy is uninterrupted by a human.
06:50Um and that software factory concept is obviously not something that can just snap your fingers and it appears, right?
06:57Instead, it requires an investment from the organization.
07:00We we like to say this is built, not bought, right?
07:03But what the platform that we've built basically provides to people are the canonical one model independent agent harness that you need to do this, because if you want to build a software factory,
07:14if you choose to build that software factory in a vendor locked solution that has like one model available to it, uh that is going to not only be expensive,
07:24but two, uh there's open questions about model independence and like what is the role of the model provider in dictating what you can or cannot build with your software factory, right?
07:34Um and if you also don't own the traces, the data, everything that flows through your software factory, um then you're probably going to be in trouble as you start to want to evolve your software factory, right?
07:45And so with Droid, the hardness that we build, you not only have model independence, but you also have access to every piece of data that flows in in and out of Droid, alongside
07:54centralized governance and control at the enterprise layer to be able to dictate where what information flows where.
08:01Um you can air gap Droid if you want.
08:03Some of our partners um in, you know, the most secure uh environments, uh think finance, health care, uh gov, uh they air gap Droid and they run their software factories entirely contained.
08:15Um One of our deployed engineers jokes that you could run Droid in a submarine if you wanted to.
08:20And that's that's honestly true.
08:21And so when we think about what the role of this deployed engineer is in that context, which I probably should have started with, um you know, you really need somebody who can go in and say, I understand
08:34this new model of building software and I understand the building blocks and the pieces.
08:40I can help enable building and constructing these software factories with your team, but I ultimately would like to one, make it so that our product effectively, you know, one click self-assembled
08:51into your environment, which is needed when you have 45,000 people, maybe hundreds of thousands of uh of of engineers, maybe you have tens of thousands of code bases.
09:01Uh you you've got to self-assemble, right?
09:02You just can't manually install this level of of complexity.
09:07Um and and also on the sort of like end loop, why do all of this, right?
09:12I I would argue that there needs to be an ROI or an outcome story that is extremely clear from the beginning so that you can say, well, we know
09:20every code change that flows through that gets AI code review, AI QA, AI security analysis is maybe 87% less likely to hit a bug.
09:30And what that means is that we can reduce our our bug rate by X, that increases our customer satisfaction by Y, and that leads to revenue or growth or new business, right?
09:41Something needs to flow from this software factory process to core business goals.
09:46And that often is a complex story that requires engineering knowledge, it requires business knowledge.
09:52And so, if those are the types of things that you think are interesting, that is what deployed engineers today are doing for us.
09:59Um I've sort of outlined it a little bit here, but that teach the model step is super important because most organizations do not have an autonomy maturity model.
10:09They do not have a road map, they don't have a conception of what it means to truly build an autonomous software organization, right?
10:17I think a lot of people ask the question, what do the humans do in this world, right?
10:20For us, we see an extremely clear role for humans in evolving, refining, and scaling software factories, right?
10:29So, you basically the engineers at a company go from directly manipulating software to directly maintaining and managing a system that builds software.
10:38And that sort of like upgrade in the level of abstraction that you operate at is actually very difficult.
10:44And a lot of people find it extremely challenging.
10:48I would argue that in fact most people, even very thoughtful software engineers, will have a learning curve in trying to shift.
10:55The people who I think are are well suited for this are DevEx people who have already been thinking about enablement of other developers.
11:02I think product managers who want to become very technical very quick can become really great at doing this.
11:09And I think that generally like people who are used to to working on teams where
11:15[clears throat]
11:16high quality dev environments were a priority, you will you will get some of the canonical things necessary to enable these agents to succeed.
11:25Um I haven't really talked about this last one, which is design the workflows.
11:31And I will get to that in a sec, but I I think that when I say tip of the spear of the product, like keep in mind I really do mean everything that is happening inside of factory.
11:41So, our product encompasses enterprise controls, the droid harness, the workflows that run on top of it, the observability tools, the cost controls, the auto model routing, the quality of the harness.
11:52Like, all of these are pro- potential opportunities of improvement that you will discover when you work very closely in these varied or diverse orgs like how to solve.
12:02Um, so making a code base agent ready, right?
12:07This is a very challenging thing to do.
12:10Uh, most organizations have some degree of consistency in how they've chosen to build deterministic validation loops inside of their company, right?
12:18So, your code base runs linters, type checkers, uh, it might run some security scans, and it's like check mark.
12:24Like, it passes or it doesn't.
12:26The end end test, they pass or it doesn't, right?
12:28Or they don't.
12:29Um, what agent readiness really is is it's a measure of how many of these deterministic validation loops are present inside of your code base.
12:36Uh, when you have a huge volume of these feedback loops, uh, agents are able to operate for greater periods of time on more complex tasks without human intervention.
12:45So, we have like a product that we call missions, which I'll also touch on in a sec.
12:49But, missions is basically an extremely elaborate harness built around the concept of working on extremely difficult knowledge work problems that are validatable, right?
13:00And so, the quality of the output of these very long-running harnesses of advanced agents is directly proportional to the degree to which you can validate their work.
13:09And so, if you introduce the ability to validate at scale, then you introduce increasing autonomy to the org.
13:16So, what we'll look at is we have tools that help scan all of these things, but often times, uh, the change is not so simple.
13:23Uh, for I'd say maybe 30 to 40% of the low-hanging fruit, you click droid, please fix all of this and it'll go in and it'll fix it, right?
13:30But for the other 60% some of them involve workflow changes.
13:33Sometimes humans are not used to the degree of I would say like nitpickiness of these automated systems.
13:40And so you have to sort of be aware of the concerns, the the humans, you have to think about like the way that people are currently developing systems and say, "How do we introduce some of these more extreme validation strategies
13:52without interrupting the dev flow of the humans who are involved in the work?"
13:57Um and and I mentioned missions because really I think this is one of the more end game of the agent era at least, pre-software factory era.
14:06But the more end game of the agent era style harnesses where it's simply a long running harness that has almost no human intervention except for the planning stage, right?
14:16Where you go in and you say, "I would like to have this very bounded task.
14:20I know that I want to solve this task and here is what solving this task means.
14:24I will now basically push a lever of inference until the task is complete, right?
14:29And so that is actually unbelievably competent at solving problems where like is complete is verifiable.
14:38So if you can frame any problem as the set of verification uh systems that need to validate it, then you can solve that problem with AI today.
14:48Uh and we've seen this work on some pretty insane problem spaces like migrating, you know, 30, 40, 50 million plus line code bases uh fully autonomously, um working on advanced
15:01uh like deep learning strategies around biomed, health care uh sort of problems, uh financial institutions that optimize equity research where you can actually build models of different equities and sort of analyze and compare and and build sort of a system that can then back prop and or trade on top of the those equities.
15:22Um like it it's mind-blowing to me every day what I what I hear people are using with these tools, but it is not something that you can just download, install, and hit play, right?
15:31It does require agent readiness.
15:34So, if your code base isn't agent ready, you won't see any of the success of the most capable AI systems in the world today, right?
15:40So, this is why we want people to go in and help our customers and say, "Hey, look, you can solve this actually very difficult problem, but it is going to require a different form of investment than you were thinking.
15:52Less so solving the problem, more so preparing the environment for verification of the problem."
15:58And by the way, if you're familiar with how these models are actually trained, like this makes total sense, right?
16:03They they get dense reward when they get post trained on all these complex tasks.
16:07Models need dense reward.
16:08These verification signals form the basis of that reward that they use to keep them on track over a long-term goal-directed problem.
16:17Um So, I sort of mentioned this earlier, but but I think that the the core goal for us really is to say, if we can hand over a model
16:26to you of how this should this transformation should go, then we should theoretically be able to say, "Let's do this in a couple of different places, and then let your team actually scale this out across the company."
16:38I always use the analogy of if you're familiar with Walt Disney's Epcot, the the theme park.
16:44Like basically that theme park was created originally Disney wanted to create like a master planned exemplary city.
16:53He said, "Look, if I can create a city that is the future city, then I can use that as a model to the rest of the world cities, and they can develop entirely new forms of transportation
17:03and flourishing."
17:04And it became a theme park.
17:06But, what's interesting is that in that small example, a lot of other cities actually did cite some of the ideas that he was writing down and sharing about what like centralized urban transit should look like.
17:17And now you have like some more contemporary cities built in the last 50 years that basically modeled after that toy example.
17:25Um what we want to do is we want to make sure that we get some of that lesson that if you have a working example of a city of the future, of a code base of the future,
17:34um people are smart.
17:35They're clever.
17:36Humans will look at that and they'll say, "Man, that's really cool.
17:39Let's bring that to my part of the code base, right?"
17:42But if you build too much of an advanced example, then people will say, "That's a theme park.
17:47That is not at all how the rest of the world works.
17:50I just can't see how that would apply to the way that we currently work today, right?"
17:54So it's kind of a delicate balance that you have to walk of building something that demonstrates the future is achievable enough, but ultimately does not scare away uh an org who is thinking, "Man, what is going to be the cost of transforming at this pace,
18:09right?"
18:10Um I always think about that quote, you know, the the future is here, it's just not evenly distributed.
18:15Um there are some code bases, and I I say code bases, not even companies, that are truly remarkable.
18:20They are effectively uh beginning to run on autopilot.
18:24Uh we ourselves have roughly 15 to 20% of what we call like autonomy, and our autonomy ratio is like in the upper 80%, which means the ratio of actions done by humans to AI systems before interruption, right?
18:37So our own code base is fairly agent-ready, pretty autonomous, um but uh the code bases of some of our customers are actually even more autonomous because they operate in more constrained
18:48uh ways, right?
18:49So it's it's sort of like a uh it is not obvious like who gets 100% autonomy first.
18:55I would argue it's probably very contained internal tools.
18:58Like we have something we call like legal droid, which is our legal workflow.
19:02That is effectively 100% autonomously maintained, but our like core harness, uh we do not yet have validators that can validate some of the hard visual problems of a like terminal based harness.
19:14Uh Things like flickering are really hard to catch in a verif- in a verifiable way.
19:19So, we're unable to close the loop on some of those challenges.
19:21It's an engineering task to build the system that can verify some of those very hard problems.
19:28And that might give you a picture into sort of like the weird world of the future where humans are sort of visually our advantages in being visual, our advantages in having context of the outside world
19:38provide us a lot of work to do in order to build these systems.
19:42So, who's great at this?
19:44I If you are a former founder, for sure you should do this.
19:47I think it's like the quickest way to basically build out I mean each like stage of the SDLC that Droid has, we think is a billion-dollar business.
19:58Like just code review, just incident response, just QA, just testing.
20:02Like each of these you will help define basically the nature of these products.
20:07If you are someone who is used to tech communication, right?
20:11If you are fluent in AI, you understand how to speak to every level, you have business acumen, you have executive presence, that is another great example of someone who should do this.
20:21And if you are a systems thinker, if you love designing systems, if you love closing loops, modeling data, and understanding how the flow through a potentially extremely complex org should look, then you are also someone who would thrive at doing this.
20:35So, if all of this seems interesting, hopefully it does.
20:40Please do reach out.
20:42And you can reach out to me directly.
20:44I'm just Yeah, I'll say it.
20:46It's on the slide.
20:47I'm eno@factory.ai.
20:49And so, you can just email me directly or you can apply on our careers page.
20:53It's called engineer, {comma} deployed.
20:57So, that's the role.
20:58Hopefully this is interesting and gives you a taste of what we're doing at Factory.
21:19[music]
21:20Woo!