How Forward Deployed Engineering is done at Cognition — Jia Wu

AI Engineer · 17 min · 229 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12It's nice to be here.
00:13I appreciate you all.
00:14My name is Gia.
00:15I'm a deployed engineering lead at Cognition.
00:17And today, hopefully, what you'll take away from this is that how we deploy Devin in the field is very much a function of how we view deployed engineering at Cognition.
00:26So, how the four deployed motion makes AI engineering actually real.
00:31Before I start, how many people like have heard of Devin or like know of Devin?
00:36Oh, cool.
00:36And I'm not talking about like the Devin of today.
00:38Like, I'm talking about the Devin back in 2024 when we first released and it was like, "Oh, SweepBench 13%.
00:43We're so We're so back."
00:44And as engineers, we were like, "We're so cooked."
00:46But, I mean, after a week, everyone's like, "Oh, this is actually like not that useful."
00:50Um "I would only use this if I was desperate and out of ideas."
00:54That's 2024.
00:55Uh and let it be said that we have a sense of humor because we took this and we ran with it.
01:00I'm sure you've seen all these ads around SF.
01:03We're actually good now.
01:05So, the reason why we're good is I'll talk a little bit about the product surface area a little bit just to give you folks a little bit of context for who
01:13might not have used Devin before, uh who might not have exposure to something like Cognition.
01:18So, if you've used Flood Code, we also expose a CLI.
01:21If you've used something like, you know, Cursor, Windsurf, we also expose an interface such as an IDE.
01:26Uh actually, how many people know of Windsurf or have used Windsurf in the past?
01:31Sweet.
01:33So, I come over from the Windsurf side after the Windsurf acquisition.
01:36Um great times.
01:38And then, what we're actually known for, specifically, is for Devin Cloud or the Devin Cloud agent.
01:44And I'm not going to bore you to death talking about like all the components and features and everything that comprise of the actual product.
01:49I'm not here to sell you on that.
01:51What I am here to sell you on is we are one of the premier software engineering functions across the enterprise, and we're actually able to deliver impact on a global scale.
01:59What do I mean by that?
02:01We can take a pause and take a look at this figure.
02:04So, internally at Cognition, over the last 6 months, for better or worse, we might have been behind on hiring, but using our agent, we were able to ship almost an order of magnitude more
02:14good quality robust PRs across the organization.
02:18So, it's a step function increase in the amount of engineering leverage that we can have by deploying our own agent.
02:24You don't have to take our word for it.
02:25If we take a look If we can take a look at the specifics of how we're actually being consumed, how we're being utilized across the enterprise, it's a parabolic growth of how companies are adopting our agent, deploying our agent, and using it in multiple different use cases and multiple different scenarios.
02:41So, what does it mean, right?
02:42Like, how does this actually happen?
02:44Well, it can only happen with the forward-deployed engineers at Cognition.
02:48And I'm going to frame up the problem from like a couple of like buckets, right?
02:51So, there's two circles in front of you on the screen.
02:53One of them can represent the domain of a product, right?
02:57As a business, as a software engineering organization, obviously, you have a product.
03:02You obviously also, on the other side, on the left-hand side or right-hand side for you guys, you specifically have like a bucket of problems that you're looking to solve, right?
03:10You have a product, you have problems that you're trying to solve, and the intersection of these two, or whatever you would call it, is the product-market fit.
03:18Right?
03:18Hopefully, you have a pretty good overlap in the sense that whatever it is that your company does, you can actually bring value to customers.
03:25And hopefully, whatever problems the customer has, you can solve with your company's stuff.
03:31So, the forward-deployed motion at Cognition essentially aims to maximize the overlap between the products that we typically build and the problems that we're experiencing across the enterprise.
03:42So, what does that mean, right?
03:43The first fundamental concept that I would like to convey is that forward-deployed engineers at Cog deeply understand the problem space at hand.
03:52And specifically, right?
03:53Like if we think of the problem of software engineering, and I'm just going to like mask the features at the bottom, we don't really care about those, but if we think about when you go ahead to take some sort of codebase,
04:02some sort of implementation, and you need to like build features, you need to maintain that software, uh you need to like review, deploy, maintain that software.
04:10All of these steps, all of these functions have a lot of business value behind them, right?
04:15You can only do like it would be great if we could build from zero to one and just like prompt stuff and not worry about like legacy code, but that's not the reality of the situation.
04:23It would be great if our product engineers or our product managers could just take user stories and say like these are the things that we need to build in order to get value and revenue.
04:32And then we come to coding.
04:34Coding itself, at least from our perspective, is a mostly solved problem, right?
04:38These models are so good now that like with any type of context, with enough context engineering, you can get the code blocks that you really care about.
04:47But the problem isn't like writing code faster, that's usually only 20% of the problem.
04:51The problem really just becomes like how do you test this code?
04:53How do you review and deploy this code?
04:55And how do you maintain this code across the enterprise?
04:57So, that's the premise of the problem.
05:00Now, I will predicate that by saying that like we also have the solution, I'm not going to bore you to death about talking to you about the solution, but for deployed engineers
05:07at Cognition, we map Devin's capabilities specifically to the customer problem.
05:11So, if the software development life cycle is extremely complex, deploying the agents for with like no specific direction, you're straight up just token maxing.
05:20Right?
05:20Like you were wasting tokens, you were burning you were burning spend, you're not getting any tangible outcomes.
05:25So, we try to identify, when we partner with customers, when we take meetings, and for example, right?
05:30Like my day might look like four or five hours of customer calls, and then four or five hours of like actual hands-on keyboard work.
05:36Those four or five hours of calls actually allow us to understand very deeply what strategic initiatives are the highest leverage for the business.
05:44Right?
05:44Once Once identified that, right?
05:46How can we automate ourselves out of the job in the sense that we set up the agent in a way that it runs all of the automations for us, right?
05:52We don't have to be there manually triggering the agents.
05:55It can respond to like specific alerts, specific events.
05:58But most importantly, as forward deployed engineer, how do you measure the return on investment?
06:05And it's very ambiguous.
06:06And it's an unsolved problem because the company that will solve this will be um you know, $5 trillion market cap.
06:12So, specifically, our forward deployed engineers will embed in the customers ecosystems, right?
06:17We take a look at, you know, the backlog of stuff that needs to be built.
06:20We take a look at the remediations that need to be done.
06:22We take a look at all of like the delayed code that never ships or the test that nobody writes uh or the automatic triage of specific alerts.
06:29And we map our product capabilities into those problems.
06:35That being said, if we all do our job and we solve the customers problems 100% and the customer is very happy, that is only half of the equation, right?
06:45Not only do we have to solve the customer's problem, we also need to solve for our product.
06:50What do I mean by that, right?
06:51If we are taking it union of like the problems that the customers have and the products that we are building, right?
06:56Solving the problem only shifts like part of the Venn diagram.
06:59But in order to get that true feedback loop where we unify like the maximal overlap between what we're doing and what the customers need, that is forward deployed engineering at Cockroach Labs.
07:08So, the second part of this that I want to emphasize is that at our company, we map the customer problems back to the capabilities at hand.
07:16Now, how do we do that, right?
07:18A lot of engineering challenges come in similar shapes um from the field, right?
07:24We have the highest fidelity evaluation set that comes back from our customers, right?
07:28We are in the field every single day.
07:30We are hearing about the problems.
07:31We are hearing about whatever the customers are doing.
07:33And we have to take that context and bring it back to product in a sensible way.
07:38So, what are the ways in which these enterprise challenges, you know, manifest?
07:42are they common across the entire enterprise or are they unique to a specific user?
07:47Should like workarounds or like hacks or bugs in in in what we're building become features, right?
07:54Because ultimately what we want to do as a company is we want to de-risk our road map.
07:59At the end of the day, it would be great if like I as an engineer knew exactly what to build to get Y percentage of revenue from particular customer.
08:08And that's exactly what the problem is that we're trying to solve for because we are ultimately at the end of the day the heralds of the change, right?
08:14We are the bridge between products.
08:16We are the bridge between problems and the feedback is actually like half of the loop that makes the next deployment better than the previous deployment.
08:24So, if we think about the T-shape or personas of the folks that we hire, and for a deployed engineer, it's just so it's so flexible in terms of how you actually define it.
08:34Like, are you a sales engineer?
08:36Are you a solutions architect?
08:38What are you, right?
08:39So, if you think about all of the skills that FDEs typically are expected to have, right?
08:43You probably want to go wide.
08:45You probably want to have, you know, good people skill, like business skill, good process, customer skill, or even technology, right?
08:51In order to be deployed in the space, you need to know like what whatever the tech is.
08:55So, we also look for very deep spikes across people, right?
08:59Folks at Cognition deployed engineers at Cognition, we will hire them from like product management backgrounds if they have a really good sense of like how products are supposed to fit into each other, right?
09:10Because if the cost of software engineering is going to zero, you actually need to know how to like design a product that makes sense.
09:16Cuz you can just prompt it.
09:17But we also hire folks that are like founders, software engineers, but specifically like very spiky in the technology domains, right?
09:25It's fine if you don't have like the strongest business sense, that can be learned, but it's also very hard to teach like technicality and being the expert in the room while you're on the job.
09:33So, there's a couple of personas that we specifically hire for.
09:37Good customer engineers or deployed engineers do everything, right?
09:42Like they can map the product back to road map, like they can solve the customer's problems.
09:47Great customer engineers are able to actually have that relentless curiosity for why.
09:52So, at Cognition, we always ask ourselves, "Why are we solving this problem?
09:57Does this problem matter to the to the business as a whole?
10:00And can I communicate this back to the road map in a way that like improves the problem for everybody else?"
10:05But we also are The second mantra that we subscribe to is that you have to be relentlessly tied in to the customer, right?
10:13They are our lifeblood at the end of the day.
10:15Making them successful is the only way that you can survive as a business and become the obvious choice for an enterprise partnership.
10:21So, these are the two mantras that we specifically subscribe to as uh deployed engineers at Cognition.
10:26So, previously, um the first approach to deployed engineering was just, you know, token maxing, right?
10:33Uh the next era that I'll talk about is intelligent orchestration, but with like outcomes that you can actually measure, and I'll give you guys some examples.
10:41So, previously, like a year, maybe a year and a half, two years ago, uh the target or KPI for whatever deployed engineers were trying to do is maximize token usage, right?
10:53It was It was like the perfect time.
10:54Didn't have to worry about budgets.
10:56Everything was subsidized.
10:57You could just like run anything you wanted.
10:59But now, the problem has really shifted into the delivery space, right?
11:04A lot of organizations that we work with, some of the largest and most regulated enterprises in the world, they really care about, "Are we getting true value out of this solution,
11:13or are we just burning tokens for no reason, right?"
11:17And that is one core differentiator that I need to call out between us and some of the other platforms.
11:21You can make engineers like 10x faster.
11:23That's fine.
11:24That's still valuable.
11:25But can you make an organization 10x faster, including every single person that might be technical or non-technical uh across the company?
11:33That's when you unlock the true value of being the partnership.
11:36And that's why like single point tools that are just like CLIs or just IDEs, they fail to do that.
11:42So, let me just give you some proof points of how we've operated across the enterprise.
11:49So, at Cognition, um when we run the agent and the agent has like a specific trace or trajectory or like the agent does something, we call that a session.
12:00A session itself, right?
12:01We have metrics that allow you to derive how many engineering hours you can actually generate and how much how many engineering hours are actually productive that users are running.
12:10Right?
12:10So, one of these examples is a case study where we embedded ourselves within a customer for 3 months.
12:16We brought them on board.
12:17And functionally, over the course of those 3 months, we delivered about 150% like plus headcount.
12:23So, if you thought about like a project that you're trying to ship or something that you're trying to develop or a migration that you're trying to go through, imagine having
12:30150 extra coworkers doing that with you by your side.
12:35You might just say, "Hey, this actually just kind of looks like token maxing, right?
12:38Like you're just giving me a metric that says it's just, you know, engineering hours.
12:41You're like running a bunch of different sessions.
12:43Like, how do we know that these sessions are true, meaningful, and valuable?"
12:48So, the second part of that is, okay, we can think about how we've reduced timelines for delivery projects on an order of magnitude.
12:55So, about like 82% reduction across like delivery.
12:59So, if you subscribe to the agile deliver agile development methodology, obviously like you have tickets, you have sprints, like you have things that need to be built.
13:06If you look at every single metric that you measure before you bring in Devin and after you bring in Devin, you can take a look at that.
13:12We can compress this timeline by a factor of like 82%.
13:17So, across the board, whenever we get developed, whenever we get deployed, and fully activated within the customer environment, not only do we deliver a massive scale in terms of like engineering capacity,
13:28but we also reduce the time to value in terms of bringing things to market.
13:33Now, the third part of that is, hey, but I actually really care about the numbers, right?
13:36I actually really want to see how many PRs are you actually shipping?
13:39Like, is this meaningful?
13:41Does this actually make sense?
13:42So, if you think about dissecting the numbers a little like one dimension further, and you want to just look at like the raw PRs that people are ripping across the enterprise,
13:51we deliver almost double the amount of PRs that engineers were able to do with single-point tools and before you brought in an agent harness like Devin.
14:00So, there's three proof points of anonymized case studies in which we are able to deliver value at scale and across like various different problem domains.
14:10What I'll say is these aren't like, you know, private case studies.
14:14We have a bunch of different public case studies as well.
14:16So, we partner with companies like Nubank, right?
14:19If you folks have ever gone to Latin America, you can understand that, you know, the there's a lot of developers there.
14:25There's a lot of projects that are tangentially related.
14:27So, specifically, like we can say that there was an ETL migration.
14:30They had 50 engineers staffing this migration.
14:32We were able to deliver this within, um, I think like 1/3 of the timeline.
14:38Just with Devin autonomously.
14:40We have another bank, right?
14:42We have another use case where we work with one of the largest banks in Latin America.
14:45They were trying to migrate like the tax identification system.
14:49I know, like rocket science, right?
14:50Um, but the idea is that we were able to deliver this with half of the amount of effort actually required.
14:57So, if you think about like legacy languages like COBOL, if you think about things like JCLs, you think about things that like people don't learn anymore just because it's like not fun and not interesting,
15:05we're able to operate across some of the most complicated codebases in the world and deliver results that actually matter.
15:13And last but not least, obviously like if we think about the built card, um, specifically, we're able to actually merge like an order of magnitude more, um, in terms of like PR acceptance rate.
15:23We deliver like 10x per sub like worth of engineering talent like every single week.
15:27And then we're actually able to, you know, generate the weekly output of like over 10 engineers at the organization.
15:32Built has great engineers, by the way, right?
15:34These guys are so cracked.
15:36So, if you take one of these engineers and multiply them by 10, you just imagine the amount of returns.
15:41So, what I'll say is I'll I'll probably like round off this talk by just saying that at Cognition, we don't just, you know, embed ourselves in the customers.
15:49We don't just, you know, propagate feedback back to everybody else.
15:53But, the core values of the company are things and principles that we subscribe to not internally to the company, but external to the company as well, right?
16:00It's really fun being on the winning team.
16:02It's really fun when you come into an organization and say, "We can actually deliver so much cool stuff and like make people our champions, right?
16:09Whoever deploys Devin within the organization, they can show results that are essentially unmatched across the board."
16:16And we go for it all, right?
16:17We leave nothing on the table.
16:19We've deployed somebody in Brazil for like 10 months.
16:22[laughter]
16:23[snorts]
16:23To to live next to one of the customers to just make them successful.
16:27So, we're down for the mission.
16:29And it's it's more about like correctness, right?
16:30Like if if there are engineering practices that we want to fix, if there are things that we want to flag and raise, like these are all things that we take back to product
16:37and there's no ego involved.
16:39At the end of the day, we are all in the same boat.
16:41We're on the same mission and we're just shipping.
16:45And everybody essentially is go-to-market.
16:46I know forward deployed engineering is kind of like this fuzzy thing where it's like, "Am I part of sales?
16:50Am I part of post-sales?
16:51Like what do I actually do as an FDE?"
16:53But, everybody is go-to-market because the target is to make the customer successful at all costs.
16:59And at the end of the day, we just do things because every second counts.
17:03So, if you're interested in, you know, forward deployed engineering at Cognition, being the intersection between some of the hardest problems in this world, being part of like all of the software disruption at scale, and then being on the other side
17:15of these problems, we should talk.
17:18Thank you.