How Forward Deployed Engineering is done at Kepler — Vinoo Ganesh

AI Engineer · 22 min · 270 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12It's kind of an emotional thing to hear uh series of folks who've kind of been talking about how Palenter used to do things.
00:18Um it's funny in the olden days and this what this talk will be about is largely uh Palin's focus of FTE became a go-to market strategy but it wasn't that in the beginning how we were figuring out how to build foundry
00:33was through the lens of a product strategy and so my talk is going to be how we used FDE as a product strategy in 2013 to make the data platform that enabled Palunteer
00:44to then become the thing that Kevin Nat and all these folks were able to build on and it ultimately comes from like the background of something pretty simple.
00:52So my background is uh I started my career at Palanteer as a software engineer um really focused on horizontally scalable and storage solutions at a high level.
01:02Uh when we say forward deployed um I forward deployed to places like Iraq and Afghanistan.
01:07That's me in Bram.
01:08It's no longer there anymore.
01:10Uh but the idea was how do we take software and build it in the most critical environments to make it actually work for people that needed it the most.
01:19So forward deployed is actually a military term and we took it very seriously.
01:24Um my claim to fame from Palanteer is I built something called project frontline which was the rotation program that took our software engineers and made them forward deploy engineers.
01:34So the folks that you see at OpenAI, Anthropic, XAI, a lot of the FDs, about 350 of them went through a training program for how we got them to understand
01:45what we needed out of forward deployed product focused individuals.
01:49Uh built the same program at Citadel afterwards.
01:52So across the hedge fund, how do we build the right data products and software products to help portfolio managers generate alpha?
01:59Um this is a quote from the guy who named uh forward deploy engineering at Palanteer.
02:03It's from a couple days ago and um Sham is the CTO of Palunteer right now and I think this captures the entire essence of where my difficulty came from in the last talk which is
02:15the first fundamental truth of FTE is that this is not a role this is a product strategy how we discover the things to build are through the lens of forward deploy engineering.
02:25So looking at market caps of companies for how do we actually evaluate what contract size this is that wasn't how it actually happened but the core tenant and the core insight was
02:35an FTE is judged by their ability to be an extension of the product team to identify areas of opportunity and generalize product solutions out of it.
02:44And so this is the story of how we actually made project Frontline at Palunteer and what I'm using to build this function out at Kepler along with Susanna here is one of our founding engineers.
02:54Um back in the days, Palenter was first trying to get into big data.
02:59So around 2013, uh we had this idea that we wanted to store transaction information.
03:04And the product that came out of that was something called Phoenix.
03:08Phoenix was built totally in isolation.
03:10We didn't talk to customers.
03:11We didn't understand our financial banks or any anything else.
03:15What we did is we designed a system in perfect isolation that worked perfectly under certain circumstances and crashed and burned when we hit actual real data.
03:24And the situation was simple.
03:26We had a large scale financial customer and the idea was we're going to store your information in bucketed key spaces.
03:33Uh this is going to enable easy rolloff and easy retention.
03:37Now what we hit fairly quickly is a lot of banks and financial institutions have data that's not perfect.
03:44So when we hit a blank data value, it defaulted to the epoch January 1st, 1970.
03:50Our retention strategy made it such that what we wanted to do is actually take each 10-minute increment between January 1st, 1970 and 2013 and spin up a time bucketed window.
04:03The problem is Cassandra requires 5 megabytes per file handle, which means with the 2.3 million key spaces we generated to start up our server would require 14 terabytes of RAM and we were dead on arrival.
04:16There was no one else we could call.
04:18We were the only individuals and only institutions in the room.
04:21And we realized something fairly simple.
04:23The gap wasn't the fact that we had not looked for information about how customers use our product.
04:29It came from the ownership about co-building a piece of software without being directly embedded with a customer.
04:37The whole purpose of this talk is to convince you of one thing.
04:41Forward deployed engineering is a product strategy.
04:44Being an FD is an extension of the product function, not the go to market function.
04:48So what I'm going to do is walk through four things and four real stories that I learned at Palanteer and other places.
04:54what we learned from those experiences and how they informed building FTE at every other institution.
04:59The first starts from something fairly simple.
05:03Palenter was onboarding uh a large-scale dispatching and shipping company.
05:08We sat down with the VP of operations who generated a 47page requirements doc.
05:13What they wanted is a custom dashboard, 14 metrics, drill down alerts, this massive BI tool, a dev project that was estimated at three months.
05:23The kind of hilarious thing that happened in this process is we took them at face value.
05:28We spent four months scoping this engagement trying to understand exactly what this customer wanted before one of us actually showed up on site just by virtue of the fact that their family happened to live there
05:39and asked what's the first thing you do with this information Monday morning.
05:43The dispatcher said, "I check if trucks are trucks are late and then I call the dispatcher to ship a new inventory or new institution."
05:52The whole thing could have been simplified to a trivial Slack alert, which is what we ended up building.
05:58In 4 hours, we were able to solve this problem cradle to grave.
06:02The first move that we learned from forward deployed perspective is detect the real problem and ship the real thing.
06:08The vast majority of the time, people don't need massive BI dashboards.
06:12They don't need fully scaled designed pieces of software cradle to grave.
06:16They have a problem and they need that problem actually solved.
06:20And so the first thing that we learned is before you build anything, you as an extension of the product team need to understand three things.
06:28First, what are you trying to accomplish?
06:30Meaning what's the core goal in actually solving this problem?
06:34What happens after you have the solution?
06:37And how is the customer actually solving it today?
06:40every single one of these open uh AI or FD deployment companies are all approaching it in a slightly different way than how Palanteer did which is we have the people are going to go on site and build things.
06:51What do we build?
06:52So I think of this in terms of an XY situation.
06:56Customers describe solutions not problems.
06:59Your job as the FTE is to understand what the problem is.
07:03Customers don't know what happens next and your job as the FD is to define it.
07:08[snorts]
07:08My simple solution and takeaway is if solving the problem is under a day of work, just build it and ship it and close the loop.
07:15Don't make it a product strategy.
07:17Don't expand your product vision and bring in your PMs and everything else.
07:21If you can solve this problem in a short curtailed way, that's the first thing we did at Palunteer.
07:26That's how we started.
07:28The secret here is whoever defines the problem actually owns the solution.
07:32If we were able the virtue of the fact that we were able to look at the problem this shipping operator had define the solution and ship the actual solution
07:41meant that we were always the owners of that system and we were able to articulate what subsequent solutions looked like.
07:47By controlling what we're going to build and by controlling the narrative around the solution, you end up in a really powerful position.
07:54And this is why FDES actually become valuable in early sales.
07:57It's not because they are really good at talking to customers or we're not socially awkward software engineers as people seem to claim.
08:05It is because we actually solve the customer's problems in small bite-sized ways that wins us trust and that gets us access to the real problem that we can then use from a product strategy perspective to generalize.
08:18Foundry is a general product solution that we can use across a number of verticals that was informed by tens of thousands of hours of doing things like this in the field.
08:26Cool.
08:28Actions speak louder than words.
08:30This is a real situation also in Palunteer days.
08:33Um we had a data quality engineer who was viscerally adamant to a pipeline adamant about how to solve a pipeline stability issue.
08:43Every day we were dropping about a terabyte of information to someone's S3 bucket and it was putting a lot of pressure on on our own data pipelines.
08:51We had an easy solution.
08:53Instead of dropping huge numbers of CSVs, we can migrate everything to parquet.
08:58Makes it easier for our pipelines.
08:59Makes it easier for like your cost.
09:02Makes it easier from a compute perspective.
09:04But we had one engineer who was viscerally against this.
09:08We couldn't figure out why.
09:10Every time we'd bring this up over the course of about a year, she would push back.
09:14Whenever I'd ask, she'd say, "Park is way worse.
09:17It doesn't work.
09:18It doesn't make sense to me."
09:19We decided to actually go on site at this customer and watch her use our tool and do this workflow end to end.
09:27What we saw is she was downloading CSVs from S3 manually onto her Windows computer, double clicking them to open it up and doing a spot check of data quality.
09:37She could not do that with Paret files because Parquet didn't have a native reader that you could just open and view.
09:44kind of a hilarious problem if you think about it because she was so viscerally opposed to this without really understanding the benefit it would give her.
09:51That night we built a parquet viewer.
09:52She approved the migration in the next 2 days massively reducing data costs.
09:56I think the pipeline execution time went from 17 hours to about two.
10:01And what this teaches us it's actually something pretty simple.
10:04Uh when you are working with a user you are effectively an observer of what happens on site.
10:11An action speaks significantly louder than words.
10:14So you as an FTE should be looking for any task a user does more than once.
10:19Meaning if they're doing something the same day, same same time multiple times a day or they're doing something multiple times a week or multiple times an hour, that is a pattern and a hint that there's a problem and an opportunity.
10:30Anytime a user copies and pastes between tools, meaning if they're moving from one tool to another, anytime they're reacting this way, if the first thing you say is, "Hey, how do you feel about this problem?"
10:41And their reaction is, "Well, I have to."
10:44And this visceral exasperation, you know, you have an opportunity there.
10:48Anytime a user switches tools or tabs, it's an opportunity.
10:52And anytime a user pulls out their phone in the middle of a task, meaning they're going through a workflow using your software or anything else, and pull out their phone, it becomes pretty easy that this is an annoying problem or this software is taking too long.
11:06Your goal as an FTE is to make tomorrow, their tomorrow different from their today.
11:11And it starts with a question, what's the most annoying part of your morning?
11:14Every morning we all come into work.
11:16Every morning we all have something we need to solve.
11:18This all goes back to again FTE is informing your product strategy.
11:22You earn the right to extract user pain and define product strategy by solving small repetitive problems.
11:31Now the most valuable intel here is never going to be in documentation.
11:35We only saw Maria's uh challenge because we were badged onto their building.
11:40We were physically present.
11:41We were in Afghanistan.
11:42We were in Iraq.
11:43We were running around Soho.
11:44We were here in Palo Alto or in San Francisco.
11:48The vast majority of stuff that matters is only going to happen live inside of the walls of the office.
11:54Your badge on site at a customer site and your email address, your contractor email address, those are your data mining permits.
12:01The first thing every FTE should do is get access to customer information by getting in the room where things actually happen.
12:09Everything lives in that environment.
12:11And you can't survey your way to this.
12:13You have to be physically present.
12:14It's very easy to be a forward deployed engineer in name sitting in a nice conference room in New York, but that's not where the actual problems are and that's not where the solutions are.
12:23And the way I've heard this put and the way we threw it around a palunteer is residents get the truth.
12:28Go on site.
12:30Third, when you define the language, you control the narrative.
12:34We had a customer where I'm sure this occurs across pretty much everyone's company here.
12:38From a logistics perspective, everyone defines customers slightly differently.
12:44Sales members calls them customers.
12:46Ops calls them clients.
12:47Finance calls them billing entities.
12:49Devs call them org IDs.
12:51The way that we talk about the same entity in a customer is fundamentally different depending on what team you're in.
12:58That results in a huge amount of pain.
13:00It generally means integrations break.
13:02It means there's data quality issues.
13:04It means pipelines are constantly not working.
13:06And it ultimately comes back to a simple fact.
13:09We're all human.
13:10We define things differently because we operate in our environments and we understand how to talk about topics with people that are similar to us.
13:19It's a feature, not a bug.
13:20And so this idea of an ontology was kind of an accidental thing that Palunteer tripped into.
13:26And what it came from was not this understanding that by making everyone stick everything in elastic search and you know scheming everything we can solve every problem.
13:35came from the simple idea that we need to enable humans to operate in the domain that they understand best.
13:42And so when I say define the language, every organization is made up of two things, nouns and verbs.
13:49The nouns define the entities, the verbs define the operations.
13:53What FDES started eventually doing was building those nouns and verbs and figuring out what the terminology to use was.
14:01In a lot of these enterprises, users don't just adopt your product, they actually adopt your language.
14:06And what we were able to do is we're able to start defining the languages that we wanted people to use.
14:12I think in this case, we called them um uh c like I think we canonicalize on customers.
14:17In other situations, you canonicalize on ids.
14:20Something like DAU, daily active users can mean something very different to different people.
14:26For product teams, high quality DAUs.
14:28for anyone logging in from infosc just the number of people that logged in.
14:32All these phrases that we take for granted are illdefined by nature.
14:37And so when a user adopts your product, they also adopt your language.
14:40How many people are calling things skills right now?
14:43How many people are calling things MCPs that are function calls with prompts?
14:47And that matters.
14:49So third piece here, defining the ontology.
14:52Your job as an FTE is to understand what the ontology is.
14:56What phrases, nouns, and terms in an enterprise are overloaded?
15:00Meaning, what words are people using to describe the same concepts?
15:04Second, where do the integration points exist in the system?
15:07Are things going from Snowflake to data bricks?
15:10Are they going from Palunteer to Tableau?
15:12Are they going from Anthropic to SAP?
15:15Where do these integration points actually exist?
15:17And what are the translation layers where it goes from one term to the other?
15:21Naturally, what are the system boundaries?
15:23What are the systems of record you're never going to be able to switch out?
15:26And what words do people use when they're describing their issues?
15:30If someone is saying, well, in AI, my agent keeps failing.
15:33What is an agent?
15:34Like we can't even define FTE.
15:36We're trying to define agents, right?
15:37Are they prompts?
15:38Are they like series of steps?
15:40Understanding that viscerally becomes important.
15:43Your job is to define the terms.
15:46If you can define the terms in your own solution, you can build the ontology and you can have customers answer questions and think through things in your terms.
15:56So the the secret here is if you become the linguistic foundation, you're locked in.
16:02And simple examples are things like skills, things like MCPs, things like any of the terms that weren't a big deal a year ago but are now day-to-day in our entire econ entire ecosystem.
16:13And so when you are able to define the vocabulary that users use and codify that in your platform, you become the foundation under which every solution and tool is built on top of
16:23foundry enabled that ontology to be constructed and built.
16:26That then enabled the next generation of FTEEs, the go-to market FTEEs to go out there and just do data integrations and sell the product.
16:35We had to learn this the hard way when we were actually building it for the first time.
16:40Last thing here, ship fast but build for production.
16:43This is a true story.
16:44Um, they all are, I guess.
16:45But, uh, we had a customer that needed to run data retention.
16:49I decided to build a very quick script.
16:51It was in Groovy, if anyone knows that language, as a temporary fix.
16:56Um, it wasn't designed for prod by any means.
16:59Like, I just randomly hacked this thing together to solve an initial problem, but it made it there.
17:0412 months later, this thing was all over the place.
17:06It was a almost 100,000 person customer.
17:09Uh it was running in a number of places for my then and my nickname at Palanteer Nat's not here anymore but she can attest to this became venue.groovy.
17:17My wedding people showed up wearing a name the shirt venue.groovy.
17:22And so the the kind of funny thing here is we solved a problem in a very hacky way that did fix an issue but we didn't actually productize it.
17:32We didn't think through the end state.
17:34So we ended up in a situation where we were forced to support a very hacky product for years that was never actually productized.
17:41And so again when people say customer FTEES are forward deployed software engineers or customerf facing software engineers and your job is to make the customer successful that's not true.
17:51When you do that things like this happen so scripts and hacks can fix problems but they're not driving your product strategy forward.
17:58Meaning you're not doing your job successfully as an FTE.
18:02We have people that jobs are making customers successful.
18:04They're solutions architects.
18:07So calibrating what you ship becomes the last mo most important skill here as an FTE.
18:13First, am I going to get a 2 a.m. phone call about this in 6 months?
18:16If you are, you probably shouldn't ship that thing.
18:19Second, what is my trade-off for solving this problem quickly?
18:22Am I making the right decision for the product as a whole?
18:25Or am I just solving this customer's painoint in a way that's going to have potentially small wins but negative compounding challenges later on?
18:34Who do I hand this off uh to when I actually leave the customer site?
18:37Is it going to a customer?
18:38Is it going to another institution?
18:40Is it going to another FDE?
18:42What happens when this breaks?
18:44Am I going to be held responsible for that?
18:45Am I building something so missionritical that if something breaks, I'm going to be hated?
18:50what happens there.
18:52So, we need to solve problems, but we need to be aware of when we fold those solutions into the core offering and when we should just discard those solutions quickly.
19:02Here's the actual reality.
19:04Every hack goes into production.
19:06If you make someone's life easier, it will go into production and you will be responsible to support that hack in perpetuity.
19:14The most dangerous words in forward deployed engineering or engineering is this is just temporary.
19:19Anyone here who's been at a software company knows this is not just temporary.
19:23And we know that we're still writing running on 40-year-old cobalt code at some IBM mainframe because of a hack that was put in place.
19:31If it solves a problem, it's not temporary.
19:33It will live forever.
19:35So, as you're thinking from an FTE perspective, ship everything like it's going to run for 18 months because it probably will.
19:42and through the lens of again the product building focus should tell you a lot about what goes into the core product offering versus what wins you customer goodwill.
19:51So the cheat sheet that we all ran frontline people to uh through rather is the following.
19:58Most people who are trying to develop an FTE function treat it like a product management task and a customer success uh operational task.
20:08They take requirements, they schedule user research, they add insights, they run the processes without actually figuring out how you steer the product from the insights.
20:17The right FTEEs who enabled the creation of something like Foundry redefined the problem.
20:23They got badged on site and they flew to places like Afghanistan, like Iraq, like Somalia, like any of these places where we had to be where our customers were.
20:32I I have a friend who was spending time on an oil rig in the middle of the ocean.
20:36They shipped the fix before leaving the customer site to win the goodwill, but they owned the fix end to end in the product ecosystem.
20:44And they took those fixes and created product leverage from it.
20:48They were able to translate the problems they saw into nouns and verbs that they were then able to use to define the foundation of every subsequent problem that was solved.
20:58And finally, they were the ones who got the call when things actually broke.
21:01They were the ones who got in the airplane to fly everywhere else.
21:04The question is not what did they learn in the process, but what did they ship from the product perspective?
21:10The goal here is actually fairly simple.
21:12I'm going to wrap up with this.
21:14Product leverage is the only thing that wins you customers.
21:17It's the only thing that backs every Citadel portfolio manager trading hundreds of millions of dollars.
21:22It's the only thing that backs every decision being made in the war fighter ecosystem at Palunteer or in customer ecosystems at Palanteer.
21:30your job as an FTE and what we learned and what Kepler is doing is using FTEE as an extension of the product function to enable us to build products that are actually sticky and that solve problems.
21:43And if you were to take one thing away, please don't treat FTEEs as go to market extensions.
21:48You can do that when you're palunteer and have 20 years and unlimited money.
21:52You don't do that when you're an early stage company and you need to figure out what product will allow you to use those FDs to maximum degree.
22:00Thank you so much.
22:01[applause]