WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ITMXwI6QL6A

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ITMXwI6QL6A.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.160 --> 00:00:13.920
Awesome.

s3
00:00:13.920 --> 00:00:14.600
Thank you, guys.

s4
00:00:14.600 --> 00:00:14.840
Awesome.

s5
00:00:14.840 --> 00:00:16.080
It's great to to meet everyone.

s6
00:00:16.080 --> 00:00:20.720
I mean, I hope that after the talk, you know, if you want to come out and we can chat, we'd love to.

s7
00:00:20.720 --> 00:00:22.000
Um Cool.

s8
00:00:22.000 --> 00:00:32.560
So, yeah, today my goal is to share with you guys the two most important principles from what we learned doing FDE at Ramp.

s9
00:00:32.560 --> 00:00:34.520
So, just yeah, briefly a little bit about myself.

s10
00:00:34.520 --> 00:00:36.600
Yeah, I'm a director of engineering at Ramp.

s11
00:00:36.600 --> 00:00:44.360
Uh I joined the company 2 and 1/2 years ago when it was just you know, FDE was just two engineers at the time.

s12
00:00:44.360 --> 00:00:55.920
And today, my org is about 30 engineers across four deployed developer API and our new AI services um business.

s13
00:00:55.920 --> 00:01:00.160
So, I know this is kind of a running theme, but like no one knows what FDE is.

s14
00:01:00.160 --> 00:01:03.240
So, I'm just going to spend a moment on that.

s15
00:01:06.240 --> 00:01:08.760
So, yeah, I I I actually kind of like this meme.

s16
00:01:08.760 --> 00:01:10.520
It's To me, it's kind of funny.

s17
00:01:10.520 --> 00:01:14.240
Um I um but I I I actually think it's like totally wrong.

s18
00:01:14.240 --> 00:01:19.280
I don't see this as the actual like true form of what FDE is.

s19
00:01:19.280 --> 00:01:27.760
Um I don't see it as like the final evolution or like boss mode of technical go-to-market roles.

s20
00:01:27.760 --> 00:01:35.520
Um now, this might be true at some companies, but at least at Ramp, it's a little bit different.

s21
00:01:35.520 --> 00:01:41.000
So, FDE at Ramp, uh we live within the engineering organization.

s22
00:01:41.000 --> 00:01:45.200
And our goal is to help Ramp win upmarket.

s23
00:01:45.200 --> 00:01:56.560
So, with that in mind, what we do is we basically work on the core product and our new agentic features and make them work really well for our largest

s24
00:01:56.560 --> 00:01:59.080
enterprise customers.

s25
00:01:59.400 --> 00:02:02.000
So, that's just a little bit of intro context.

s26
00:02:02.000 --> 00:02:06.479
I want to dig in and and today, like I said, there's just two things I'm going to share with you.

s27
00:02:06.479 --> 00:02:07.280
Literally two things.

s28
00:02:07.280 --> 00:02:09.680
Very easy talk.

s29
00:02:10.200 --> 00:02:16.160
And these are the principles that I would say have really guided us and I would say probably the two most important things that we have.

s30
00:02:16.160 --> 00:02:20.959
Always be scoping and scale with tokens.

s31
00:02:21.320 --> 00:02:23.280
So, let's get with start with the first one.

s32
00:02:23.280 --> 00:02:25.480
On scoping.

s33
00:02:25.480 --> 00:02:33.920
So, I would say there's this thing where like people many many people think that as an FDE, your job is to just say yes to the customer.

s34
00:02:33.920 --> 00:02:36.000
But that's wrong.

s35
00:02:36.000 --> 00:02:45.120
If you were just to say yes, you know, instead of like beautiful Waymos that we have driving us around in San Francisco, you'd have something like this, you know.

s36
00:02:45.120 --> 00:02:48.840
Yeah, horses with like rockets strapped to their legs.

s37
00:02:48.840 --> 00:02:55.040
And the point is you you want to help the customer be successful.

s38
00:02:55.040 --> 00:02:57.520
You want to try to figure out a way to say yes.

s39
00:02:57.520 --> 00:02:59.800
But you actually want to deliver good software.

s40
00:02:59.800 --> 00:03:01.720
You need to build the right thing.

s41
00:03:01.720 --> 00:03:04.519
So, you don't just endlessly say yes to people.

s42
00:03:04.519 --> 00:03:15.400
And I I do want to share an example of something that I would say happens somewhat regularly in one form or another at Ramp.

s43
00:03:15.400 --> 00:03:32.080
So, it's Friday night and an enterprise sales rep comes to us with an urgent request that this super important strategic logo is only going to close if we build out an SAP S/4HANA integration.

s44
00:03:32.080 --> 00:03:37.160
And I think that the default engineering reflex is like, "Shit.

s45
00:03:37.160 --> 00:03:39.600
Like, where are the SAP API docs?

s46
00:03:39.600 --> 00:03:42.920
Like, where do I find them and how do I build this integration?"

s47
00:03:42.920 --> 00:03:49.519
But what an what a well-trained FDE would do is like pause for a second and say, "Okay.

s48
00:03:49.519 --> 00:03:52.480
First of all, like, what's driving the urgency here?"

s49
00:03:52.480 --> 00:04:03.560
Like, one thing I've seen is I've seen sales reps who like go kind of crazy because it's like the end of the quarter and they're trying to hit their quota and close the deal and not because the customer is the one driving the urgency.

s50
00:04:03.560 --> 00:04:05.627
So, that's like one example.

s51
00:04:05.627 --> 00:04:06.320
[clears throat]

s52
00:04:06.320 --> 00:04:16.840
But, you know, as an FDE, you're asking tons of questions to gather context about what's important um and what actually is the right thing to build.

s53
00:04:16.840 --> 00:04:21.120
And so, you might ask like, "Who's using this integration?

s54
00:04:21.120 --> 00:04:23.080
Have we exhausted all the different workarounds?

s55
00:04:23.080 --> 00:04:25.320
Is there something manual that we can do in the meantime?

s56
00:04:25.320 --> 00:04:27.800
Does the customer have like technical resources?

s57
00:04:27.800 --> 00:04:32.240
Can they hit our API such that we don't have to build this thing?"

s58
00:04:32.240 --> 00:04:47.000
But, I'd say the most important thing that an FDE does is also looks beyond this one request and looks at the other prospects that are coming down the pipeline and other customers to see if anyone else would benefit

s59
00:04:47.000 --> 00:04:48.760
from this as well.

s60
00:04:48.760 --> 00:04:57.800
And the point is that by gaining all this context, you can do a better job of building the right thing.

s61
00:04:57.800 --> 00:05:02.600
So, I want to share another story that was really really painful for us in the early days.

s62
00:05:02.600 --> 00:05:09.880
We had this large enterprise customer and they needed this reimbursement feature on mobile.

s63
00:05:10.000 --> 00:05:13.880
Unfortunately, our mobile team was totally swamped.

s64
00:05:13.880 --> 00:05:20.520
So, we basically just had to roll up our sleeves as FDEs and just get find out a way to get things done.

s65
00:05:20.520 --> 00:05:25.919
And we had two of the engineers on the team just like learn how to do iOS and Android development.

s66
00:05:25.919 --> 00:05:27.520
And it was awesome.

s67
00:05:27.520 --> 00:05:28.200
We were super excited.

s68
00:05:28.200 --> 00:05:29.880
We're like, "Okay, we're going to ship this feature.

s69
00:05:29.880 --> 00:05:30.880
It's going to be so good.

s70
00:05:30.880 --> 00:05:32.400
Like, hell yeah."

s71
00:05:32.400 --> 00:05:36.800
So, we grinded for a couple weeks, got the feature done on both platforms.

s72
00:05:36.800 --> 00:05:39.160
And we go to the customer and we're like, "Awesome.

s73
00:05:39.160 --> 00:05:44.480
Like, can you send us your list of, you know, beta users for Android?"

s74
00:05:44.480 --> 00:05:54.200
And that's when they told us they only they they they require they mandate all of their employees to use iOS devices.

s75
00:05:54.880 --> 00:06:00.640
So, you're like, what the Like Not not to the customer, you know, just internally.

s76
00:06:00.640 --> 00:06:04.320
But like obviously it was super disappointing for us because we'd put all this effort in.

s77
00:06:04.320 --> 00:06:08.840
And so, it was a a big lesson for us to remember the importance of scoping.

s78
00:06:08.840 --> 00:06:20.280
Even some of the most basic assumptions like which you know mobile platform you build on it's it's super important um to validate them and and thus kind of emphasizes the importance of scoping up front.

s79
00:06:20.280 --> 00:06:26.480
Now, Okay, so let's say that you and your team have become masters of scoping.

s80
00:06:26.480 --> 00:06:28.520
You know, you're you're amazing.

s81
00:06:28.520 --> 00:06:32.560
In today's world, this is not enough.

s82
00:06:33.240 --> 00:06:40.320
So, unless you are scaling with model capabilities, you are going to fall behind.

s83
00:06:40.320 --> 00:06:41.800
Now, I'm not going to belabor this point too much.

s84
00:06:41.800 --> 00:06:52.760
I think like every talk in this uh in in this conference is probably some flavor of this, but like the point is that we basically have to reinvent our jobs constantly now.

s85
00:06:52.760 --> 00:07:01.240
So, whatever work we are doing today, you know, for the most part it's knowledge work, we have to figure out how to have models and agents do it for us.

s86
00:07:01.240 --> 00:07:13.040
And so, that brings me to the second half of this talk and the the other point that I want to convey today, which is all of us have to figure out how to scale with tokens.

s87
00:07:13.040 --> 00:07:21.840
And the way that I interpret scaling with tokens for FTE is take a look at the whole life cycle of what an FTE does.

s88
00:07:21.840 --> 00:07:33.560
From gathering context to scoping out a request to writing out a spec and then implementing the feature, each stage of that pipeline can be replaced with agents.

s89
00:07:33.560 --> 00:07:35.840
And at first it seems kind of daunting.

s90
00:07:35.840 --> 00:07:39.360
You're like, like how are you going to go and approach and like solve that?

s91
00:07:39.360 --> 00:07:44.120
But if you break the problem down and then make progress on it, it's it's actually pretty tractable.

s92
00:07:44.120 --> 00:07:47.800
And so, I'll share share with you guys one example of something Oops.

s93
00:07:47.800 --> 00:07:49.920
Something that we um that we've done at Ramp.

s94
00:07:49.920 --> 00:08:09.560
So, we have this internal Slack channel called FDE requests, and this is where account managers, solutions, uh sales reps will post whenever there is a blocker for a prospect or customer that's large enough, basically.

s95
00:08:09.560 --> 00:08:11.440
And so, we get these requests.

s96
00:08:11.440 --> 00:08:16.560
In this case, actually uh one of the CSMs on our team, Greg, posted here.

s97
00:08:16.560 --> 00:08:20.240
And um if you were to It's actually a Notion workflow.

s98
00:08:20.240 --> 00:08:22.240
If any of you work at Notion, by the way, thank you.

s99
00:08:22.240 --> 00:08:24.160
We like use Notion so much.

s100
00:08:24.160 --> 00:08:31.400
Um if you were to click open in Notion, you'd see like a pretty long request that has all the details of what what exactly it is.

s101
00:08:31.400 --> 00:08:33.880
And the problem is there's a super high variance.

s102
00:08:33.880 --> 00:08:39.840
Like, some people will submit like really detailed, good requests from the customer.

s103
00:08:39.840 --> 00:08:46.560
And others are just going to submit like one line, like, "We need uh you know, we need this SAP integration."

s104
00:08:46.560 --> 00:08:53.480
And before what would happen is we would have FDEs manually kind of go through this request.

s105
00:08:53.480 --> 00:08:59.440
We'd read the whole thing, understand it, figure out what exists in the product, do a bunch of back and forth with the customer.

s106
00:08:59.440 --> 00:09:04.080
And this is like exactly what the first half of the talk was about, always be scoping.

s107
00:09:04.080 --> 00:09:11.280
You know, we would spend a lot of time really digging in and validating what exactly was uh you know, absolutely necessary.

s108
00:09:11.280 --> 00:09:24.200
And so, you can see here what we what we did then was we basically um used Notion uh Notion agents to build a V1, which literally just took the request and asked a couple of questions.

s109
00:09:24.200 --> 00:09:25.480
That was it.

s110
00:09:25.480 --> 00:09:30.080
And um after It was kind of astonishing.

s111
00:09:30.080 --> 00:09:39.040
Literally, after a couple of weeks, we found that it was like saving us a lot of time because, first of all, immediate like the latency of replies went from like hours or days to like, you know, seconds.

s112
00:09:39.040 --> 00:09:45.560
And immediately, like the account reps that the account managers, the reps would start kind of engaging with this agent.

s113
00:09:45.560 --> 00:09:51.160
And one of the things that we did was because it went so well, this this is actually um the more recent iteration of it.

s114
00:09:51.160 --> 00:09:55.680
It's very cute, you know, the little penguin actually helps make it seem a little more friendly and approachable.

s115
00:09:55.680 --> 00:10:09.000
Um and what it does is it actually goes and does several rounds of back and forth questioning with the submitter until it deems that it's ready to create a lot a spec, basically.

s116
00:10:09.000 --> 00:10:12.880
And it's actually been incredible how helpful this has been for us.

s117
00:10:12.880 --> 00:10:22.360
I I would say it's probably saved us like a large percentage, I don't know, 20% of the time that we'd spend on scoping out these requests.

s118
00:10:22.640 --> 00:10:26.400
So, you know, this is this is a great example.

s119
00:10:26.400 --> 00:10:27.800
For us, this has been really helpful.

s120
00:10:27.800 --> 00:10:29.120
It's I'm super excited about this.

s121
00:10:29.120 --> 00:10:31.920
It's going to help automate a lot of the work that we've been doing manually.

s122
00:10:31.920 --> 00:10:37.480
But, um it's really just the first stage of this pipeline that I was alluding to.

s123
00:10:37.480 --> 00:10:44.200
So, if you look at the first part here, like we've been able to make some progress on it.

s124
00:10:44.200 --> 00:10:53.320
The last step as well, going from a a well-shaped spec to like a working product, obviously like Frontier models can like one-shot medium-size features.

s125
00:10:53.320 --> 00:10:56.520
And so, the last part is also is is a lot easier for us.

s126
00:10:56.520 --> 00:11:03.760
It's this middle part that I would say is super like gnarly and like unformed and difficult.

s127
00:11:03.760 --> 00:11:14.080
And I'm I'm really excited about our team kind of investing a lot more and spending a lot more of our time just like building out this factory, building out agents to replace each one of these steps.

s128
00:11:14.080 --> 00:11:24.400
And the thing is, if you look at if I were to say 6 to 12 months from now, like what does FDE at Ramp look like?

s129
00:11:24.400 --> 00:11:30.000
Like these are the sorts of applied AI problems that we're going to be spending all of our time on, I think.

s130
00:11:30.000 --> 00:11:36.360
Like, you know, making sure that the agent harness that's running each of those steps is running super smoothly.

s131
00:11:36.360 --> 00:11:43.680
Um making sure that the the output quality of each of the outputs of the the pipeline is is actually good.

s132
00:11:43.680 --> 00:11:48.240
That you know, with with evals, with rubrics, with human feedback.

s133
00:11:48.240 --> 00:11:57.560
Um and there's of course like one of the biggest challenges, which is getting your agent the right context, you know, when you're making the alarm call, ensuring that it has the right context.

s134
00:11:57.560 --> 00:12:01.200
So, there's like a lot of historical data, data about the the product.

s135
00:12:01.200 --> 00:12:05.720
Imagine like all the knowledge that a product manager has in their head about their product.

s136
00:12:05.720 --> 00:12:08.040
Like, how do you get that into an agent?

s137
00:12:08.040 --> 00:12:13.800
Like, Notion docs and all your existing knowledge base and help articles only give you so much of that.

s138
00:12:13.800 --> 00:12:16.160
Um yeah, skills, memories, tools.

s139
00:12:16.160 --> 00:12:28.680
I could go on for a bit, but ultimately the most important thing here is that as an FD, we still have the responsibility of taste and judgment over the final output.

s140
00:12:28.680 --> 00:12:33.000
So, that's going to be like the underlying kind of throughline.

s141
00:12:33.480 --> 00:12:35.040
Okay.

s142
00:12:35.040 --> 00:12:40.200
So, let's say that you've done an amazing job building out this factory.

s143
00:12:40.200 --> 00:12:44.040
But the problem isn't to tie this to the first half of the talk.

s144
00:12:44.040 --> 00:12:56.600
If you don't do a good job of scoping out requests or or building upon the principles of scoping things well, you're going to get a token maxing slop cannon.

s145
00:12:56.600 --> 00:13:04.200
And so, the whole point is that you have to do these both because the other way around is actually quite bad as well.

s146
00:13:04.200 --> 00:13:14.960
If you are, you know, amazing at scoping, but don't invest in building out this, you know, agent factory, you know, it's going to be over for you.

s147
00:13:14.960 --> 00:13:21.800
Like, uh your your agent native competitors are just going to overtake you and outcompete.

s148
00:13:21.800 --> 00:13:36.320
And so, that's why um in the end here, I want to close with the the the most important thing is that if you have both of these, it can set you up for success in the future.

s149
00:13:36.320 --> 00:13:41.160
Always be scoping and scaling with tokens.

s150
00:13:41.160 --> 00:13:43.720
The future of FD needs both.

s151
00:13:43.720 --> 00:13:44.160
That's all.

s152
00:13:44.160 --> 00:13:45.964
Thank you, guys.

s153
00:13:45.964 --> 00:13:47.964
[applause]

s154
00:14:02.339 --> 00:14:04.339
[music]