WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Emo5FGGY-wM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Emo5FGGY-wM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:13.840
Can everybody hear me?

s3
00:00:13.840 --> 00:00:16.160
Okay, perfect.

s4
00:00:16.160 --> 00:00:19.119
So, first thanks for joining everyone.

s5
00:00:19.119 --> 00:00:26.400
Uh, my name is Filipe and I am a member of technical staff at the Amazon AGI lab.

s6
00:00:26.400 --> 00:00:31.199
Um, as part of my role, I help us to build the next generation.

s7
00:00:31.199 --> 00:00:35.760
I manage the programs to build the next generation of products our lab is building.

s8
00:00:35.760 --> 00:00:41.440
And today I will talk to you about designing evolves that earn customer trust.

s9
00:00:41.600 --> 00:00:47.600
Before talking about evolves, I just need to give you some context about what our lab was doing the past year.

s10
00:00:47.600 --> 00:00:54.320
So everything started in March last year where we launched our Nova research preview.

s11
00:00:54.320 --> 00:01:03.120
for context like Nova is a tool that we build as part of our lab and it's a tool that helps you to build browser agents.

s12
00:01:03.120 --> 00:01:09.760
So basically everything that you do in a web browser you can use Nova to automate those tasks.

s13
00:01:09.760 --> 00:01:20.000
Uh and as part of the research preview our main goal was we wanted to ship a product to get as much feedback as we could from our customers.

s14
00:01:20.000 --> 00:01:22.720
Uh in March we launched that.

s15
00:01:22.720 --> 00:01:34.960
Then in July we made one additional improvement in our product which was basically develop a whole new developer experience and this was because basically during the first couple months

s16
00:01:34.960 --> 00:01:43.040
we noticed like some gaps in the developer experiment experience and we wanted to make sure customer had the best experience building agents.

s17
00:01:43.040 --> 00:01:48.399
Then in December we finally launched our service on AWS.

s18
00:01:48.399 --> 00:02:00.320
So today Novax is a top tier service on AWS that anyone can use to build production level agents and during this period I was basically responsible for the customer enablement

s19
00:02:00.320 --> 00:02:01.280
piece of that.

s20
00:02:01.280 --> 00:02:03.759
What's customer enablement just for context?

s21
00:02:03.759 --> 00:02:13.360
It's basically I was working daytoday with all of our customers to understand what was their problem and how we were helping them to solve that.

s22
00:02:13.360 --> 00:02:18.160
So that was my main uh role uh during last year.

s23
00:02:18.160 --> 00:02:23.120
And then let's go to uh the problem that we will be discussing today.

s24
00:02:23.120 --> 00:02:26.800
So I like to call this is the benchmark illusion.

s25
00:02:26.800 --> 00:02:30.080
So you all working on AI.

s26
00:02:30.080 --> 00:02:36.000
You are probably working on and building things based on benchmarks, right?

s27
00:02:36.000 --> 00:02:49.599
It could be public benchmarks that everybody's using and everybody's trying to optimize the products on that and also benchmarks that you are creating your own synthetic data that you are creating and you are evaluating your product.

s28
00:02:49.599 --> 00:03:04.159
Uh and the main goal here is you want to optimize right you want to be the best as you can there in those benchmarks but then when you ship that to production and your customers start using that what happen is it's this right

s29
00:03:04.159 --> 00:03:15.200
they start seeing problems and it's basically everything that you were not expecting them to do they try to do that and they break your product and this is happening

s30
00:03:15.200 --> 00:03:22.080
because you basically most of us we are doing we are working on evals that that are static.

s31
00:03:22.080 --> 00:03:24.640
So what are static evals?

s32
00:03:24.640 --> 00:03:37.360
Basically let's talk a little bit about what's the standard development process of a product right first thing that you do you receive uh the requirements from your product chain and you start building

s33
00:03:37.360 --> 00:03:48.879
right everybody does that I hope then when you have a product in a good shape the next thing that you will do is running your evals again public benchmark

s34
00:03:48.879 --> 00:03:59.280
synthetic data that you generated and you optimize for that when you start feeling comfortable you're like okay now I will ship that and you ship that to the customers

s35
00:03:59.280 --> 00:04:13.920
but if you have a static evals what's the only thing that you can do after that it's basically this is hope right you just hope your evals were reflecting exactly what your customers would do with your product

s36
00:04:13.920 --> 00:04:24.800
and if not it will start failing right and this is the gap this is exactly what we will be discussing today and what we were doing in our lab to solve this problem.

s37
00:04:24.800 --> 00:04:30.080
And just a spoiler, so this is not about better benchmarks.

s38
00:04:30.080 --> 00:04:32.479
It's about closing the customer loop.

s39
00:04:32.479 --> 00:04:38.560
So what we want you to do here is get the feedback from your customer and close the loop.

s40
00:04:39.600 --> 00:04:46.479
So with that in mind, what we did in our lab was basically propose an eval flywheel.

s41
00:04:46.479 --> 00:04:56.560
So if you are not aware about the terminology but flywheel is basically a loop that you keep repeating like all the time and the first step of this evolve

s42
00:04:56.560 --> 00:05:01.040
is define success and very important here.

s43
00:05:01.040 --> 00:05:07.520
So define success here it's not what you think it's success it's what your customer think it's success.

s44
00:05:07.520 --> 00:05:09.280
Okay.

s45
00:05:09.280 --> 00:05:19.759
So next step as soon as you understand what your customer really needs and the problem you are trying to solve you go to capture signals and we as engineers

s46
00:05:19.759 --> 00:05:30.479
what's the first thing that comes to your mind it's like instrumentation right let's get metrics let's get data as much as you can about what your customer is doing

s47
00:05:30.479 --> 00:05:44.560
but one very important piece of that that we are missing is you should talk to your customer like talk to your customer go schedule meetings uh visit them and understand exactly what they are trying to do.

s48
00:05:44.560 --> 00:05:47.120
Your data might be a little bit misleading.

s49
00:05:47.120 --> 00:05:52.320
You might be missing exactly what's the problem that your customer is trying to solve.

s50
00:05:52.320 --> 00:05:55.440
Third step, you go to diagnose gaps.

s51
00:05:55.440 --> 00:05:56.160
That's easy.

s52
00:05:56.160 --> 00:05:59.039
That's basically you get all these signals from customers.

s53
00:05:59.039 --> 00:06:04.960
Could be instrumentation but also your inputs, your insights from the meetings that you had with them.

s54
00:06:04.960 --> 00:06:07.360
And you just start categorizing those.

s55
00:06:07.360 --> 00:06:14.800
And when I say categorizing there are basically very high level but there are three things that you could find from those discussions.

s56
00:06:14.800 --> 00:06:17.919
First it's basically inputs about your model.

s57
00:06:17.919 --> 00:06:23.840
So where your model is failing there and this feeds into evals right to your model.

s58
00:06:23.840 --> 00:06:28.400
The the second point it's inputs to your engineering team.

s59
00:06:28.400 --> 00:06:33.120
So basically what uh is the gap in your hardness that you should fix.

s60
00:06:33.120 --> 00:06:35.280
This is more traditional bug fixing.

s61
00:06:35.280 --> 00:06:38.800
And the third one is also on the product side.

s62
00:06:38.800 --> 00:06:46.400
So what are you missing in terms of product like do your customer understand exactly what your product is trying to solve?

s63
00:06:46.400 --> 00:06:49.360
Is your product like properly positioned?

s64
00:06:49.360 --> 00:06:51.360
So that's very important.

s65
00:06:51.360 --> 00:06:58.560
So as soon as you have the gaps di the gaps categorized you just feed that into decisions.

s66
00:06:58.560 --> 00:07:06.240
So feed that into decisions is basically you prioritize that and you ask your science, your engineering, your product team to address those.

s67
00:07:06.240 --> 00:07:07.599
That's basic, right?

s68
00:07:07.599 --> 00:07:12.639
And then you just continue repeating this loop over and over.

s69
00:07:13.599 --> 00:07:21.199
So with that in mind, like in our lab, we repeated this flywheel several several times since March last year.

s70
00:07:21.199 --> 00:07:33.919
And what I also noticed is basically as part of my journey as a product, each of my c each customer will have their own journey using my product.

s71
00:07:33.919 --> 00:07:35.360
What that means?

s72
00:07:35.360 --> 00:07:40.160
So my customer evolves as they use my product, right?

s73
00:07:40.160 --> 00:07:46.160
So there are basically four things that you can learn while your customer use your product.

s74
00:07:46.160 --> 00:07:51.759
The first one is that's the first thing that you will do the first time that you meet a customer.

s75
00:07:51.759 --> 00:07:54.000
It's basically use case discovery.

s76
00:07:54.000 --> 00:08:00.560
So what you want to do here at this first step it's basically understand what's their problem.

s77
00:08:00.560 --> 00:08:06.400
So what's the problem of your customer and make sure what you are building solves that.

s78
00:08:06.400 --> 00:08:07.520
Right?

s79
00:08:07.520 --> 00:08:11.840
Then after that you will have couple early adopters.

s80
00:08:11.840 --> 00:08:12.479
Right?

s81
00:08:12.479 --> 00:08:16.960
from these early adopters the main signal that you want to get.

s82
00:08:16.960 --> 00:08:19.280
It's basically what's possible.

s83
00:08:19.280 --> 00:08:23.120
So that's the time where customer will try crazy things.

s84
00:08:23.120 --> 00:08:27.039
You will find things that you cannot help them at at all, right?

s85
00:08:27.039 --> 00:08:36.560
But you will also find things that your customers are using your product to solve and many times you will be surprised because it's something that you would never imagine.

s86
00:08:37.120 --> 00:08:40.320
Third point, this is when you start scaling, right?

s87
00:08:40.320 --> 00:08:43.440
You get your early adopters but now you start scaling.

s88
00:08:43.440 --> 00:08:49.040
you are getting more customers and you are able to identify some patterns, right?

s89
00:08:49.040 --> 00:08:50.399
What are the patterns?

s90
00:08:50.399 --> 00:08:54.080
The pattern is basically your hero use case, right?

s91
00:08:54.080 --> 00:09:00.000
That's the case, that's the problem that everybody is using your product to solve.

s92
00:09:00.000 --> 00:09:05.440
And last one, this is more future looking, but these are your product gaps.

s93
00:09:05.440 --> 00:09:14.720
So when you were working with early adopters and also after you start scaling you will also be finding some gaps on your product.

s94
00:09:14.720 --> 00:09:23.040
Your product team can help you to prioritize those and these are the bats for your future right what you should build next.

s95
00:09:23.040 --> 00:09:34.959
Um as I said I work with several several customers during this journey and there are a couple learnings that I want to share that some might be obvious but others it's very very insightful.

s96
00:09:34.959 --> 00:09:42.160
So the first one is the the trust cliff that I I like to call.

s97
00:09:42.160 --> 00:09:44.959
So what's the trust cliff?

s98
00:09:44.959 --> 00:09:56.320
So imagine you build a product, you are ready to ship and you and you measure the reliability of this product and you found 80% reliability.

s99
00:09:56.320 --> 00:09:58.160
That seems fine, right?

s100
00:09:58.160 --> 00:10:03.839
You're like, okay, I can give that to my customer and they they can start using it, right?

s101
00:10:03.839 --> 00:10:12.000
But what your customer thinks when you say you are 80% of reliability is I cannot trust this product.

s102
00:10:12.000 --> 00:10:13.760
I cannot trust.

s103
00:10:13.760 --> 00:10:16.320
Uh and why?

s104
00:10:16.320 --> 00:10:27.760
Because when you show 9 80% reliability, what they will think is oh okay so if it's 80% I still have to monitor this agent.

s105
00:10:27.760 --> 00:10:31.360
If they fail I will need to do some manual work.

s106
00:10:31.360 --> 00:10:35.440
So in fact it's actually more work for me not less work.

s107
00:10:35.440 --> 00:10:51.519
I need to build this agent plus monitor that and also do some manual work after but when you get to a point around 90% reliability the message changes to your customer and what they think it's okay 90%

s108
00:10:51.519 --> 00:10:59.360
I can actually trust this right to handle my tasks and I completely hand over like a workflow to my customer.

s109
00:10:59.360 --> 00:11:04.240
So it's basically reliability is a zero to one, right?

s110
00:11:04.240 --> 00:11:06.640
Or you trust or you don't.

s111
00:11:06.640 --> 00:11:10.240
There is nothing like in the middle there.

s112
00:11:10.240 --> 00:11:13.920
Second learning, it's the eval transparency.

s113
00:11:13.920 --> 00:11:22.800
So every time that you start working with a customer, you basically need to be transparent about three things that your product does.

s114
00:11:22.800 --> 00:11:25.279
First thing is that's easy.

s115
00:11:25.279 --> 00:11:27.120
Everybody likes to do that.

s116
00:11:27.120 --> 00:11:29.120
So what works?

s117
00:11:29.120 --> 00:11:33.600
So you basically show to your customer I am really good doing this.

s118
00:11:33.600 --> 00:11:35.519
That's the level of reliability.

s119
00:11:35.519 --> 00:11:37.120
You can trust my product.

s120
00:11:37.120 --> 00:11:42.000
Just use my agent and it will give you the reliability that you need.

s121
00:11:42.000 --> 00:11:46.959
Second point is these are the things that I'm doing to improve my product.

s122
00:11:46.959 --> 00:11:47.360
Okay.

s123
00:11:47.360 --> 00:11:53.920
So there are some things that I know it's a gap but I am investing time right now to make this better.

s124
00:11:53.920 --> 00:11:56.320
And the third thing that's the tough one.

s125
00:11:56.320 --> 00:12:00.320
Nobody wants to do that that it's what is out of scope.

s126
00:12:00.320 --> 00:12:02.640
So you really need to be transparent.

s127
00:12:02.640 --> 00:12:07.600
Like if something doesn't work for your product, just go ahead and tell your customer.

s128
00:12:07.600 --> 00:12:09.360
Don't let them try.

s129
00:12:09.360 --> 00:12:19.760
Otherwise, like there are so many tools in the market right now that if you do some if you show something for them that doesn't work, they will just forget about our product and they will move to the next.

s130
00:12:19.760 --> 00:12:22.079
So be transparent.

s131
00:12:22.079 --> 00:12:30.880
And the main learning here is transparent on product limitation increase trust more than higher benchmarks.

s132
00:12:30.880 --> 00:12:32.959
That's a very interesting finding, right?

s133
00:12:32.959 --> 00:12:39.279
Be transparent about what your product is capable of doing and in more important what's not capable of.

s134
00:12:39.279 --> 00:12:42.560
That's how you earn customer trust.

s135
00:12:43.200 --> 00:12:51.839
Third learning is if you work work very close with customers, you might find a lot of use case that you would never expected.

s136
00:12:51.839 --> 00:12:54.880
So some examples here is Amazon Leo.

s137
00:12:54.880 --> 00:12:58.320
It's the satellite internet satellite company from Amazon.

s138
00:12:58.320 --> 00:13:05.279
Working with them, we actually learn some inputs uh about creating features to do caching of trajectories.

s139
00:13:05.279 --> 00:13:10.720
So instead of doing inference all the time that they use our product, we cache trajectories.

s140
00:13:10.720 --> 00:13:13.360
If it fail, we just fall back to the model.

s141
00:13:13.360 --> 00:13:16.000
So we optimize the cost for them.

s142
00:13:16.000 --> 00:13:20.959
Second learning is with Herz, the car rental company.

s143
00:13:20.959 --> 00:13:34.800
So they were also using our product for QA automation and we learned that as part of their company they had some QA engineers that were very technical and were capable of building like

s144
00:13:34.800 --> 00:13:37.760
scripts with Python and other coding languages.

s145
00:13:37.760 --> 00:13:42.720
However, some technical uh some people from QA they are not that technical.

s146
00:13:42.720 --> 00:13:53.839
So that was the input that we needed to actually build a whole non-technical experience for our product that would enable more customers to use it.

s147
00:13:53.839 --> 00:13:57.600
Last one, this is a startup called Sol.

s148
00:13:57.600 --> 00:14:00.720
Solar is basically a RPA company.

s149
00:14:00.720 --> 00:14:10.320
If you are not familiar with the term, RPA is repetitive process automation and they offer tools to automate process for other companies.

s150
00:14:10.320 --> 00:14:13.839
So imagine the variety of use cases that they might have.

s151
00:14:13.839 --> 00:14:16.240
It's a lot of different use cases.

s152
00:14:16.240 --> 00:14:27.760
And what we learned from then is we could give some additional flexibility in our product for them to customize especially the actuation stack of our tool.

s153
00:14:27.760 --> 00:14:37.279
The learnings here are less important than keeping in mind that you will learn a lot about what your customers is trying to do here, right?

s154
00:14:37.279 --> 00:14:51.040
and what should be the next features you should build for your product and then like running this flywheel several times we came up with four principles for this flywheel.

s155
00:14:51.040 --> 00:14:57.440
First one is derive eval uh eval scenarios from production not imagination.

s156
00:14:57.440 --> 00:15:08.880
So again like creating synthetic data might be very important at the beginning of the process but as soon as you start getting customer signals make sure that feeds your eval

s157
00:15:08.880 --> 00:15:12.959
second categorize into actionable areas.

s158
00:15:12.959 --> 00:15:20.959
Again product issues, engineering issues, research issues, make sure you prioritize that well to your team.

s159
00:15:20.959 --> 00:15:24.240
Third one shared capabilities honestly.

s160
00:15:24.240 --> 00:15:27.279
Again, transparency builds trust.

s161
00:15:27.279 --> 00:15:33.760
Make sure you share to your customer, especially what your product is not good at.

s162
00:15:33.760 --> 00:15:38.399
Third one is run the flywheel as fast as you can.

s163
00:15:38.399 --> 00:15:46.160
So here it's basically the faster you run that flywheel, faster your product will improve, right?

s164
00:15:46.160 --> 00:15:48.000
That's almost obvious, right?

s165
00:15:48.000 --> 00:15:52.160
But keep it that in mind like run your flywheel as fast as you can.

s166
00:15:52.160 --> 00:16:03.839
And then like with those four principles what you will see is uh your evals will should get smarter every week.

s167
00:16:03.839 --> 00:16:11.920
But very important if you are not seeing your evals getting smarter every week that means your evals are getting stale.

s168
00:16:11.920 --> 00:16:13.680
Okay.

s169
00:16:13.680 --> 00:16:18.480
And just to finalize the talk here there are two ideas that I want you guys to take home.

s170
00:16:18.480 --> 00:16:20.240
Just two.

s171
00:16:20.240 --> 00:16:28.480
So first idea is eval must reflect what customer cares about not what you think they care about.

s172
00:16:28.480 --> 00:16:29.279
Okay.

s173
00:16:29.279 --> 00:16:37.040
All about getting customer input here and make sure your evals reflects what's their problem.

s174
00:16:37.040 --> 00:16:40.880
Second is evals are not built once.

s175
00:16:40.880 --> 00:16:45.839
It's a flywheel that evolves as your customer evolves.

s176
00:16:45.839 --> 00:16:57.440
One note here, remember that customer journey, as your customer moves through that customer journey, what they are trying to do with your product will also change.

s177
00:16:57.440 --> 00:16:59.920
Like it will get more complex, right?

s178
00:16:59.920 --> 00:17:07.839
So your evals needs to be up to date and reflecting exactly what your customer is trying to do right now.

s179
00:17:07.839 --> 00:17:16.640
And with that what you have to do is simply like build the loop ship evolves that matter and then you earn your customer trust.

s180
00:17:16.640 --> 00:17:18.640
And that's all that I have for today.

s181
00:17:18.640 --> 00:17:20.720
Uh thanks very much for joining.

s182
00:17:20.720 --> 00:17:24.559
Uh I will stay in our booth after for more discussions.

s183
00:17:24.559 --> 00:17:27.679
But again enjoy the conference.