Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab

AI Engineer · 17 min · 183 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Can everybody hear me?
00:13Okay, perfect.
00:16So, first thanks for joining everyone.
00:19Uh, my name is Filipe and I am a member of technical staff at the Amazon AGI lab.
00:26Um, as part of my role, I help us to build the next generation.
00:31I manage the programs to build the next generation of products our lab is building.
00:35And today I will talk to you about designing evolves that earn customer trust.
00:41Before talking about evolves, I just need to give you some context about what our lab was doing the past year.
00:47So everything started in March last year where we launched our Nova research preview.
00:54for context like Nova is a tool that we build as part of our lab and it's a tool that helps you to build browser agents.
01:03So basically everything that you do in a web browser you can use Nova to automate those tasks.
01:09Uh and as part of the research preview our main goal was we wanted to ship a product to get as much feedback as we could from our customers.
01:20Uh in March we launched that.
01:22Then in July we made one additional improvement in our product which was basically develop a whole new developer experience and this was because basically during the first couple months
01:34we noticed like some gaps in the developer experiment experience and we wanted to make sure customer had the best experience building agents.
01:43Then in December we finally launched our service on AWS.
01:48So today Novax is a top tier service on AWS that anyone can use to build production level agents and during this period I was basically responsible for the customer enablement
02:00piece of that.
02:01What's customer enablement just for context?
02:03It's basically I was working daytoday with all of our customers to understand what was their problem and how we were helping them to solve that.
02:13So that was my main uh role uh during last year.
02:18And then let's go to uh the problem that we will be discussing today.
02:23So I like to call this is the benchmark illusion.
02:26So you all working on AI.
02:30You are probably working on and building things based on benchmarks, right?
02:36It could be public benchmarks that everybody's using and everybody's trying to optimize the products on that and also benchmarks that you are creating your own synthetic data that you are creating and you are evaluating your product.
02:49Uh and the main goal here is you want to optimize right you want to be the best as you can there in those benchmarks but then when you ship that to production and your customers start using that what happen is it's this right
03:04they start seeing problems and it's basically everything that you were not expecting them to do they try to do that and they break your product and this is happening
03:15because you basically most of us we are doing we are working on evals that that are static.
03:22So what are static evals?
03:24Basically let's talk a little bit about what's the standard development process of a product right first thing that you do you receive uh the requirements from your product chain and you start building
03:37right everybody does that I hope then when you have a product in a good shape the next thing that you will do is running your evals again public benchmark
03:48synthetic data that you generated and you optimize for that when you start feeling comfortable you're like okay now I will ship that and you ship that to the customers
03:59but if you have a static evals what's the only thing that you can do after that it's basically this is hope right you just hope your evals were reflecting exactly what your customers would do with your product
04:13and if not it will start failing right and this is the gap this is exactly what we will be discussing today and what we were doing in our lab to solve this problem.
04:24And just a spoiler, so this is not about better benchmarks.
04:30It's about closing the customer loop.
04:32So what we want you to do here is get the feedback from your customer and close the loop.
04:39So with that in mind, what we did in our lab was basically propose an eval flywheel.
04:46So if you are not aware about the terminology but flywheel is basically a loop that you keep repeating like all the time and the first step of this evolve
04:56is define success and very important here.
05:01So define success here it's not what you think it's success it's what your customer think it's success.
05:07Okay.
05:09So next step as soon as you understand what your customer really needs and the problem you are trying to solve you go to capture signals and we as engineers
05:19what's the first thing that comes to your mind it's like instrumentation right let's get metrics let's get data as much as you can about what your customer is doing
05:30but one very important piece of that that we are missing is you should talk to your customer like talk to your customer go schedule meetings uh visit them and understand exactly what they are trying to do.
05:44Your data might be a little bit misleading.
05:47You might be missing exactly what's the problem that your customer is trying to solve.
05:52Third step, you go to diagnose gaps.
05:55That's easy.
05:56That's basically you get all these signals from customers.
05:59Could be instrumentation but also your inputs, your insights from the meetings that you had with them.
06:04And you just start categorizing those.
06:07And when I say categorizing there are basically very high level but there are three things that you could find from those discussions.
06:14First it's basically inputs about your model.
06:17So where your model is failing there and this feeds into evals right to your model.
06:23The the second point it's inputs to your engineering team.
06:28So basically what uh is the gap in your hardness that you should fix.
06:33This is more traditional bug fixing.
06:35And the third one is also on the product side.
06:38So what are you missing in terms of product like do your customer understand exactly what your product is trying to solve?
06:46Is your product like properly positioned?
06:49So that's very important.
06:51So as soon as you have the gaps di the gaps categorized you just feed that into decisions.
06:58So feed that into decisions is basically you prioritize that and you ask your science, your engineering, your product team to address those.
07:06That's basic, right?
07:07And then you just continue repeating this loop over and over.
07:13So with that in mind, like in our lab, we repeated this flywheel several several times since March last year.
07:21And what I also noticed is basically as part of my journey as a product, each of my c each customer will have their own journey using my product.
07:33What that means?
07:35So my customer evolves as they use my product, right?
07:40So there are basically four things that you can learn while your customer use your product.
07:46The first one is that's the first thing that you will do the first time that you meet a customer.
07:51It's basically use case discovery.
07:54So what you want to do here at this first step it's basically understand what's their problem.
08:00So what's the problem of your customer and make sure what you are building solves that.
08:06Right?
08:07Then after that you will have couple early adopters.
08:11Right?
08:12from these early adopters the main signal that you want to get.
08:16It's basically what's possible.
08:19So that's the time where customer will try crazy things.
08:23You will find things that you cannot help them at at all, right?
08:27But you will also find things that your customers are using your product to solve and many times you will be surprised because it's something that you would never imagine.
08:37Third point, this is when you start scaling, right?
08:40You get your early adopters but now you start scaling.
08:43you are getting more customers and you are able to identify some patterns, right?
08:49What are the patterns?
08:50The pattern is basically your hero use case, right?
08:54That's the case, that's the problem that everybody is using your product to solve.
09:00And last one, this is more future looking, but these are your product gaps.
09:05So when you were working with early adopters and also after you start scaling you will also be finding some gaps on your product.
09:14Your product team can help you to prioritize those and these are the bats for your future right what you should build next.
09:23Um as I said I work with several several customers during this journey and there are a couple learnings that I want to share that some might be obvious but others it's very very insightful.
09:34So the first one is the the trust cliff that I I like to call.
09:42So what's the trust cliff?
09:44So imagine you build a product, you are ready to ship and you and you measure the reliability of this product and you found 80% reliability.
09:56That seems fine, right?
09:58You're like, okay, I can give that to my customer and they they can start using it, right?
10:03But what your customer thinks when you say you are 80% of reliability is I cannot trust this product.
10:12I cannot trust.
10:13Uh and why?
10:16Because when you show 9 80% reliability, what they will think is oh okay so if it's 80% I still have to monitor this agent.
10:27If they fail I will need to do some manual work.
10:31So in fact it's actually more work for me not less work.
10:35I need to build this agent plus monitor that and also do some manual work after but when you get to a point around 90% reliability the message changes to your customer and what they think it's okay 90%
10:51I can actually trust this right to handle my tasks and I completely hand over like a workflow to my customer.
10:59So it's basically reliability is a zero to one, right?
11:04Or you trust or you don't.
11:06There is nothing like in the middle there.
11:10Second learning, it's the eval transparency.
11:13So every time that you start working with a customer, you basically need to be transparent about three things that your product does.
11:22First thing is that's easy.
11:25Everybody likes to do that.
11:27So what works?
11:29So you basically show to your customer I am really good doing this.
11:33That's the level of reliability.
11:35You can trust my product.
11:37Just use my agent and it will give you the reliability that you need.
11:42Second point is these are the things that I'm doing to improve my product.
11:46Okay.
11:47So there are some things that I know it's a gap but I am investing time right now to make this better.
11:53And the third thing that's the tough one.
11:56Nobody wants to do that that it's what is out of scope.
12:00So you really need to be transparent.
12:02Like if something doesn't work for your product, just go ahead and tell your customer.
12:07Don't let them try.
12:09Otherwise, like there are so many tools in the market right now that if you do some if you show something for them that doesn't work, they will just forget about our product and they will move to the next.
12:19So be transparent.
12:22And the main learning here is transparent on product limitation increase trust more than higher benchmarks.
12:30That's a very interesting finding, right?
12:32Be transparent about what your product is capable of doing and in more important what's not capable of.
12:39That's how you earn customer trust.
12:43Third learning is if you work work very close with customers, you might find a lot of use case that you would never expected.
12:51So some examples here is Amazon Leo.
12:54It's the satellite internet satellite company from Amazon.
12:58Working with them, we actually learn some inputs uh about creating features to do caching of trajectories.
13:05So instead of doing inference all the time that they use our product, we cache trajectories.
13:10If it fail, we just fall back to the model.
13:13So we optimize the cost for them.
13:16Second learning is with Herz, the car rental company.
13:20So they were also using our product for QA automation and we learned that as part of their company they had some QA engineers that were very technical and were capable of building like
13:34scripts with Python and other coding languages.
13:37However, some technical uh some people from QA they are not that technical.
13:42So that was the input that we needed to actually build a whole non-technical experience for our product that would enable more customers to use it.
13:53Last one, this is a startup called Sol.
13:57Solar is basically a RPA company.
14:00If you are not familiar with the term, RPA is repetitive process automation and they offer tools to automate process for other companies.
14:10So imagine the variety of use cases that they might have.
14:13It's a lot of different use cases.
14:16And what we learned from then is we could give some additional flexibility in our product for them to customize especially the actuation stack of our tool.
14:27The learnings here are less important than keeping in mind that you will learn a lot about what your customers is trying to do here, right?
14:37and what should be the next features you should build for your product and then like running this flywheel several times we came up with four principles for this flywheel.
14:51First one is derive eval uh eval scenarios from production not imagination.
14:57So again like creating synthetic data might be very important at the beginning of the process but as soon as you start getting customer signals make sure that feeds your eval
15:08second categorize into actionable areas.
15:12Again product issues, engineering issues, research issues, make sure you prioritize that well to your team.
15:20Third one shared capabilities honestly.
15:24Again, transparency builds trust.
15:27Make sure you share to your customer, especially what your product is not good at.
15:33Third one is run the flywheel as fast as you can.
15:38So here it's basically the faster you run that flywheel, faster your product will improve, right?
15:46That's almost obvious, right?
15:48But keep it that in mind like run your flywheel as fast as you can.
15:52And then like with those four principles what you will see is uh your evals will should get smarter every week.
16:03But very important if you are not seeing your evals getting smarter every week that means your evals are getting stale.
16:11Okay.
16:13And just to finalize the talk here there are two ideas that I want you guys to take home.
16:18Just two.
16:20So first idea is eval must reflect what customer cares about not what you think they care about.
16:28Okay.
16:29All about getting customer input here and make sure your evals reflects what's their problem.
16:37Second is evals are not built once.
16:40It's a flywheel that evolves as your customer evolves.
16:45One note here, remember that customer journey, as your customer moves through that customer journey, what they are trying to do with your product will also change.
16:57Like it will get more complex, right?
16:59So your evals needs to be up to date and reflecting exactly what your customer is trying to do right now.
17:07And with that what you have to do is simply like build the loop ship evolves that matter and then you earn your customer trust.
17:16And that's all that I have for today.
17:18Uh thanks very much for joining.
17:20Uh I will stay in our booth after for more discussions.
17:24But again enjoy the conference.