78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble

AI Engineer · 14 min · 159 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Uh my name is Supriya.
00:14Uh happy to be here today.
00:15I'm here to talk to you all about why AB testing for AI products is weird.
00:21So, for those of you who don't know what AB testing is, it's just a method used to evaluate two versions of something with each other and see which is better.
00:31So, that could be two versions of an email where you're trying to see which one's getting more clicks.
00:37It could be two versions of a feature.
00:39It could just be a feature against what's already live right now.
00:44So, let me ask a quick question.
00:47Can I get a show of hands?
00:49Who thinks that if an AB test comes back positive, that the feature should actually ship?
00:57Well, looks like a lot of you are smart and already know that uh when it comes to AI, things can get a little weird.
01:04But here, let's talk about why.
01:06So, what I found when um testing with AI features, specifically GenAI and using LLMs in consumer tech features, a positive result from an AB test is actually just where the journey begins.
01:20It's not where it ends.
01:22And we're going to go over um five reasons that I've specifically discovered that were problems that we had to overcome.
01:31So, I'm a product manager.
01:33I work at Bumble.
01:34Bumble's a dating app with over 20 million uh monthly active users.
01:39And so, what does a product manager do?
01:42We represent our users.
01:44We know the problems that they face and we try to deliver things that can solve the problems while still being responsible for the P&L of a company because we don't have unlimited resources to be able to solve the problems.
01:56So, in order to talk through the five reasons that I think AB testing with AI was weird, I figured it'll be good for you all to have a visual example
02:05of what we actually shipped so that you can try to draw parallels with things that you're working on as well.
02:11The feature that we shipped was an AI photo feedback feature.
02:16In a dating app, naturally, everyone needs a dating profile.
02:20So, they would add pictures and they wouldn't know what good looks like.
02:24So, we essentially came up with with the expertise that we had of what works and what doesn't work in a dating profile, we gave AI-generated suggestions of how people could improve the pictures that they were adding.
02:38So, let's go into the first one.
02:40That users ask for it and then they ignore it.
02:44Now, I mentioned that a PM is supposed to know the problems of our users, right?
02:48The tools that we have to do that is research.
02:52The journey begins with discovery.
02:54What are the problems that people have and how can we solve them?
02:57Surveys, customer interviews are great tools to find out.
03:02When we do research at Bumble, one of the almost unanimous problems that everyone always talked about was that they didn't know whether their dating profile was good enough and they didn't get any support from us in understanding whether they were doing a good job or not.
03:17So, when AI came in and there was all that potential with LLMs, we knew immediately that this was a problem that we could solve.
03:25And so, we came up with different versions of how we thought AI could help people's dating journeys without actually doing dating for them.
03:35And so, when we showed people examples of AI support and guidance that we could give them, the the response was great.
03:43Almost 78% of the people that we spoke to came back and said they would love getting feedback from us and and tell us and and tell them how to do better.
03:53But when we shipped the AB test, which means that with 50% of the people getting the feature and 50% of the people not getting the feature, we only saw 10 to 15%
04:04people actually using the feature.
04:07What that taught us was that openness to AI doesn't actually predict whether people are going to use the feature.
04:13And that's where we go back to product management 101, which is you need to really bake in the entire journey of using AI into what people are already doing in the app.
04:25Education and awareness are super critical.
04:29What we did was instead of building a standalone feature, which people could just somehow find within the app, we went into the journeys that people were already using.
04:39So, when they were adding pictures, we asked them, "Hey, do you want feedback?"
04:43When they were hitting a wall when they were trying to find someone to date and not finding any matches, we would ask them, "Hey, did you go and check the feedback in your profile?
04:52Maybe you could change something."
04:53And that drove up adoption.
04:56It's really important that we obviously measure what it is we're trying to do.
05:01Um that is going to form the foundation cuz ultimately, if the survey is coming back with a positive result and you ship the feature, and you haven't actually looked at whether people are using it or not, and you won't be able to improve that adoption in the first place.
05:19The second thing that we found was that the average win hit the tail.
05:24So, when we shipped this feature again, the top-line results were great.
05:28More people were adding pictures to their profiles.
05:31The number of changes that they were making to adjust their profiles was more than the people who weren't receiving the feedback.
05:38But when we went into the details, there were weird things happening.
05:42The AI-generated tips were actually telling people to smile more when they were already smiling in their pictures.
05:50And so, what that told us is that the positive average was not enough to evaluate the success anymore.
05:55We had to go into the details.
05:58Let's look at two examples before I share what we actually did.
06:02So, I think Spotify has something called an AI DJ.
06:05It generates playlists for people using AI.
06:09And then, let's imagine that there's your banking app which helps you budget and decide what you should spend on and how much you should spend in a month.
06:19The scope of error is quite different in both of these things.
06:23It's okay if two out of four of the songs that are generated by AI are not good, but it's not okay if an app is telling you to spend $2,000
06:31when you really can't.
06:33So, that's where guardrails come into place.
06:35That's where we started.
06:37Setting strong guardrails of what our AI-generated feedback should and should not do was super critical.
06:43We couldn't be telling people to change things that they can't control, their skin color, their height.
06:49These are things that the AI should not touch.
06:52The next thing we had to do was reading real outputs.
06:56So, this is where evals were super helpful.
06:59Now, there are way more experts here that know about evals than me.
07:04Uh you're in the right place to learn about that, but um setting up the right eval tools were super critical for us.
07:10We were able to dig in and see what real users were seeing when they were getting feedback.
07:15And that was helping us actually see whether our prompt was actually doing a good job or not.
07:22And so, ultimately, the success criteria changed from looking at overall results of green versus red to what does the long tail of data have?
07:30Are we happy with the way different segments of users are receiving feedback?
07:34Um are women responding the same way to our feedback as men?
07:38Are different languages responding like people in different countries responding in the same way to feedback as well?
07:46Because maybe the language generated in French is different from the language generated in English.
07:54The third thing is that the green dashboard and the red bill.
07:58So, naturally, with generative AI, the success comes with a cost.
08:05When we started, we knew that this was such a big problem for our users that we wanted the best quality of feedback and the highest accuracy.
08:12So, we went with the best LLM.
08:15Now, naturally, that was about 10x more expensive than a simpler approach.
08:20And then you add to that the cost of running evals on the solution that you've built.
08:25That was just too much.
08:28Now, Bumble is not a stranger to AI.
08:31I mean, the matching algorithm is also a use of AI.
08:35But, when it comes to ML, the cost is different compared to GenAI.
08:40In In an algorithm, there's a lot of upfront cost of compu- compute as well as uh training.
08:47But, then after after it's working, the incremental cost for every user to use it is not that much.
08:55With generative AI, the more successful your product, the more people that use it, and the more often they use it, the more your bill increases.
09:04And so, what do you do?
09:06Maybe don't jump to the using the best model from the start.
09:09And it's very important to understand why you're using AI in the first place.
09:15There are a lot of my peers that I've spoken to who have just gotten mandates from their companies to use AI in their features and their product.
09:24But, you really need to understand why you're doing it.
09:26Is it for long-term benefits or is it for something short-term?
09:31Now, let's look at Amazon as an example.
09:34They have um an AI assistant that can help you buy something on Amazon.
09:39Now, why would Amazon need to launch this assistant?
09:43Is it for short-term benefits that good recommendations lead to more purchases and therefore more revenue in the short-term?
09:52Or is it But, what if the recommendations aren't great?
09:56Does that mean they don't do it?
09:58Maybe not.
09:59In the long-term, having an assistant that you can rely on might make people choose Amazon over other e-commerce companies.
10:07And so, the reason to launch the assistant is just for brand equity and a long-term benefit of keeping your users on your platform.
10:16And so really understanding why you're trying to use AI is going to be critical here because your budget and how much you're willing to spend is going to come from that.
10:25From a short-term benefit, yes, maybe the ROI can be a quick decision, but if it's for a long-term play, maybe it's worth getting that extra budget.
10:37So I spoke about the fact that we just jumped to using the best model.
10:41Naturally, the cost was way too high and we could not scale that.
10:45So we switched what we made the simple decision to switch to a cheaper model.
10:50But as the cost went down, so did the impact of the results because the accuracy went down as well.
10:57Which basically told us that the AI feedback that we were giving was and the the results that we saw from our AB test was due to the model that we were testing, not the concept of giving feedback itself.
11:09And so what did we do?
11:11What we learned at this point was just that we really need to test a lot and being able to be data-driven was critical.
11:20And eval setups really help us do that.
11:23Looking at being able to test how much different models can impact different data sets, the difference in the cost, and then being able to retest different versions of prompts and
11:35feedback quite quickly, that was critical.
11:39And ultimately, we resulted in the cheapest option, which helped us get the good result.
11:49And now the last one, which is sometimes there just may not be a global rollout.
11:54So Meta recently launched Muse AI.
11:57I'm not sure if you all have used it.
12:00I'm from London, so I haven't yet.
12:03And that's because Muse AI is only available in the US and Canada.
12:07And this is because AI regulation is constantly changing and is different all over the world.
12:13And there is a financial and legal risk to companies to not be compliant with that.
12:19So, when we started our project, we started in the US.
12:23We shipped in the US, but then that's it.
12:25Like we had to stop.
12:27We were not able to ship the feature that was available in the US as is to other countries.
12:34And so, what did we do?
12:35We realized almost immediately that whether you're a product manager or not, whether you're an aspiring product engineer, forward deployed engineer, everyone needs to start being a generalist and thinking about
12:49compliance and legal applications of products that they're building early on.
12:56It can't come later.
12:57So, what that means is you can have multiple approaches.
13:02You could start with, "Oh, I definitely know I want to ship this feature all over the world.
13:07Therefore, let me understand all the restrictions and build something within all of those things."
13:13Or you could go the approach of, "Hey, I know that US is where I'm going to get my maximum impact and therefore I will launch the best thing possible in the US
13:23and then maybe not anywhere else."
13:25So, there's going to be a spectrum of decision-making that you will have to make with this new set of data.
13:31It's not just about building anymore.
13:33If you build the best thing, but you can't get it in the hands of users, it doesn't mean anything.
13:38So, you could take the path of, "Okay, one country."
13:42Or you could take the additional cost of making a version of the app which works differently in different countries as well.
13:49That's the decision that now needs to be made.
13:54So, to summarize, I think whenever it comes to AI features, when you all are building, ask yourself these five questions.
14:04Are people using it or are they just saying that they will use it?
14:09Do you read real outputs or just look at the averages?
14:14Does it still pay off after factoring in all the costs that went into the solution?
14:20Is a win on the best model something you can afford?
14:25And are you clear legally on which markets you can ship in?
14:30Because the question is not just did the feature win the AB test anymore.
14:36Thank you.
14:38[applause]