WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=CvmZtAPmAW4 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/CvmZtAPmAW4.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.720 --> 00:00:14.360 Uh my name is Supriya. s3 00:00:14.360 --> 00:00:15.800 Uh happy to be here today. s4 00:00:15.800 --> 00:00:21.680 I'm here to talk to you all about why AB testing for AI products is weird. s5 00:00:21.680 --> 00:00:31.520 So, for those of you who don't know what AB testing is, it's just a method used to evaluate two versions of something with each other and see which is better. s6 00:00:31.520 --> 00:00:37.080 So, that could be two versions of an email where you're trying to see which one's getting more clicks. s7 00:00:37.080 --> 00:00:39.480 It could be two versions of a feature. s8 00:00:39.480 --> 00:00:44.120 It could just be a feature against what's already live right now. s9 00:00:44.120 --> 00:00:47.480 So, let me ask a quick question. s10 00:00:47.640 --> 00:00:49.280 Can I get a show of hands? s11 00:00:49.280 --> 00:00:55.680 Who thinks that if an AB test comes back positive, that the feature should actually ship? s12 00:00:57.000 --> 00:01:04.120 Well, looks like a lot of you are smart and already know that uh when it comes to AI, things can get a little weird. s13 00:01:04.120 --> 00:01:06.880 But here, let's talk about why. s14 00:01:06.880 --> 00:01:20.760 So, what I found when um testing with AI features, specifically GenAI and using LLMs in consumer tech features, a positive result from an AB test is actually just where the journey begins. s15 00:01:20.760 --> 00:01:22.520 It's not where it ends. s16 00:01:22.520 --> 00:01:30.920 And we're going to go over um five reasons that I've specifically discovered that were problems that we had to overcome. s17 00:01:31.640 --> 00:01:33.480 So, I'm a product manager. s18 00:01:33.480 --> 00:01:34.640 I work at Bumble. s19 00:01:34.640 --> 00:01:39.520 Bumble's a dating app with over 20 million uh monthly active users. s20 00:01:39.520 --> 00:01:42.360 And so, what does a product manager do? s21 00:01:42.360 --> 00:01:44.320 We represent our users. s22 00:01:44.320 --> 00:01:56.960 We know the problems that they face and we try to deliver things that can solve the problems while still being responsible for the P&L of a company because we don't have unlimited resources to be able to solve the problems. s23 00:01:56.960 --> 00:02:05.412 So, in order to talk through the five reasons that I think AB testing with AI was weird, I figured it'll be good for you all to have a visual example s24 00:02:05.412 --> 00:02:11.972 of what we actually shipped so that you can try to draw parallels with things that you're working on as well. s25 00:02:11.972 --> 00:02:16.972 The feature that we shipped was an AI photo feedback feature. s26 00:02:16.972 --> 00:02:20.452 In a dating app, naturally, everyone needs a dating profile. s27 00:02:20.452 --> 00:02:24.412 So, they would add pictures and they wouldn't know what good looks like. s28 00:02:24.412 --> 00:02:37.452 So, we essentially came up with with the expertise that we had of what works and what doesn't work in a dating profile, we gave AI-generated suggestions of how people could improve the pictures that they were adding. s29 00:02:38.532 --> 00:02:40.732 So, let's go into the first one. s30 00:02:40.732 --> 00:02:44.252 That users ask for it and then they ignore it. s31 00:02:44.252 --> 00:02:48.812 Now, I mentioned that a PM is supposed to know the problems of our users, right? s32 00:02:48.812 --> 00:02:52.252 The tools that we have to do that is research. s33 00:02:52.252 --> 00:02:54.452 The journey begins with discovery. s34 00:02:54.452 --> 00:02:57.972 What are the problems that people have and how can we solve them? s35 00:02:57.972 --> 00:03:02.612 Surveys, customer interviews are great tools to find out. s36 00:03:02.612 --> 00:03:17.772 When we do research at Bumble, one of the almost unanimous problems that everyone always talked about was that they didn't know whether their dating profile was good enough and they didn't get any support from us in understanding whether they were doing a good job or not. s37 00:03:17.772 --> 00:03:25.972 So, when AI came in and there was all that potential with LLMs, we knew immediately that this was a problem that we could solve. s38 00:03:25.972 --> 00:03:35.172 And so, we came up with different versions of how we thought AI could help people's dating journeys without actually doing dating for them. s39 00:03:35.172 --> 00:03:43.172 And so, when we showed people examples of AI support and guidance that we could give them, the the response was great. s40 00:03:43.172 --> 00:03:53.772 Almost 78% of the people that we spoke to came back and said they would love getting feedback from us and and tell us and and tell them how to do better. s41 00:03:53.772 --> 00:04:04.654 But when we shipped the AB test, which means that with 50% of the people getting the feature and 50% of the people not getting the feature, we only saw 10 to 15% s42 00:04:04.654 --> 00:04:07.294 people actually using the feature. s43 00:04:07.294 --> 00:04:13.934 What that taught us was that openness to AI doesn't actually predict whether people are going to use the feature. s44 00:04:13.934 --> 00:04:25.934 And that's where we go back to product management 101, which is you need to really bake in the entire journey of using AI into what people are already doing in the app. s45 00:04:25.934 --> 00:04:29.014 Education and awareness are super critical. s46 00:04:29.014 --> 00:04:39.134 What we did was instead of building a standalone feature, which people could just somehow find within the app, we went into the journeys that people were already using. s47 00:04:39.134 --> 00:04:43.054 So, when they were adding pictures, we asked them, "Hey, do you want feedback?" s48 00:04:43.054 --> 00:04:52.134 When they were hitting a wall when they were trying to find someone to date and not finding any matches, we would ask them, "Hey, did you go and check the feedback in your profile? s49 00:04:52.134 --> 00:04:53.974 Maybe you could change something." s50 00:04:53.974 --> 00:04:56.574 And that drove up adoption. s51 00:04:56.574 --> 00:05:01.294 It's really important that we obviously measure what it is we're trying to do. s52 00:05:01.294 --> 00:05:17.934 Um that is going to form the foundation cuz ultimately, if the survey is coming back with a positive result and you ship the feature, and you haven't actually looked at whether people are using it or not, and you won't be able to improve that adoption in the first place. s53 00:05:19.254 --> 00:05:24.454 The second thing that we found was that the average win hit the tail. s54 00:05:24.454 --> 00:05:28.654 So, when we shipped this feature again, the top-line results were great. s55 00:05:28.654 --> 00:05:31.654 More people were adding pictures to their profiles. s56 00:05:31.654 --> 00:05:38.614 The number of changes that they were making to adjust their profiles was more than the people who weren't receiving the feedback. s57 00:05:38.614 --> 00:05:42.454 But when we went into the details, there were weird things happening. s58 00:05:42.454 --> 00:05:50.174 The AI-generated tips were actually telling people to smile more when they were already smiling in their pictures. s59 00:05:50.174 --> 00:05:55.414 And so, what that told us is that the positive average was not enough to evaluate the success anymore. s60 00:05:55.414 --> 00:05:58.414 We had to go into the details. s61 00:05:58.414 --> 00:06:02.614 Let's look at two examples before I share what we actually did. s62 00:06:02.614 --> 00:06:05.935 So, I think Spotify has something called an AI DJ. s63 00:06:05.935 --> 00:06:09.535 It generates playlists for people using AI. s64 00:06:09.535 --> 00:06:19.335 And then, let's imagine that there's your banking app which helps you budget and decide what you should spend on and how much you should spend in a month. s65 00:06:19.335 --> 00:06:23.135 The scope of error is quite different in both of these things. s66 00:06:23.135 --> 00:06:31.215 It's okay if two out of four of the songs that are generated by AI are not good, but it's not okay if an app is telling you to spend $2,000 s67 00:06:31.215 --> 00:06:33.015 when you really can't. s68 00:06:33.015 --> 00:06:35.655 So, that's where guardrails come into place. s69 00:06:35.655 --> 00:06:37.415 That's where we started. s70 00:06:37.415 --> 00:06:43.855 Setting strong guardrails of what our AI-generated feedback should and should not do was super critical. s71 00:06:43.855 --> 00:06:49.215 We couldn't be telling people to change things that they can't control, their skin color, their height. s72 00:06:49.215 --> 00:06:52.495 These are things that the AI should not touch. s73 00:06:52.495 --> 00:06:56.695 The next thing we had to do was reading real outputs. s74 00:06:56.695 --> 00:06:59.615 So, this is where evals were super helpful. s75 00:06:59.615 --> 00:07:04.215 Now, there are way more experts here that know about evals than me. s76 00:07:04.215 --> 00:07:10.335 Uh you're in the right place to learn about that, but um setting up the right eval tools were super critical for us. s77 00:07:10.335 --> 00:07:15.655 We were able to dig in and see what real users were seeing when they were getting feedback. s78 00:07:15.655 --> 00:07:22.095 And that was helping us actually see whether our prompt was actually doing a good job or not. s79 00:07:22.095 --> 00:07:30.895 And so, ultimately, the success criteria changed from looking at overall results of green versus red to what does the long tail of data have? s80 00:07:30.895 --> 00:07:34.975 Are we happy with the way different segments of users are receiving feedback? s81 00:07:34.975 --> 00:07:38.775 Um are women responding the same way to our feedback as men? s82 00:07:38.775 --> 00:07:46.335 Are different languages responding like people in different countries responding in the same way to feedback as well? s83 00:07:46.335 --> 00:07:52.655 Because maybe the language generated in French is different from the language generated in English. s84 00:07:54.695 --> 00:07:58.655 The third thing is that the green dashboard and the red bill. s85 00:07:58.655 --> 00:08:05.055 So, naturally, with generative AI, the success comes with a cost. s86 00:08:05.055 --> 00:08:12.977 When we started, we knew that this was such a big problem for our users that we wanted the best quality of feedback and the highest accuracy. s87 00:08:12.977 --> 00:08:15.097 So, we went with the best LLM. s88 00:08:15.097 --> 00:08:20.137 Now, naturally, that was about 10x more expensive than a simpler approach. s89 00:08:20.137 --> 00:08:25.057 And then you add to that the cost of running evals on the solution that you've built. s90 00:08:25.057 --> 00:08:28.057 That was just too much. s91 00:08:28.057 --> 00:08:31.337 Now, Bumble is not a stranger to AI. s92 00:08:31.337 --> 00:08:35.697 I mean, the matching algorithm is also a use of AI. s93 00:08:35.697 --> 00:08:40.617 But, when it comes to ML, the cost is different compared to GenAI. s94 00:08:40.617 --> 00:08:47.777 In In an algorithm, there's a lot of upfront cost of compu- compute as well as uh training. s95 00:08:47.777 --> 00:08:55.017 But, then after after it's working, the incremental cost for every user to use it is not that much. s96 00:08:55.017 --> 00:09:04.097 With generative AI, the more successful your product, the more people that use it, and the more often they use it, the more your bill increases. s97 00:09:04.097 --> 00:09:06.177 And so, what do you do? s98 00:09:06.177 --> 00:09:09.737 Maybe don't jump to the using the best model from the start. s99 00:09:09.737 --> 00:09:15.097 And it's very important to understand why you're using AI in the first place. s100 00:09:15.097 --> 00:09:24.017 There are a lot of my peers that I've spoken to who have just gotten mandates from their companies to use AI in their features and their product. s101 00:09:24.017 --> 00:09:26.537 But, you really need to understand why you're doing it. s102 00:09:26.537 --> 00:09:31.377 Is it for long-term benefits or is it for something short-term? s103 00:09:31.377 --> 00:09:34.657 Now, let's look at Amazon as an example. s104 00:09:34.657 --> 00:09:39.857 They have um an AI assistant that can help you buy something on Amazon. s105 00:09:39.857 --> 00:09:43.857 Now, why would Amazon need to launch this assistant? s106 00:09:43.857 --> 00:09:52.177 Is it for short-term benefits that good recommendations lead to more purchases and therefore more revenue in the short-term? s107 00:09:52.177 --> 00:09:56.297 Or is it But, what if the recommendations aren't great? s108 00:09:56.297 --> 00:09:58.097 Does that mean they don't do it? s109 00:09:58.097 --> 00:09:59.297 Maybe not. s110 00:09:59.297 --> 00:10:07.977 In the long-term, having an assistant that you can rely on might make people choose Amazon over other e-commerce companies. s111 00:10:07.977 --> 00:10:16.378 And so, the reason to launch the assistant is just for brand equity and a long-term benefit of keeping your users on your platform. s112 00:10:16.378 --> 00:10:25.178 And so really understanding why you're trying to use AI is going to be critical here because your budget and how much you're willing to spend is going to come from that. s113 00:10:25.178 --> 00:10:35.578 From a short-term benefit, yes, maybe the ROI can be a quick decision, but if it's for a long-term play, maybe it's worth getting that extra budget. s114 00:10:37.538 --> 00:10:41.458 So I spoke about the fact that we just jumped to using the best model. s115 00:10:41.458 --> 00:10:45.178 Naturally, the cost was way too high and we could not scale that. s116 00:10:45.178 --> 00:10:50.138 So we switched what we made the simple decision to switch to a cheaper model. s117 00:10:50.138 --> 00:10:57.178 But as the cost went down, so did the impact of the results because the accuracy went down as well. s118 00:10:57.178 --> 00:11:09.738 Which basically told us that the AI feedback that we were giving was and the the results that we saw from our AB test was due to the model that we were testing, not the concept of giving feedback itself. s119 00:11:09.738 --> 00:11:11.498 And so what did we do? s120 00:11:11.498 --> 00:11:20.418 What we learned at this point was just that we really need to test a lot and being able to be data-driven was critical. s121 00:11:20.418 --> 00:11:23.698 And eval setups really help us do that. s122 00:11:23.698 --> 00:11:35.618 Looking at being able to test how much different models can impact different data sets, the difference in the cost, and then being able to retest different versions of prompts and s123 00:11:35.618 --> 00:11:39.098 feedback quite quickly, that was critical. s124 00:11:39.098 --> 00:11:46.058 And ultimately, we resulted in the cheapest option, which helped us get the good result. s125 00:11:49.258 --> 00:11:54.858 And now the last one, which is sometimes there just may not be a global rollout. s126 00:11:54.858 --> 00:11:57.858 So Meta recently launched Muse AI. s127 00:11:57.858 --> 00:12:00.498 I'm not sure if you all have used it. s128 00:12:00.498 --> 00:12:03.458 I'm from London, so I haven't yet. s129 00:12:03.458 --> 00:12:07.858 And that's because Muse AI is only available in the US and Canada. s130 00:12:07.858 --> 00:12:13.339 And this is because AI regulation is constantly changing and is different all over the world. s131 00:12:13.339 --> 00:12:19.699 And there is a financial and legal risk to companies to not be compliant with that. s132 00:12:19.699 --> 00:12:23.979 So, when we started our project, we started in the US. s133 00:12:23.979 --> 00:12:25.979 We shipped in the US, but then that's it. s134 00:12:25.979 --> 00:12:27.219 Like we had to stop. s135 00:12:27.219 --> 00:12:34.419 We were not able to ship the feature that was available in the US as is to other countries. s136 00:12:34.419 --> 00:12:35.899 And so, what did we do? s137 00:12:35.899 --> 00:12:49.979 We realized almost immediately that whether you're a product manager or not, whether you're an aspiring product engineer, forward deployed engineer, everyone needs to start being a generalist and thinking about s138 00:12:49.979 --> 00:12:56.019 compliance and legal applications of products that they're building early on. s139 00:12:56.019 --> 00:12:57.739 It can't come later. s140 00:12:57.739 --> 00:13:02.819 So, what that means is you can have multiple approaches. s141 00:13:02.819 --> 00:13:07.779 You could start with, "Oh, I definitely know I want to ship this feature all over the world. s142 00:13:07.779 --> 00:13:13.739 Therefore, let me understand all the restrictions and build something within all of those things." s143 00:13:13.739 --> 00:13:23.659 Or you could go the approach of, "Hey, I know that US is where I'm going to get my maximum impact and therefore I will launch the best thing possible in the US s144 00:13:23.659 --> 00:13:25.659 and then maybe not anywhere else." s145 00:13:25.659 --> 00:13:31.219 So, there's going to be a spectrum of decision-making that you will have to make with this new set of data. s146 00:13:31.219 --> 00:13:33.099 It's not just about building anymore. s147 00:13:33.099 --> 00:13:38.019 If you build the best thing, but you can't get it in the hands of users, it doesn't mean anything. s148 00:13:38.019 --> 00:13:42.819 So, you could take the path of, "Okay, one country." s149 00:13:42.819 --> 00:13:49.539 Or you could take the additional cost of making a version of the app which works differently in different countries as well. s150 00:13:49.539 --> 00:13:53.099 That's the decision that now needs to be made. s151 00:13:54.899 --> 00:14:04.019 So, to summarize, I think whenever it comes to AI features, when you all are building, ask yourself these five questions. s152 00:14:04.019 --> 00:14:09.019 Are people using it or are they just saying that they will use it? s153 00:14:09.019 --> 00:14:13.619 Do you read real outputs or just look at the averages? s154 00:14:14.259 --> 00:14:20.501 Does it still pay off after factoring in all the costs that went into the solution? s155 00:14:20.621 --> 00:14:25.141 Is a win on the best model something you can afford? s156 00:14:25.501 --> 00:14:30.381 And are you clear legally on which markets you can ship in? s157 00:14:30.861 --> 00:14:36.541 Because the question is not just did the feature win the AB test anymore. s158 00:14:36.701 --> 00:14:38.365 Thank you. s159 00:14:38.365 --> 00:14:40.365 [applause]