{
  "video": {
    "id": "CvmZtAPmAW4",
    "title": "78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble",
    "duration": 884,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "CvmZtAPmAW4",
    "title": "78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble",
    "one_liner": "A positive AI A/B test is where the work starts, not where it ends: Bumble's AI photo-feedback feature got 78% enthusiasm in research but only 10–15% usage, the average win hid bad outputs in the tail, the best model cost about 10x too much, and regulation kept it in the US.",
    "summary": "Bumble PM Supriya Subudhi argues that A/B testing GenAI features is 'weird' because a green result can't be taken at face value. Using Bumble's AI photo-feedback feature for dating profiles as the example, she gives five reasons: stated interest doesn't predict adoption, averages hide bad long-tail outputs, GenAI costs grow with success, a win may belong to the model rather than the concept, and regulation can block a global rollout. She ends with five questions to ask before shipping an AI feature.",
    "key_points": [
      "In research, almost 78% of users said they would love feedback on their dating profile, but in a 50/50 A/B test only 10–15% actually used the feature. Openness to AI does not predict usage.",
      "Adoption went up when feedback moved out of a standalone feature and into journeys users were already on: offering feedback while they added photos, and nudging them to check it when they weren't getting matches.",
      "Top-line metrics looked good (more photos added, more profile edits), but the AI was telling people to smile more in photos where they were already smiling. A positive average was not enough.",
      "Bumble set guardrails so the AI never suggests changing things people can't control, such as skin colour or height. She contrasts the error tolerance of Spotify's AI DJ (two bad songs out of four is fine) with a banking app (telling you to spend $2,000 you don't have is not).",
      "Evals let the team read the real outputs users saw and judge the prompt. The success criteria moved from green vs red to checking the long tail across segments: women vs men, and different languages and countries (French output may differ from English).",
      "The best LLM cost about 10x more than a simpler approach, before adding the cost of running evals. Unlike ML such as the matching algorithm, where the cost is mostly up front, GenAI costs grow with every user and every use.",
      "Switching to a cheaper model lowered accuracy and impact, which showed the A/B win came from the model, not the concept of giving feedback. Testing models, datasets, costs and prompts quickly through evals led them to the cheapest option that still gave a good result.",
      "Regulation limited the rollout. The feature shipped in the US and couldn't go to other countries as it was, much as Meta's Muse AI is available only in the US and Canada."
    ],
    "takeaways": [
      "Measure real usage, not survey intent, and build the AI feature into journeys users already follow, with education and awareness, instead of shipping it as a standalone feature.",
      "Read real outputs with evals, set guardrails on what the AI must never suggest, and judge success by the long tail across segments and languages, not the average.",
      "Don't start with the best model. Work out whether the AI is for short-term ROI or long-term brand equity (like Amazon's shopping assistant), and set the budget from that answer.",
      "Test several models and prompt versions against cost, so you know whether a win depends on an expensive model you can't afford to scale.",
      "Think about compliance and legal limits early. Decide up front whether to build within every market's restrictions, launch the best version in one market, or pay to build versions that differ by country."
    ],
    "topics": [
      "a/b testing",
      "product management",
      "evals",
      "llm cost",
      "guardrails",
      "ai regulation",
      "adoption",
      "model selection"
    ],
    "tools": [
      "Bumble",
      "Spotify AI DJ",
      "Amazon AI shopping assistant",
      "Meta Muse AI"
    ],
    "quotes": [
      {
        "text": "a positive result from an AB test is actually just where the journey begins. It's not where it ends.",
        "at": "01:15",
        "url": "https://www.youtube.com/watch?v=CvmZtAPmAW4&t=75s"
      },
      {
        "text": "openness to AI doesn't actually predict whether people are going to use the feature.",
        "at": "04:07",
        "url": "https://www.youtube.com/watch?v=CvmZtAPmAW4&t=247s"
      },
      {
        "text": "With generative AI, the more successful your product, the more people that use it, and the more often they use it, the more your bill increases.",
        "at": "08:55",
        "url": "https://www.youtube.com/watch?v=CvmZtAPmAW4&t=535s"
      },
      {
        "text": "If you build the best thing, but you can't get it in the hands of users, it doesn't mean anything.",
        "at": "13:31",
        "url": "https://www.youtube.com/watch?v=CvmZtAPmAW4&t=811s"
      }
    ],
    "words": 2798
  },
  "summary_url": "/#CvmZtAPmAW4",
  "transcript": {
    "html": "/transcripts/CvmZtAPmAW4.html",
    "txt": "/transcripts/CvmZtAPmAW4.txt",
    "vtt": "/transcripts/CvmZtAPmAW4.vtt"
  }
}