{
  "video": {
    "id": "Emo5FGGY-wM",
    "title": "Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab",
    "duration": 1065,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "Emo5FGGY-wM",
    "title": "Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab",
    "one_liner": "Static evals leave you just hoping production matches your benchmarks. Instead, run an eval flywheel: define success, capture signals, diagnose gaps, feed decisions. Base eval scenarios on what customers actually do, because at 80% reliability customers think 'I cannot trust this product', and around 90% they hand over the workflow.",
    "summary": "Drawing on a year of customer enablement for Nova, the Amazon AGI Lab's tool for building browser agents (research preview March, new developer experience July, AWS service December), the speaker argues that optimising for public or synthetic benchmarks creates a 'benchmark illusion'. Static evals break once real customers use the product. His fix is not better benchmarks but 'closing the customer loop' with a repeating eval flywheel. He adds lessons from customers: a reliability 'trust cliff', being open about limitations, and use cases from Amazon Leo, Hertz and an RPA startup that no one expected.",
    "key_points": [
      "The 'benchmark illusion': teams optimise for public benchmarks and their own synthetic data, ship, and then customers do everything they weren't expected to do and break the product. With static evals, the only thing left after shipping 'is hope'.",
      "Eval flywheel, four repeating steps: (1) define success as the customer sees it, not as you do; (2) capture signals through instrumentation and also by talking to and visiting customers, because data can mislead; (3) diagnose gaps and sort them into model failures (these feed evals), engineering or harness bugs, and product gaps such as positioning or understanding; (4) feed them into decisions prioritised across science, engineering and product.",
      "Customers evolve through a journey: use case discovery (what is their problem), early adopters (what's possible, crazy and surprising uses), scaling (patterns and your 'hero use case'), and product gaps (bets on what to build next).",
      "The 'trust cliff': at 80% reliability customers still have to monitor the agent and do manual work when it fails, so it means more work, not less. Around 90% they trust it and hand over a whole workflow. 'Reliability is a zero to one.'",
      "Eval transparency: tell customers three things. What works, at what reliability; what you are improving; and what is out of scope. 'Transparent on product limitation increase trust more than higher benchmarks.' With so many tools on the market, customers who hit something that doesn't work will move to the next one.",
      "Unexpected use cases from close customer work: Amazon Leo (Amazon's satellite internet company) led to caching trajectories instead of running inference every time, falling back to the model on failure, to cut costs. Hertz used it for QA automation and needed a whole non-technical experience for less technical QA staff. Sol, an RPA startup with very varied use cases, needed more flexibility to customise the actuation stack.",
      "Four flywheel principles: derive eval scenarios from production, not imagination; categorise into actionable areas (product, engineering, research); share capabilities honestly; run the flywheel as fast as you can.",
      "Your evals should get smarter every week. If they aren't, they are getting stale."
    ],
    "takeaways": [
      "Use synthetic data only at the start. Once customer signals arrive, base eval scenarios on production usage.",
      "Go beyond instrumentation: schedule meetings with customers and visit them to learn what they are really trying to do, then sort findings into model, engineering-harness and product buckets with clear owners.",
      "Treat reliability as a threshold, not a gradient. Aim for the roughly 90% level where customers will hand over a workflow, because at 80% they see more work, not less.",
      "Tell customers up front what your product is out of scope for, as well as what works and what you are improving, rather than letting them find failures themselves.",
      "Run the loop as often as you can and update evals as customers move through their journey and their tasks get more complex. Check every week whether your evals got smarter."
    ],
    "topics": [
      "evals",
      "agents",
      "browser agents",
      "reliability",
      "customer feedback",
      "product development",
      "benchmarks"
    ],
    "tools": [
      "Amazon Nova (browser agent tool)",
      "Amazon AGI Lab",
      "AWS",
      "Amazon Leo",
      "Hertz",
      "Sol (RPA startup)",
      "Python"
    ],
    "quotes": [
      {
        "text": "So it's basically reliability is a zero to one, right? Or you trust or you don't.",
        "at": "11:00",
        "url": "https://www.youtube.com/watch?v=Emo5FGGY-wM&t=660s"
      },
      {
        "text": "transparent on product limitation increase trust more than higher benchmarks.",
        "at": "12:25",
        "url": "https://www.youtube.com/watch?v=Emo5FGGY-wM&t=745s"
      },
      {
        "text": "if you are not seeing your evals getting smarter every week that means your evals are getting stale.",
        "at": "16:06",
        "url": "https://www.youtube.com/watch?v=Emo5FGGY-wM&t=966s"
      },
      {
        "text": "this is not about better benchmarks. It's about closing the customer loop.",
        "at": "04:28",
        "url": "https://www.youtube.com/watch?v=Emo5FGGY-wM&t=268s"
      }
    ],
    "words": 3027
  },
  "summary_url": "/#Emo5FGGY-wM",
  "transcript": {
    "html": "/transcripts/Emo5FGGY-wM.html",
    "txt": "/transcripts/Emo5FGGY-wM.txt",
    "vtt": "/transcripts/Emo5FGGY-wM.vtt"
  }
}