{
  "video": {
    "id": "ra8X7_bnujw",
    "title": "Why Waiting for the Next AI Model Can Pay Off - John Stecher, Blackstone & Scott Worland, Norm AI",
    "duration": 1245,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "ra8X7_bnujw",
    "title": "Why Waiting for the Next AI Model Can Pay Off - John Stecher, Blackstone & Scott Worland, Norm AI",
    "one_liner": "Weigh each AI project's ROI against what it costs to build today: Blackstone has deliberately delayed builds by about 6 months when newer models would make them cheaper and nothing was lost by waiting, but it still pushes engineers to get quickly to clickable POCs rather than always waiting for the next model.",
    "summary": "Blackstone's CTO John Stecher and Norm AI's CTO discuss how to deploy agents in high-stakes financial and regulatory settings without falling into the bitter-lesson trap of hand-built systems that better models make obsolete. They argue this is a gradient, not a binary: invest heavily in data, evals and locked-down core primitives, and invest conservatively in anything the frontier is likely to absorb. Norm's own example is hundreds of thousands of lines of carefully written workflow scaffolding that turned out to be mostly obsolete once agentic approaches were run against its existing benchmark set. They also argue that cheap software changes the economics: internal apps can be disposable, and rewriting is now a legitimate alternative to versioning forward.",
    "key_points": [
      "Stecher says agent programs will fail without a cohesive data strategy that moves data out of transactional systems into something agents can easily access and recall.",
      "Straight-through processing still matters: Stecher uses agents to understand workflows, writes code optimized for straight-through processing, and injects agent intelligence only where humans used to make rote decisions.",
      "Blackstone balances the ROI of a request against the true cost of building it today, and has chosen to wait when models or compute would make it cheaper in 6 months and building now adds no real business value.",
      "Norm's 2024 production system was heavily structured, resembling a large map-reduce workflow. Regression testing it produced tens of thousands of expert hand-labeled examples, which were then immediately reusable to test agentic approaches, showing most of the hundreds of thousands of lines of code were pretty obsolete.",
      "After that, Norm's engineering leadership drove a rented Sprinter van to an Airbnb in Upstate New York and asked what they would build if starting over. They left with specs for a platform that delivers agentic experiences to clients and is amenable to AIs as co-developers.",
      "Evals are paramount in financial services because answers and accuracy shift when model versions, harnesses or data change.",
      "Norm replaces traditional roadmaps with lawyer-developed benchmarks and scorecards that judge whether an agent or a human completed a task successfully across various dimensions, then watches for uplift as the system improves.",
      "POCs are now fast, but the hard yards remain production: infrastructure inside the enterprise, cybersecurity, supportability, and software that is traceable and monitorable through the SRE platform. Norm keeps its core primitives under complete human oversight, including design, code ownership and change approval."
    ],
    "takeaways": [
      "Before building, compare the ROI with today's cost and ask whether a 6-month delay loses any business value; if not, consider waiting for a better model, but don't wait indefinitely.",
      "Build a large expert-labeled eval set and rerun it whenever models, harnesses or data change, so you can measure system health and quickly test new approaches.",
      "Build conduits, not dams, for frontier improvements: invest heavily in data capture, evals and locked-down core primitives, invest conservatively elsewhere, and stay willing to rip and replace.",
      "Go straight to a clickable POC instead of PowerPoints, put it in front of people, iterate quickly, throw it away if it isn't needed, and focus on value produced rather than counting tokens.",
      "Regularly ask whether rewriting an internal application would cost less than versioning it forward, and treat many internal apps as disposable."
    ],
    "topics": [
      "agents",
      "evals",
      "bitter lesson",
      "build vs wait",
      "data strategy",
      "financial services",
      "production readiness",
      "disposable software"
    ],
    "tools": [
      "Blackstone",
      "Norm AI",
      "Norm Law",
      "Factory",
      "Fable",
      "Claude Code (transcribed as \"cloud code\")",
      "GPT-1",
      "IBM"
    ],
    "quotes": [
      {
        "text": "We have made that decision before is like let's actually wait for the model to become slightly more sophisticated so that we can solve these problems with it because it will cost less at that time",
        "at": "10:00",
        "url": "https://www.youtube.com/watch?v=ra8X7_bnujw&t=600s"
      },
      {
        "text": "every time I fire a linguist my models improve",
        "at": "08:33",
        "url": "https://www.youtube.com/watch?v=ra8X7_bnujw&t=513s"
      },
      {
        "text": "You don't want to have dams where like the intelligence is coming in and it can't quite get into your stack",
        "at": "17:03",
        "url": "https://www.youtube.com/watch?v=ra8X7_bnujw&t=1023s"
      },
      {
        "text": "does it cost me less to rewrite it than it cost me to keep versioning the current version forward?",
        "at": "18:24",
        "url": "https://www.youtube.com/watch?v=ra8X7_bnujw&t=1104s"
      }
    ],
    "words": 5612
  },
  "summary_url": "/#ra8X7_bnujw",
  "transcript": {
    "html": "/transcripts/ra8X7_bnujw.html",
    "txt": "/transcripts/ra8X7_bnujw.txt",
    "vtt": "/transcripts/ra8X7_bnujw.vtt"
  }
}