{
  "video": {
    "id": "9-Y_oS9OBLc",
    "title": "When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec",
    "duration": 1110,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "9-Y_oS9OBLc",
    "title": "When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec",
    "one_liner": "imec ran more than a thousand SWE-bench Pro runs and found that self-hosting open-weight models for coding agents only pays off around the GLM class on big hardware, such as 8× B200 at about 15% GPU utilisation, where it saves roughly 24–25%. Below that, the cost is a step function, concurrency hurts more than accuracy, and the harness you pick can halve your GPU bill.",
    "summary": "Steven Latré of imec describes how organisations mature in their AI use. They start with closed APIs, then cut costs with model routing, then host open-weight models on leased infrastructure, and finally buy their own. In those last two phases the design space blows up: model choice, trust, harnesses, hardware architecture, CPU-to-GPU balance and KV cache all start to matter. imec benchmarked long-horizon coding tasks (SWE-bench Pro) across three coding harnesses, a range of open-weight models and matching hardware, measuring resolve rate, latency and cost. The results show that the API-versus-own-infrastructure choice is non-linear and constantly moving. imec is now building a public benchmarking dataset and launching an alliance to bring the software and hardware worlds together.",
    "key_points": [
      "The maturity path: closed APIs, then cost optimisation and model routing (the 'low-hanging fruit'), then hosting open-weight models on leased infrastructure, then buying your own. imec itself is in the third phase.",
      "Long-horizon tasks are the first pivot point where owning your AI stack makes sense. Very interactive use may not have the same economics.",
      "Resolve rates on SWE-bench Pro: smaller models of up to a few hundred billion parameters (Qwen, DeepSeek) reached about 32.5–40%. GLM was in the Opus range. Kimi K3 reached 86%, though the speaker notes the tasks were probably partly in its training set.",
      "Qwen and DeepSeek performed about the same, but DeepSeek needed hardware about four times as expensive to rent. Moving from GLM to Kimi K3 gave about 20% more accuracy for only about 15% more GPU rental cost. Cost is a step function, not a linear curve.",
      "Concurrency is probably more important than accuracy. Most systems are memory-bound: the model parameters and every user's KV cache have to fit in memory. Kimi K3 supported only about 4 reliable concurrent users and was still about three times slower than the Claude Code API baseline.",
      "Total cost of ownership: on low-tier hardware you need about 89% GPU utilisation to beat the API, which is 'completely infeasible'. On larger systems like 8× B200, about 15% utilisation is enough, which is realistic. GLM-class models are the threshold where self-hosting starts to win, at about 24–25% savings for coding assistance.",
      "The harnesses gave roughly the same task accuracy. But Codex generated only half the tokens with GLM, so it used half the GPU wall time and would produce half the GPU bill on your own infrastructure.",
      "Not yet covered: quantisation, speculative decoding (applied but not studied in detail), mixed enterprise workloads, and the fact that new models and hardware keep arriving."
    ],
    "takeaways": [
      "Before self-hosting, measure cost at the level of a whole task (resolve rate, latency, total cost of ownership), not just token prices. A slightly better or worse model can force a completely different hardware tier.",
      "Plan capacity around concurrency and KV-cache memory, not just whether the model's weights fit. Check how many users a configuration can serve comfortably before you commit.",
      "Only move off APIs when you can realistically keep the GPUs busy. That favours larger hardware (such as 8× B200) and GLM-class or bigger models, not small models on low-tier boxes.",
      "Once you pay for GPU wall time, treat the choice of harness as a cost lever: benchmark token consumption per harness and model pair, because accuracy may barely change while the bill halves.",
      "Start with long-horizon, non-interactive workloads such as coding agents when testing self-hosting."
    ],
    "topics": [
      "self-hosting",
      "open-weight models",
      "inference cost",
      "total cost of ownership",
      "coding agents",
      "benchmarks",
      "gpu infrastructure",
      "concurrency"
    ],
    "tools": [
      "imec",
      "SWE-bench Pro",
      "Codex",
      "Claude Code",
      "Qwen",
      "DeepSeek",
      "GLM",
      "Kimi K3",
      "Opus",
      "Artificial Analysis",
      "NVIDIA B200"
    ],
    "quotes": [
      {
        "text": "there's definitely not a linear curve to be found here",
        "at": "09:42",
        "url": "https://www.youtube.com/watch?v=9-Y_oS9OBLc&t=582s"
      },
      {
        "text": "concurrency is actually probably even more important than accuracy",
        "at": "12:10",
        "url": "https://www.youtube.com/watch?v=9-Y_oS9OBLc&t=730s"
      },
      {
        "text": "that's really realistic and 89% is just completely infeasible",
        "at": "14:00",
        "url": "https://www.youtube.com/watch?v=9-Y_oS9OBLc&t=840s"
      },
      {
        "text": "I think that the software world and the hardware world deserves to be much closer together",
        "at": "17:53",
        "url": "https://www.youtube.com/watch?v=9-Y_oS9OBLc&t=1073s"
      }
    ],
    "words": 3861
  },
  "summary_url": "/#9-Y_oS9OBLc",
  "transcript": {
    "html": "/transcripts/9-Y_oS9OBLc.html",
    "txt": "/transcripts/9-Y_oS9OBLc.txt",
    "vtt": "/transcripts/9-Y_oS9OBLc.vtt"
  }
}