{
  "video": {
    "id": "l1-D89bAuOA",
    "title": "Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google",
    "duration": 967,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "l1-D89bAuOA",
    "title": "Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google",
    "one_liner": "Two Google engineers who benchmark inference for a living argue that most irreproducible LLM performance numbers are the fault of the benchmark client — Python GIL-bound load generators that silently miss the requested QPS and inflate latency — and present inference-perf, a CNCF load generator that reports whether it actually met the load it was asked for.",
    "summary": "Ashok Chandrasekar and Jason Kramberger walk through why they repeatedly fail to reproduce LLM serving benchmark results other people publish, and trace it to four pitfalls: metric accuracy, no observability into the harness itself, reproducibility, and dataset quality. They show a single-process Python harness asked for 200 QPS delivering only 38 QPS on a shared-core machine and capping around 170 QPS on a much bigger one — while still reporting as though it ran the requested rate — plus client-side latency inflation of up to 58 seconds against a simulated server. The fix they propose is inference-perf, a CNCF project out of Kubernetes WG Serving with a declarative config, a multi-process load generator that sidesteps the GIL, and client metrics reported alongside server metrics; it kept up at 5,000 QPS where tools like k6 did not, and — crucially — reported that it kept up. They close with a published workload catalog (multi-turn, tree of thought, agentic generation, batch summarization) and Prism, the llm-d UI for sharing benchmark results.",
    "key_points": [
      "The benchmark ecosystem splits into four segments: model-server built-in Python scripts (vLLM, SGLang), competitive analysis tools (MLPerf, SemiAnalysis, Artificial Analysis), HTTP load tools (Locust, k6, Grafana), and production-scale LLM benchmarks — the last is the gap they target.",
      "Production stacks like llm-d break simple harnesses: online plus batch workloads, large inference pools, prefill/decode disaggregation and workload autoscaling, at a scale ordinary harnesses can't drive.",
      "Four pitfalls behind irreproducible results: accurate metrics, observability into whether the harness itself is failing to maintain load, reproducibility given dataset randomness, and dataset quality.",
      "Concrete experiment: asked for 200 QPS, a single-process harness on a small shared-core machine delivered only 38 QPS; on a much more powerful machine single-process harnesses capped out around 170 QPS — and reported the numbers as if the request had been honoured. Python's GIL is the root cause when CPU-bound.",
      "Client-side latency inflation of up to 58 seconds was observed in one test, caused by the harness thrashing while collecting streaming token responses — a result that looks like a bottlenecked server but isn't.",
      "Stochastic variables corrupt comparisons: a claimed 20% throughput win turned out to come from the harness setting model temperature to 0 (deterministic outputs) instead of a realistic ~0.7; and the same ShareGPT dataset produced different input token counts across two harnesses because of different sampling and truncation.",
      "inference-perf is a CNCF project spun out of Kubernetes WG Serving: declarative user-defined config for run-to-run reproducibility, a multi-process load generator (main process queues requests by planned execution time — Poisson, constant rate, or constant concurrency — spread over a request queue channel to worker processes that record executed vs planned time), and client metrics reported alongside server metrics.",
      "At 5,000 QPS inference-perf kept up where other tools including k6 did not, and reported that it kept up; benchmarking should also cover a load sweep to find the server's saturation point and check SLO conformance (e.g. TTFT P90).",
      "They publish a workload catalog — multi-turn generation, tree of thought, agentic generation, batch summarization — each with a natural-language scenario definition plus config expressed both in inference-perf terms and in generic terms so workloads can be standardized across tools.",
      "Prism, part of llm-d, is the UI for sharing workloads and results; a shown example benchmarked agentic code generation on TPUs at production scale with eight replicas, with combined optimizations reaching almost hundreds of thousands of tokens per second versus a plain Kubernetes-service baseline in front of multiple model server replicas."
    ],
    "takeaways": [
      "Before blaming your server or inference stack for bad numbers, check whether the benchmark client actually delivered the load you asked for — demand client-side observability of achieved vs planned request rate, not just a results table.",
      "Don't drive production-scale benchmarks with a single-process Python harness; the GIL will cap you (38 QPS out of a requested 200 on a small machine, ~170 on a big one) and inflate measured latency by tens of seconds.",
      "Pin and publish the stochastic variables: sampling temperature (use something like 0.7 that matches real traffic, not 0), dataset sampling and truncation, whether generation is forced to EOS, and prefix cache rates — otherwise cross-tool comparisons are meaningless.",
      "Benchmark with datasets and workloads that resemble what you actually intend to serve; start from the published workload catalog (multi-turn, tree of thought, agentic, batch summarization) rather than synthetic prompts.",
      "Sweep a range of loads to find the saturation point and validate SLO conformance (e.g. TTFT P90), instead of reporting a single QPS figure."
    ],
    "topics": [
      "benchmarking",
      "llm-inference",
      "performance",
      "reproducibility",
      "kubernetes",
      "load-testing",
      "observability",
      "open-source"
    ],
    "tools": [
      "inference-perf",
      "llm-d",
      "llm-d Prism",
      "llm-d-benchmark",
      "vLLM",
      "SGLang",
      "MLPerf",
      "SemiAnalysis",
      "Artificial Analysis",
      "Locust",
      "k6",
      "Grafana",
      "Kubernetes (WG Serving)",
      "CNCF",
      "ShareGPT",
      "Python",
      "Google",
      "TPUs"
    ],
    "quotes": [
      {
        "text": "it is a problem because you ask for 200 QPS and then you don't know whether it actually delivered it. It will just say I ran it these are the numbers. So you think okay you ran 200 QPS but in fact you you have not.",
        "at": "06:20",
        "url": "https://www.youtube.com/watch?v=l1-D89bAuOA&t=380s"
      },
      {
        "text": "as a user you don't have insight into this right you run it you trust the numbers it produces but they are wildly different",
        "at": "08:01",
        "url": "https://www.youtube.com/watch?v=l1-D89bAuOA&t=481s"
      },
      {
        "text": "a lot of times you diagnose it as a your server or inference stack problem but in a lot of cases it could be your benchmark harness",
        "at": "08:30",
        "url": "https://www.youtube.com/watch?v=l1-D89bAuOA&t=510s"
      },
      {
        "text": "even at 5,000 QPS inference perf was able to keep up due to this architecture and most importantly actually report that it was able to keep up.",
        "at": "10:46",
        "url": "https://www.youtube.com/watch?v=l1-D89bAuOA&t=646s"
      }
    ],
    "words": 2824
  },
  "summary_url": "/#l1-D89bAuOA",
  "transcript": {
    "html": "/transcripts/l1-D89bAuOA.html",
    "txt": "/transcripts/l1-D89bAuOA.txt",
    "vtt": "/transcripts/l1-D89bAuOA.vtt"
  }
}