{
  "video": {
    "id": "YXowceUKYJI",
    "title": "KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat",
    "duration": 1308,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "YXowceUKYJI",
    "title": "KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat",
    "one_liner": "Red Hat's inference team explains how llm-d's KV-cache-aware routing (endpoint picker) and prefill/decode disaggregation address agentic workloads — multi-turn, >90% cache hit rates, 100:1 input/output ratios — and, crucially, when PD is *not* worth it.",
    "summary": "Ashish Kamra and Yuchen Fama argue that public inference benchmarks show steady-state, sanitized numbers that hide the chaotic reality of agentic workloads: up to 3,000 turns, cache hit rates well over 90%, input/output ratios over 100:1, and high-variance context that forces you to plan on P90 distributions rather than averages. They walk through llm-d's endpoint-picker plugins scoring pods on running/waiting requests, KV cache utilization and prefix cache availability, then break down prefill/decode disaggregation as a fix for phase interference — a long prefill stalling ongoing decode and jittering user streaming latency. They show internal Red Hat results where PD cuts P99 inter-token latency from ~900ms to ~100ms, but insist PD is a phase-separation trade-off, not a magic bullet, and present a decision matrix for when to stick with aggregated serving. They close with an in-progress case study serving GLM 5.2 on H200 clusters rather than the B200s customers don't have.",
    "key_points": [
      "Agentic workloads break classic LLM serving assumptions: SWE-bench and real Claude Code session traces show multi-turn from a few to 3,000 turns, cache hit rates 'oftentimes well exceeding 90%' because agents reuse system prompts and tool definitions, and input/output ratios over 100:1.",
      "Because variance is so high, capacity planning must use distributions and P90 numbers, not averages. Red Hat collaborated with Google and IBM to add a trace replay tool to inference-perf so the community can study these patterns.",
      "The economics justify measuring cached throughput separately: Anthropic's API pricing shows a 10x cost difference between cached and non-cached tokens — 'a pretty serious impact on your business.'",
      "llm-d's router uses endpoint picker (EPP) plugins that continuously probe each vLLM pod's metrics — running and waiting requests, KV cache utilization, prefix cache availability — to score pods and route to the one with lowest load and highest cache-hit probability.",
      "KV cache management work in progress: more offloading tiers (NVMe SSD, XFS filesystem), KV-centric stores like Mooncake, and session-aware eviction policies including priority and session pinning.",
      "Live demo of KV-cache-aware routing: first request populates cache in ~3s with no hit; second turn with same system prompt reuses the cache on the same pod address in ~1s; a new system prompt routes to a different pod at ~3s; changing only the user prompt returns to ~1s.",
      "PD disaggregation exists because of phase interference: prefill is compute-bound, bursty, high-FLOPs and thrives on batch parallelism; decode is memory-bandwidth-hungry, latency-sensitive and needs heavy cache residency. Co-located, a sudden long prefill 'will completely stall the ongoing decode token generation process.'",
      "Results: P99 ITL drops from ~900ms (aggregated) to ~100ms on PD — 'almost nine times better' and much smoother. On GPT-OSS 120B / 16 H100s (aggregated: 4 replicas TP4; disaggregated: 2P2D TP4) with a 10,000-token prefix and 128 tokens per turn, KV-cache-aware routing alone beats default Kubernetes scheduling, and PD wins specifically in the *middle* concurrency regime — similar to aggregated at both low and high concurrency. On 64 H100s (8 replicas TP8 vs 3P5D TP8, 5,000 ISL / 500 OSL), the PD Pareto curve dominates aggregated across the entire interactivity spectrum.",
      "GLM 5.2 case study on H200 (not B200, 'because customers don't have that luxury'): 3 prefill workers tuned for throughput, 1 dedicated decode worker tuned for low latency, NIXL for KV transfer, leader-worker-set groups at TP1/DP8/EP8. With 2P1D on a 45:1 ISL/OSL agentic dataset they got 4x faster TTFT and 60% more requests. A fun fact found days earlier: BF16 KV cache is faster than FP8 KV cache for longer prefill."
    ],
    "takeaways": [
      "Don't trust published steady-state benchmark numbers for agentic serving — replay real multi-turn traces (the new inference-perf trace replay tool) and plan capacity from P90 distributions, not averages.",
      "Measure and report cached throughput as its own metric; at a 10x price gap between cached and uncached tokens, cache hit rate is a line on your balance sheet, not a performance footnote.",
      "Reach for prefix/KV-cache-aware routing first — it fixes TTFT and gives visible gains over default Kubernetes scheduling without any PD complexity. PD is the tool for inter-token latency, not TTFT.",
      "Adopt PD only if you check the boxes: long context with high ISL/OSL, a large model that can absorb rich model parallelism, mid-range concurrency, and strict ITL streaming requirements — and you have RDMA or RoCE fabric for the KV transfer. Short/moderate context, low concurrency, strict TTFT targets, or no high-speed fabric means stay aggregated.",
      "Treat the PD ratio as dynamic, not a config constant: pair it with autoscaling that scales prefill and decode pools independently as traffic mix changes, and keep tuning TP/DP alongside it.",
      "Re-test KV cache dtype assumptions on your own workload — Red Hat found BF16 KV cache beating FP8 for long prefill."
    ],
    "topics": [
      "inference",
      "kv-cache",
      "pd-disaggregation",
      "kubernetes",
      "agentic-workloads",
      "vllm",
      "benchmarking",
      "gpu-serving"
    ],
    "tools": [
      "llm-d",
      "vLLM",
      "Kubernetes",
      "OpenShift",
      "Red Hat",
      "CNCF",
      "GuideLLM",
      "LLM Compressor",
      "Speculators",
      "Red Hat AI (Hugging Face model hub)",
      "inference-perf",
      "Mooncake",
      "NIXL",
      "GLM 5.2",
      "GPT-OSS 120B",
      "NVIDIA H100 / H200 / B200",
      "Anthropic API",
      "Claude Code",
      "SWE-bench",
      "leader-worker set / disaggregated set APIs",
      "RDMA / RoCE",
      "Google",
      "IBM",
      "CoreWeave",
      "NVIDIA",
      "deeplearning.ai"
    ],
    "quotes": [
      {
        "text": "when you look at public inference benchmark results you are typically looking at very steady state isolated highly sanitized numbers and what those benchmarks actually don't show you is the chaotic reality of multi-turn interactions, massive context fluctuations which are very typical of agentic workloads",
        "at": "00:45",
        "url": "https://www.youtube.com/watch?v=YXowceUKYJI&t=45s"
      },
      {
        "text": "There's 10x cost difference between cash and non-cash tokens. So, 10x difference on your token balance sheet is pretty serious impact on your business.",
        "at": "06:11",
        "url": "https://www.youtube.com/watch?v=YXowceUKYJI&t=371s"
      },
      {
        "text": "if there's a sudden influx of a long prefilled palm, it will completely stall the ongoing decode token generation process causing massive problems and jitter in user streaming latency",
        "at": "11:43",
        "url": "https://www.youtube.com/watch?v=YXowceUKYJI&t=703s"
      },
      {
        "text": "I don't want to leave you guys that PD is the answer to everything and it's a magic bullet. But it's essentially a phase separation trade-off and not a magic bullet.",
        "at": "15:37",
        "url": "https://www.youtube.com/watch?v=YXowceUKYJI&t=937s"
      }
    ],
    "words": 4004
  },
  "summary_url": "/#YXowceUKYJI",
  "transcript": {
    "html": "/transcripts/YXowceUKYJI.html",
    "txt": "/transcripts/YXowceUKYJI.txt",
    "vtt": "/transcripts/YXowceUKYJI.vtt"
  }
}