{
  "video": {
    "id": "cQQbJqvZkpo",
    "title": "Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave",
    "duration": 922,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "cQQbJqvZkpo",
    "title": "Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave",
    "one_liner": "CoreWeave's head of inference walks through the single platform they've built to serve everything from small models to trillion-parameter workloads — two consumption models (serverless and dedicated), KV-cache-aware routing, optional prefill/decode disaggregation, and NVFP4 quantization plus trained speculators as the biggest performance levers.",
    "summary": "Sitanshu Gupta argues that if you derive an inference platform from the real consumption models and workload shapes up front, you don't have to keep rebuilding it — you just keep adding knobs. He describes CoreWeave's serverless (pay-per-token, plus a provisioned-throughput option to kill the noisy-neighbour problem) and dedicated (per-GPU-per-hour, private gateway, customer picks the engine and deployment) offerings, and the four workload shapes — agentic, chat, real-time voice/video, and batch — whose different latency and sequence-length profiles drive the design. The technical core is KV cache: 80–90% of agentic input sequences are identical across requests, prefill is compute-bound and expensive, so the router prioritises KV cache locality and the platform offloads cache to high-bandwidth storage rather than evicting and re-prefilling. He closes with leaderboard screenshots from Artificial Analysis (Kimi 2.6/2.7) and OpenRouter (GLM), where CoreWeave's speed lands close to Fireworks.",
    "key_points": [
      "Two consumption models: serverless (API/UI, pay per token, breadth of model catalog, no hardware or orchestration to manage) and dedicated (customer knows exactly which hardware they run on, billed per GPU per hour, private gateway with isolation, customer owns model deployment and performance tuning).",
      "Serverless has a third mode — provisioned throughput: if the customer shares their traffic profile, CoreWeave carves out capacity behind the scenes so they avoid the noisy-neighbour timeout problem, while still being charged per token and still not seeing the hardware.",
      "Four workload shapes drive the design: agentic and chat both have very high input sequence lengths and low output, but agentic multi-turns are low-latency back-to-back while chat has human read time between turns; voice/video are streaming and absolutely latency-sensitive; batch SLAs run to seconds, minutes, or even 10–12 hours. Fitting these together over time is 'a game of tetris' for infrastructure utilisation.",
      "80–90% of the input sequence length in agentic workloads is the same across requests (varies by customer), so re-running prefill is wasted compute-bound spend — which is why cached input tokens are priced far cheaper than fresh input tokens everywhere.",
      "The router does KV-cache-aware routing across heterogeneous capacity spanning GPU generations, zones and regions; the load-balancing priority order is KV cache locality first, then least-loaded fallback.",
      "Prefill/decode disaggregation is a platform capability, not a default — 'PD disaggregation is not cheap for every type of use case', so serverless picks per use case and dedicated customers choose it themselves, along with which engine (vLLM, SGLang, TensorRT-LLM) and whether to run one scalable deployment or many.",
      "Instead of evicting KV cache between chat turns and re-prefilling, they offload it to high-bandwidth storage and load it straight back into HBM when the follow-up request arrives; externally comparable work is LMCache and Mooncake, CoreWeave uses its own.",
      "Biggest current performance levers are quantization to NVFP4 and speculative decoding; CoreWeave will train speculators on a customer's own dataset asynchronously for better acceptance lengths and significantly higher output throughput, then deploy them into that customer's deployment.",
      "Results shown: top-of-leaderboard placements on Kimi 2.6/2.7 from Artificial Analysis, plus GLM numbers from OpenRouter — deliberately chosen because Artificial Analysis runs specific benchmark workloads while OpenRouter reflects actual user traffic (CoreWeave appears there under Weights & Biases branding, acquired about a year ago), with speed close to Fireworks.",
      "Serverless usage metering tracks token counts only, not the tokens themselves, to maintain zero data retention (ZDR)."
    ],
    "takeaways": [
      "Design routing around KV cache locality before anything else — with 80–90% input overlap in agentic traffic, cache hits avoid the compute-bound prefill that dominates cost, and the price gap between input and cached-input tokens is the market telling you this.",
      "Don't turn on prefill/decode disaggregation by default; treat it as a per-workload choice, because it isn't cheap for every use case.",
      "For chat workloads, offload KV cache to high-bandwidth storage across the human's think-time gap rather than evicting it and re-prefilling on the next turn — look at LMCache and Mooncake if you're not building your own.",
      "Classify your traffic by shape (input/output sequence length, inter-turn latency, SLA looseness) rather than by model, then schedule loose-SLA batch work onto reserved real-time capacity overnight via scale-up/scale-down APIs.",
      "If you're evaluating an inference provider, read both Artificial Analysis (specific benchmark workloads) and OpenRouter (real user traffic) — they measure different things.",
      "Pursue NVFP4 quantization and speculative decoding, and consider training speculators on your own dataset offline for higher acceptance lengths and output throughput."
    ],
    "topics": [
      "inference",
      "kv-cache",
      "gpu-infrastructure",
      "serving",
      "agentic",
      "quantization",
      "speculative-decoding",
      "batch"
    ],
    "tools": [
      "CoreWeave",
      "vLLM",
      "SGLang",
      "TensorRT-LLM",
      "LMCache",
      "Mooncake",
      "NVIDIA GPUs",
      "NVFP4",
      "Artificial Analysis",
      "OpenRouter",
      "Weights & Biases",
      "Fireworks",
      "Kimi 2.6 / 2.7",
      "GLM",
      "AWS Annapurna Labs",
      "SambaNova"
    ],
    "quotes": [
      {
        "text": "prefill is supercomputebound very expensive that's why as much as you can hit the cache more you can save",
        "at": "07:57",
        "url": "https://www.youtube.com/watch?v=cQQbJqvZkpo&t=477s"
      },
      {
        "text": "in the time dimension you have to play the game of tetris on how you can fit it in to utilize the underlying infrastructure the most",
        "at": "05:03",
        "url": "https://www.youtube.com/watch?v=cQQbJqvZkpo&t=303s"
      },
      {
        "text": "the priority order that we typically take is uh first KV cache locality and then the least loaded fallback",
        "at": "10:04",
        "url": "https://www.youtube.com/watch?v=cQQbJqvZkpo&t=604s"
      },
      {
        "text": "the speed over here that we have from our deployment is pretty close to what fireworks is providing us fireworks fast right",
        "at": "14:13",
        "url": "https://www.youtube.com/watch?v=cQQbJqvZkpo&t=853s"
      }
    ],
    "words": 2905
  },
  "summary_url": "/#cQQbJqvZkpo",
  "transcript": {
    "html": "/transcripts/cQQbJqvZkpo.html",
    "txt": "/transcripts/cQQbJqvZkpo.txt",
    "vtt": "/transcripts/cQQbJqvZkpo.vtt"
  }
}