{ "video": { "id": "_cRgl3maiT8", "title": "How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs", "duration": 1367, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "_cRgl3maiT8", "title": "How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs", "one_liner": "Inference keeps up with token demand by shifting the speed-vs-throughput Pareto frontier to the right. Runtime tricks came first: batching, paged KV cache, prompt caching and chunked prefill. The next step is disaggregation: splitting prefill/decode, draft/verify models, or attention/FFN layers onto whichever hardware suits each piece best.", "summary": "Daniel (Cerebras) and Michelle (Gimlet Labs) start from the cause: LLM decoding is autoregressive and memory-bound, so compute sits idle up to 90% of the time. Adding GPUs gives diminishing returns because of all-reduce overhead, which forces a trade-off between speed and throughput. They first cover runtime optimizations inside a single worker: continuous batching, paged KV cache, prompt caching and chunked prefill. They then cover system-level disaggregation across heterogeneous hardware: prefill/decode split, speculative decoding and attention/FFN split. They close by arguing that agents themselves will drive the next gains, for example in kernel generation, chip design, compilers and model bring-up.", "key_points": [ "Demand numbers: about $765B in compute spend this year, rising to $1.6T by 2031. Google went from serving 480 trillion tokens/month (last year's I/O) to 3.2 quadrillion/month. Llama 3.3 70B is now 25% cheaper and 3.6x faster than when it launched.", "Tensor-parallel splitting across 4 GPUs needs an all-reduce twice per layer, so 4x the compute can give only a 2x speedup. Energy per token rises steeply as more GPUs are added. Claude Code fast mode is cited as an example from the speed end of the frontier: roughly 6x the cost for 2x the speed.", "KV cache memory explodes with long context. For a 70B model the rule of thumb is 1 GB per 3,000 tokens, so a 128k-context coding agent needs about 42 GB for one request. Paged KV cache allocates memory like an OS virtual memory system and serves about 3x more requests.", "Long-context coding agents (200K to 1M tokens) clog pipelines because no other prefill or decode can run while a long prefill runs. Chunked prefill cuts long prefills into pieces so other requests can be interleaved.", "Workload shape decides the bottleneck. Summarization (long prompt, short answer) is prefill/compute-bound. Story writing (short prompt, long answer) is decode/memory-bound. Prefill/decode disaggregation lets the two pools scale independently.", "Memory bandwidth: H100 has 3.35 TB/s. Cerebras claims 21,000 TB/s from on-chip SRAM, which they describe as 7,000x more. That gives about 3,000 tokens/s of decode versus 600 on an H100. In a partnership with AWS, Trainium handles prefill, the KV cache moves over EFA, and Cerebras CS-3 handles decode, claimed to give 5x high-speed token capacity in the same footprint.", "Gimlet's multi-silicon cloud breaks workloads into pieces and maps each piece to the best hardware (Nvidia, AMD, Trainium, Cerebras, d-Matrix, Intel Gaudi). It claims 3–10x faster inference than homogeneous systems at the same power budget. Running speculative decoding's draft model on fast small-model hardware (d-Matrix or Cerebras) gave 10x the interactivity at the same throughput.", "Attention/FFN disaggregation is aimed at models of about 3T parameters. Memory-bound attention runs on high-bandwidth hardware and matmul-heavy FFN runs on compute-heavy hardware, with each layer ping-ponging between machines. Gimlet's KForge kernel-generation agent sped up GPT-OSS-120B on AMD by 33% end-to-end in under a week, on top of kernels that were already optimized." ], "takeaways": [ "Profile your workload's prompt-to-output ratio before choosing hardware or a scaling strategy. Prefill-heavy and decode-heavy workloads need different resources.", "Turn on runtime optimizations (continuous batching, paged KV cache, prompt caching for long system prompts, chunked prefill) before adding GPUs. Extra GPUs give diminishing returns because of communication overhead.", "If you serve long-context coding agents next to short chat requests, use chunked prefill so a large prefill doesn't block everyone else.", "Consider speculative decoding with a draft model that is fine-tuned or trained from the full model, ideally on hardware that runs small models very fast.", "There is no single best form of disaggregation and no best hardware. Choose based on your model, workload, latency needs and throughput guarantees." ], "topics": [ "inference", "llm serving", "kv cache", "disaggregation", "speculative decoding", "hardware", "kernel optimization", "agents" ], "tools": [ "Cerebras", "Cerebras CS-3", "Gimlet Labs", "KForge", "AWS Trainium", "AWS EFA", "Nvidia H100", "AMD", "d-Matrix", "Intel Gaudi", "Google", "Claude Code fast mode", "Anthropic", "ChatGPT", "Llama 3.3 70B", "GPT-OSS-120B", "Recursive" ], "quotes": [ { "text": "So, up to 90% of the time, the computer is just sitting there waiting on memory to load.", "at": "02:23", "url": "https://www.youtube.com/watch?v=_cRgl3maiT8&t=143s" }, { "text": "the rule of thumb here for a 70 billion parameter model is that for every 3,000 tokens, you need 1 GB of additional KV cache.", "at": "05:50", "url": "https://www.youtube.com/watch?v=_cRgl3maiT8&t=350s" }, { "text": "Cerebras has 7,000 times the bandwidth of a H100 with 21,000 terabytes per second.", "at": "12:43", "url": "https://www.youtube.com/watch?v=_cRgl3maiT8&t=763s" }, { "text": "there is no best form of disagg. There is actually no best hardware.", "at": "18:09", "url": "https://www.youtube.com/watch?v=_cRgl3maiT8&t=1089s" } ], "words": 4916 }, "summary_url": "/#_cRgl3maiT8", "transcript": { "html": "/transcripts/_cRgl3maiT8.html", "txt": "/transcripts/_cRgl3maiT8.txt", "vtt": "/transcripts/_cRgl3maiT8.vtt" } }