{ "video": { "id": "75ckHC2LU_0", "title": "What's New in Inference Engineering — Philip Kiely, Baseten", "duration": 1149, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "75ckHC2LU_0", "title": "What's New in Inference Engineering — Philip Kiely, Baseten", "one_liner": "Baseten's Philip Kiely gives the first public supplement to his book *Inference Engineering*, covering what changed after February 2026 across the \"big three\" — quantization, KV caching and speculation — with the headline that diffusion-based speculative decoding (DFlash) beats Eagle 3 by more than 3x in production.", "summary": "Kiely splits inference engineering into two worlds — local inference (batch size one, make it run at all on whatever GPUs you have) and data-center inference (batch size N, make it less slow) — and argues the two have a lot to learn from each other. He then updates the book's \"big three\" optimizations: TurboQuant's 4-bit KV cache turned out to be great for local but unusable in the data center because the extra forward-pass math halves TPS, so Baseten stays on NVFP4 weight quantization; KV cache compaction is moving from inference-time tricks to learned representations (Baseten's \"Still\"); and speculative decoding has jumped from Eagle 3 to DFlash, a diffusion draft model. A recurring theme is that the line between training and inference is blurring — more and more inference optimizations come out of a special training process. He closes with predictions: NVFP4 on Rubin, PD disaggregation, and continuous speculator retraining.", "key_points": [ "Two kinds of inference engineering have emerged: local (batch size one — compress via quantization/distillation/pruning, split across whatever GPUs are at home, then repair the catastrophic damage compression caused) and data centers (batch size N — get it running on the LLM build, then make it less slow with KV-cache-aware routing, speculation, disaggregation).", "TurboQuant uses polar coordinates to quantize the KV cache to 4 bits instead of 8 — 2x the space and bandwidth savings — and went viral enough (roughly 20 million views) to cause a short-term drop in memory stocks. But it needs extra computation in the decode forward pass, cutting TPS by more than half, which Kiely calls an unacceptable trade for industrial workloads. Baseten evaluated it carefully and runs none of it in production.", "Baseten instead uses traditional full NVFP4 quantization focused on weights rather than KV cache, trying to preserve bounds in the quantized weights and avoid flattening probability distributions, and is extending NVFP4 across modalities to image and video models.", "For the KV cache itself Baseten leans on KV-aware routing, offloading (including to CPU/regular memory), and KV sharing using NCCL and NVIDIA Dynamo rather than compressing it with TurboQuant.", "KV cache is lossless memory that scales linearly with context; RAG, search, agentic context compression and writing to files all scale sublinearly. Compaction is the middle ground: attention matching and cartridges get high compression ratios but both operate at inference time.", "\"Still,\" from the Baseten research team (work by Charlie and Mudit on the post-training team, presented publicly and on YouTube), is a bottleneck that takes a fixed set of learned query vectors, cross-attends them against the full KV cache and emits compact keys and values in a single forward pass — a fast, differentiable compressed memory the LLM attends to as if it were real context, with the cost amortized into training.", "Speculation lineage: SpecDec used a small same-family model (\"small models are not very good at generating token drafts, they are just good small models\"), then Medusa added heads to the model, then Eagle 3 trained a ~1B-parameter model on the target model's hidden states — the best method as of February.", "DFlash is a diffusion language model as the drafter: it emits a whole sequence of draft tokens. The model may be 2-4x slower per step but predicts 8 or 16 tokens at once versus Eagle's one, so a single DFlash forward pass is faster than Eagle's entire draft phase and produces more tokens with cross-talk between them, raising the acceptance rate. Measured on a single B200 with a 38B quantized model, Baseten sees more than 3x improvement over Eagle in both acceptance rate and tokens per second. DFlash models are trained with an attention mask that keeps causal consistency across blocks while allowing bidirectional attention within them.", "DSpot appeared a few days before the talk — same diffusion model combined with a sequential/autoregressive speculator to push acceptance rates higher. Baseten is studying it and has no production results yet. DFlash is already in production.", "Continuously retraining the speculator on your own production requests and responses yields 20% to 2x better token acceptance, because speculation depends heavily on actual traffic content — but it is very hard: lots of memory, huge compute, constant data movement, data-use permission, and a new speculator every time the base model changes." ], "takeaways": [ "Don't adopt TurboQuant for a served, batched workload — the decode-time compute more than halves TPS. Do consider it if you're running long-context models locally where KV cache memory is the bottleneck and the extra forward-pass math matters less.", "For data-center serving, put quantization effort into NVFP4 weights (preserving bounds, not flattening distributions) and solve KV cache pressure with routing, offloading and sharing via NCCL/NVIDIA Dynamo instead of compressing it.", "Move off Eagle 3 to DFlash for speculative decoding — it's lossless, already in production at Baseten, and worth >3x on a B200 with a 38B quantized model.", "If you run a very large system, budget for continuous speculator retraining on your own traffic (20%–2x acceptance gains), and confirm you have permission to train on the data you're serving.", "Expect more optimizations to come out of the training process rather than the serving stack, and plan the stack so you can confidently run NVFP4 end to end before Rubin systems land." ], "topics": [ "inference", "quantization", "kv-cache", "speculative-decoding", "diffusion-models", "gpu-hardware", "serving-infrastructure", "post-training" ], "tools": [ "Baseten", "Inference Engineering (book)", "TurboQuant", "NVFP4", "NCCL", "NVIDIA Dynamo", "Still (Baseten research)", "cartridges", "attention matching", "SpecDec", "Medusa", "Eagle 3", "DFlash", "DSpot", "NVIDIA B200", "Rubin", "Blackwell", "Hopper", "Ampere" ], "quotes": [ { "text": "We see a cycle where you get faster inference, which yields more data, which you use to train a better model, which yields even faster inference, and you just keep doing it until you become extremely rich.", "at": "04:23", "url": "https://www.youtube.com/watch?v=75ckHC2LU_0&t=263s" }, { "text": "TurboQuant is still a fantastic piece of research, a truly remarkable method that ultimately turned out to be less suitable for use in data centers than it might have seemed at first glance.", "at": "08:15", "url": "https://www.youtube.com/watch?v=75ckHC2LU_0&t=495s" }, { "text": "It turns out that small models are not very good at generating token drafts, they are just good small models.", "at": "12:54", "url": "https://www.youtube.com/watch?v=75ckHC2LU_0&t=774s" }, { "text": "These tokens can leverage cross-talk between themselves and typically create a higher level of adoption, because in speculative execution, adoption is everything.", "at": "14:13", "url": "https://www.youtube.com/watch?v=75ckHC2LU_0&t=853s" } ], "words": 3841 }, "summary_url": "/#75ckHC2LU_0", "transcript": { "html": "/transcripts/75ckHC2LU_0.html", "txt": "/transcripts/75ckHC2LU_0.txt", "vtt": "/transcripts/75ckHC2LU_0.vtt" } }