What's New in Inference Engineering — Philip Kiely, Baseten
https://www.youtube.com/watch?v=75ckHC2LU_0
AI Engineer · 19 min · 190 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:12] Um, I'm here to talk about news in impact engineering.
[00:17] So, hello, I'm Philip, and I'm here because I wrote a book.
[00:20] Um, this is my third year at the AI ​​ Engineer World's Fair.
[00:23] This is my favorite conference in the world.
[00:26] This is the main event of the year every time.
[00:29] I really started my journey as a speaker and engineer right here in 2024.
[00:33] I came back in 2025 and did a bunch of things .
[00:36] I'm here again.
[00:38] I like it here, and I am very grateful to the organizers for always inviting me.
[00:42] Um, I wrote this book called " Influence Engineering."
[00:46] We released it three or four months ago, and I'm just amazed by the reaction.
[00:51] Um , we did more than...I, yes, I posted it on February 23rd.
[00:56] Um, at this point we've sold over 11,000 hard copies, we're approaching 30,000 digital copies, and 24 million people in the world, or 24 million Twitter accounts, so let's see how many people
[01:11] actually saw something from the influence engineering.
[01:14] And despite this, you know, wonderful reception, there's one question that people ask me all the time.
[01:21] Why did you even have to do this?
[01:23] So, why write a book about something that is changing so quickly?
[01:27] Well, you know, I think a lot of the principles of impact engineering are pretty well established at this point , and there's a lot that we can, you know, learn from and kind of iterate
[01:37] over generations of models; but today I'm here to talk about what's new in impact engineering.
[01:43] This is the first public supplement with all the new information that has appeared since the book was released.
[01:49] We'll review the principles of impact engineering a bit.
[01:53] And then let's talk about everything that happened after February 23, 2026 in the world of influence.
[01:59] We'll talk about what happened to Turbo Quant and a little bit about KV-compactness .
[02:04] We'll be spending a lot of time on D-flash and some other new cool stuff in speculative decoding.
[02:10] And then I'll get a little philosophical, make a little prediction about what I think will happen in the sphere of influence in the near future, which I'll be happy to talk about, hopefully, the
[02:21] next time you see me here.
[02:25] Class.
[02:25] So, let's begin.
[02:28] So, you know, one thing that I've picked up now from a lot of conversations with people about influence is a few common principles.
[02:35] And one of the most important topics: I was on a podcast with Salo the other day.
[02:42] We talked about inference, and you know, two types of inference engineering have emerged.
[02:48] There is local inference, where the main strategy is to simply make the model run on any hardware, compressing it using quantization, distillation, pruning, whatever, splitting it across whatever GPUs you have at home.
[03:04] First you make her work, then you make her a little smarter.
[03:09] You fix all the catastrophic problems that the model compression caused and try to get it back to a baseline level of intelligence at a batch size of one.
[03:20] And then there's my world—the world of data centers with a batch size of N, where the main thing is to simply run the model, configure the LLM build to make it work, and then make it less slow.
[03:30] Do things like KV cache-aware routing, speculation, disaggregation.
[03:35] And you know, within these two worlds, I think we have a lot to learn from each other.
[03:40] In this talk, I'll focus on advances in inference engineering for data centers, because that's what I know, but there's a lot of cool stuff happening in the on-premises world too
[03:49] .
[03:50] So, in a book about inference engineering, I usually assume that the scale is a finished product.
[03:57] And I think the moment of transfer from learning to conclusion is important to remember, it's a good way to delineate these areas.
[04:07] However, lately I've been noticing more and more that many optimizations for inference come from a special learning process.
[04:15] So the lines between learning and inference are becoming increasingly blurred.
[04:19] And this is an interesting point worth remembering.
[04:22] We see a cycle where you get faster inference, which yields more data, which you use to train a better model, which yields even faster inference, and you just keep
[04:32] doing it until you become extremely rich.
[04:34] So, with regard to learning for inference, we have a number of new methods to discuss within what I call the "big three."
[04:44] So we'll talk about news in quantization, news in caching, in particular the KV cache mechanism, and news in the area of ​​ speculation.
[04:53] Because that's, you know, there's a lot more to the world of inference, including what I'll talk about at the end.
[05:00] But when it comes to practical, everyday questions about how to speed up a Model X, people usually resort to these three methods.
[05:08] So, first of all .
[05:11] I am publishing a book.
[05:13] This is February.
[05:14] I feel great.
[05:15] I think, "Wow, everything I need to know about inference in one place."
[05:20] And here, uh, we have news in the world of quantization.
[05:26] So, for a quick overview: quantization is the use of a less precise numerical format to save bandwidth and computation to improve TTFT and TPS.
[05:39] This usually depends on the equipment, saves money, but may slightly reduce the quality of the model.
[05:47] By the way, if you want to hear my full story on quantization, I spoke at AI Engineer Miami last month about how quantization isn't all that "evil" and there are many ways to preserve quality during the
[06:00] process.
[06:02] I was happy with my explanation of quantization, and then 20 million people saw Octo Quant, and it even caused a short-term drop in memory stocks because everyone thought, "Oh, memory is going to be a lot more efficient
[06:16] now."
[06:18] "We don't need flash memory anymore."
[06:20] Which was a mistake.
[06:22] But, in any case, it was a new approach to quantization, popular this year, that uses polar coordinates and allows quantizing the KV cache to 4 bits.
[06:34] It became incredibly popular, and I thought, "Wow, I did n't even think about that, how does that even work?"
[06:43] So our team did a bunch of research on this .
[06:47] If you know, congratulations Waterloo in tone on Twitter.
[06:51] Ali is from our model performance team.
[06:55] Actually, I'm not sure if he's still in tone.
[06:57] But yes, he is from Waterloo.
[06:59] In short, he wrote a great article about the mathematics of Turbo Quant, and the main advantage is that you can represent the KV cache with 4 bits instead of 8. You save twice as much space and bandwidth
[07:13] , effectively getting twice as much speed when moving the KV cache in system memory.
[07:19] But there is a significant drawback to, uh, Turbo Quant.
[07:22] It turns out that additional calculations need to be performed during the direct pass for decoding, and this reduces the TPS by more than half.
[07:29] And this is simply an unacceptable compromise for many industrial use scenarios.
[07:34] So we've looked carefully at TurboQuant, but we don't use it for any of these real-world workloads.
[07:40] We still use traditional full quantization NVFP4.
[07:44] However, this is actually a great method for those who run models locally.
[07:49] So if you're running a model, especially a language model with a long context, on your local computer, on a GPU in the basement, you have very limited memory.
[07:59] This is the main " bottleneck".
[08:02] So anything that can free up memory from the KV cache and allow you to fit longer sequences there will be very valuable.
[08:10] And the additional calculations of the direct pass will be a less significant disadvantage.
[08:15] So, TurboQuant is still a fantastic piece of research, a truly remarkable method that ultimately turned out to be less suitable for use in data centers than it might have seemed at first glance.
[08:27] Instead, we focus on NVFP4 with an emphasis on weight quantization rather than KV cache, trying to preserve the bounds in the quantized weights as much as possible and making sure we don't flatten
[08:39] our probability distributions.
[08:42] For the KV cache itself, we focus on KV-aware routing, offloading, KV sharing, using NCCL, Nvidia Dynamo, and other tools to move the KV cache around the system and potentially
[08:57] offload it to CPU, regular memory, etc., rather than trying to use TurboQuant to compress it.
[09:06] We also focus on cross- modality quantization, considering how to apply the benefits of NVFP4 not only to language models, but also to image and video processing models.
[09:18] Ali also wrote a lot of interesting things about this on Twitter, be sure to check it out.
[09:22] So, despite everything, KV cache remains very important.
[09:25] And let's talk about him.
[09:27] Let's talk about KV cache compaction .
[09:29] Again, a brief overview.
[09:30] In KV cache, if you enter the same query with the same prefix, you can reuse the tokens you calculated during the previous fill.
[09:37] This makes your entire system faster and more efficient.
[09:42] In general, KV cache is lossless memory.
[09:45] There are only a few sources of lossless memory when we think about our output system.
[09:51] We have the request content, the context, and we have the KV cache.
[09:54] And it will scale linearly depending on the amount of data you transfer.
[09:58] And now, if you think about , you know, sequences that are a million tokens long, it becomes quite significant.
[10:04] So , many people are thinking about how to compress memory?
[10:06] How to compress context?
[10:08] Agent systems will compress context.
[10:11] RAG, search, all these methods that we've been talking about for years are kind of compressing a large context into something that can be passed to a model.
[10:19] You can write to files.
[10:21] All of these things scale sublinearly with the amount of data you have.
[10:25] But what if there was a middle ground?
[10:28] What if there was a way to get a fairly significant compression of the data you remember, with almost complete preservation of the information?
[10:38] So, we have many different ways to think about what to cache.
[10:42] You know, recent compaction techniques have shown that we can replace the cache with a much shorter one.
[10:48] We have work like attention matching and cartridges that have produced really promising results with high compression ratios, but both of them work at output time.
[10:59] Again , one of the techniques, or one of the topics I want to talk about , is learning to infer.
[11:05] So in this case, I want to introduce something called Still from the Base 10 research team, where synthesis on top of a cache, where we store a learned representation of
[11:15] information, rather than the information itself or a deterministic subset of it, is amortized through learning.
[11:22] So Charlie and Mudit from my post-workout training team recently gave a great presentation at Coster Compile.
[11:30] It's on YouTube.
[11:31] I would recommend you to watch it if you are interested in learning about KV compactification.
[11:37] Unfortunately, I have neither the time nor the genius to explain everything here .
[11:42] But the basic mechanism is that Still is a kind of bottleneck that takes a fixed set of learned query vectors, cross-matches them against the full KV cache, and produces a set of compact keys and values ​​ in a single direct
[11:57] pass.
[11:58] This creates a fast differentiated compressed memory that the LLM can pay attention to as if it were a real context.
[12:04] So, if you are interested in KV compactification, be sure to check out Charlie and Mudit's work.
[12:10] It was really great to learn.
[12:13] So, those are two methods.
[12:15] We talked about quantization, we talked about caching.
[12:19] The latter is a speculative execution, and there have been many changes here.
[12:23] So, to recap : speculative decoding—we take draft tokens, validate them during a direct pass, and use this to generate more than one token per pass.
[12:34] This helps significantly with the number of tokens per second and is a lossless optimization , which is great because we don't have to worry about quality at all.
[12:43] Now, regarding the history of speculation, we started with speculative decoding.
[12:47] All of this is in the book.
[12:49] You have SpecDec, which uses a small model from the same family to generate token drafts.
[12:54] It turns out that small models are not very good at generating token drafts, they are just good small models.
[13:01] So we as an industry invented a bunch of new methods, like Medusa, where you just add heads to the model, and then Eagle 3 came along, the idea of ​​ which was: what if instead of a tiny model, we train
[13:12] a billion-parameter model on the hidden states of the target model to generate drafts?
[13:19] And it worked really well.
[13:22] So, as of February this year, Eagle 3 was the best method in speculative execution.
[13:30] Now we have Dflash.
[13:31] Dflash is even better.
[13:32] This is diffusion for speculative execution.
[13:35] Dflash creates a sequence of token drafts instead of a single token.
[13:39] The model is a diffusion language model, meaning that it generates an entire sequence of tokens in the same way that video or image generation generates a sequence of frames or pixels, iteratively improving
[13:53] them, instead of autoregressive token generation.
[13:57] Dflash models may be two or four times slower, but they predict eight or 16 tokens simultaneously in this window, while Eagle only does one at a time.
[14:07] Therefore, a single direct Dflash pass is faster than the entire Eagle draft generation phase and predicts more tokens.
[14:13] These tokens can leverage cross-talk between themselves and typically create a higher level of adoption, because in speculative execution, adoption is everything.
[14:23] So, in practice, we see more than a threefold improvement thanks to Dflash.
[14:29] This is measured on a single B200 with 38B quantization, and we see that compared to Eagle this is a significant improvement in both token acceptance rate and number of tokens per second.
[14:43] These Dflash models are trained with an attention mask for bidirectional draft generation.
[14:51] So the target model will provide context, and within each block we will have a subset of pure tokens being selected, and the attention mask will provide causal consistency, but it will still allow for
[15:02] bidirectional attention, whereas in a traditional autoregressive model you only look at tokens in one direction.
[15:10] That's why we can take advantage of this diffusion-based architecture.
[15:17] And so I thought I was done, but a few days ago Dspot appeared.
[15:22] We have already launched Dflash into production.
[15:24] And Dspot is a new study.
[15:27] So all I can say about this is, “Hey, it exists.
[15:30] This is cool.
[15:31] We are studying it."
[15:33] The difference compared to Dflash is that it still has the same diffusion model, but also combines it with a sequential model.
[15:41] The idea is that we will improve the adoption rates by having these two models work together, instead of using just the iterative speculator, just the diffusion speculator, or just the autoregressive
[15:52] speculator.
[15:53] So, Dspot is very interesting, but we don't have any results from using it in production yet.
[15:59] Hopefully we'll have them next time .
[16:02] However, we have the results of the productive use of continuous retraining of the speculator.
[16:09] We return to Dflash again; the idea is that speculative decoding depends very much on the actual content of the requests and responses you see on your system.
[16:19] So, if you continuously retrain on these requests and responses in your production system, you can get an improvement in token acceptance rates from 20% to 2 times.
[16:32] In fact, this is very difficult to implement.
[16:35] This requires a lot of memory, and you should make sure you have permission to use the data you are processing in this way.
[16:41] This requires huge computing power, you need to constantly move all this information around, and if you change the base model, you also have to change the speculator model.
[16:51] But looking to the future, I believe that continuous speculation for very large systems will be a worthwhile optimization.
[17:01] So what's next in inference?
[17:02] What follows is my personal opinion, assumptions, and publicly available information; if I knew anything about what was actually being prepared for release, I wouldn't be able to talk about it.
[17:13] So, this is just what I think will happen.
[17:17] You know, I went through three hardware cycles: the Ampere, Hopper, and Blackwell releases.
[17:22] It always takes time from the moment chips are shipped to their installation in data centers for the entire software stack to be able to fully utilize their capabilities.
[17:33] But there's one thing that particularly excites me: it looks like the performance of NVFP4 on a Rubin system will be fantastic.
[17:43] So the more we can borrow methods from local inference and confidently run models in the NVFP4 data format, the better we can leverage the amazing performance of future Rubin systems.
[17:57] I believe that disaggregation and system-wide communication will become increasingly important.
[18:03] We are seeing great early results from PD disaggregation, and the ability to move information like KV cache data across the system will become even more important.
[18:14] As I said, the topic of training for breeding will continue to have a significant impact on the development of the industry.
[18:24] So thank you very much to everyone who came to this lecture.
[18:27] I am on Twitter.
[18:28] I am on LinkedIn.
[18:29] And I also give away free books.
[18:32] You can download the PDF using the QR code or come with me to the Base 10 booth to get your copy of "Influence Engineering."
[18:38] We have a lot of them there.
[18:40] Maybe there will be enough for everyone.
[18:42] If not, we'll ask Koyo to bring some more from the office.
[18:45] So yes, I'll be down by the Base 10 booth.
[18:47] Thank you very much again everyone and have a great day.