What's New in Inference Engineering — Philip Kiely, Baseten

AI Engineer · 19 min · 190 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Um, I'm here to talk about news in impact engineering.
00:17So, hello, I'm Philip, and I'm here because I wrote a book.
00:20Um, this is my third year at the AI Engineer World's Fair.
00:23This is my favorite conference in the world.
00:26This is the main event of the year every time.
00:29I really started my journey as a speaker and engineer right here in 2024.
00:33I came back in 2025 and did a bunch of things .
00:36I'm here again.
00:38I like it here, and I am very grateful to the organizers for always inviting me.
00:42Um, I wrote this book called " Influence Engineering."
00:46We released it three or four months ago, and I'm just amazed by the reaction.
00:51Um , we did more than...I, yes, I posted it on February 23rd.
00:56Um, at this point we've sold over 11,000 hard copies, we're approaching 30,000 digital copies, and 24 million people in the world, or 24 million Twitter accounts, so let's see how many people
01:11actually saw something from the influence engineering.
01:14And despite this, you know, wonderful reception, there's one question that people ask me all the time.
01:21Why did you even have to do this?
01:23So, why write a book about something that is changing so quickly?
01:27Well, you know, I think a lot of the principles of impact engineering are pretty well established at this point , and there's a lot that we can, you know, learn from and kind of iterate
01:37over generations of models; but today I'm here to talk about what's new in impact engineering.
01:43This is the first public supplement with all the new information that has appeared since the book was released.
01:49We'll review the principles of impact engineering a bit.
01:53And then let's talk about everything that happened after February 23, 2026 in the world of influence.
01:59We'll talk about what happened to Turbo Quant and a little bit about KV-compactness .
02:04We'll be spending a lot of time on D-flash and some other new cool stuff in speculative decoding.
02:10And then I'll get a little philosophical, make a little prediction about what I think will happen in the sphere of influence in the near future, which I'll be happy to talk about, hopefully, the
02:21next time you see me here.
02:25Class.
02:25So, let's begin.
02:28So, you know, one thing that I've picked up now from a lot of conversations with people about influence is a few common principles.
02:35And one of the most important topics: I was on a podcast with Salo the other day.
02:42We talked about inference, and you know, two types of inference engineering have emerged.
02:48There is local inference, where the main strategy is to simply make the model run on any hardware, compressing it using quantization, distillation, pruning, whatever, splitting it across whatever GPUs you have at home.
03:04First you make her work, then you make her a little smarter.
03:09You fix all the catastrophic problems that the model compression caused and try to get it back to a baseline level of intelligence at a batch size of one.
03:20And then there's my world—the world of data centers with a batch size of N, where the main thing is to simply run the model, configure the LLM build to make it work, and then make it less slow.
03:30Do things like KV cache-aware routing, speculation, disaggregation.
03:35And you know, within these two worlds, I think we have a lot to learn from each other.
03:40In this talk, I'll focus on advances in inference engineering for data centers, because that's what I know, but there's a lot of cool stuff happening in the on-premises world too
03:49.
03:50So, in a book about inference engineering, I usually assume that the scale is a finished product.
03:57And I think the moment of transfer from learning to conclusion is important to remember, it's a good way to delineate these areas.
04:07However, lately I've been noticing more and more that many optimizations for inference come from a special learning process.
04:15So the lines between learning and inference are becoming increasingly blurred.
04:19And this is an interesting point worth remembering.
04:22We see a cycle where you get faster inference, which yields more data, which you use to train a better model, which yields even faster inference, and you just keep
04:32doing it until you become extremely rich.
04:34So, with regard to learning for inference, we have a number of new methods to discuss within what I call the "big three."
04:44So we'll talk about news in quantization, news in caching, in particular the KV cache mechanism, and news in the area of speculation.
04:53Because that's, you know, there's a lot more to the world of inference, including what I'll talk about at the end.
05:00But when it comes to practical, everyday questions about how to speed up a Model X, people usually resort to these three methods.
05:08So, first of all .
05:11I am publishing a book.
05:13This is February.
05:14I feel great.
05:15I think, "Wow, everything I need to know about inference in one place."
05:20And here, uh, we have news in the world of quantization.
05:26So, for a quick overview: quantization is the use of a less precise numerical format to save bandwidth and computation to improve TTFT and TPS.
05:39This usually depends on the equipment, saves money, but may slightly reduce the quality of the model.
05:47By the way, if you want to hear my full story on quantization, I spoke at AI Engineer Miami last month about how quantization isn't all that "evil" and there are many ways to preserve quality during the
06:00process.
06:02I was happy with my explanation of quantization, and then 20 million people saw Octo Quant, and it even caused a short-term drop in memory stocks because everyone thought, "Oh, memory is going to be a lot more efficient
06:16now."
06:18"We don't need flash memory anymore."
06:20Which was a mistake.
06:22But, in any case, it was a new approach to quantization, popular this year, that uses polar coordinates and allows quantizing the KV cache to 4 bits.
06:34It became incredibly popular, and I thought, "Wow, I did n't even think about that, how does that even work?"
06:43So our team did a bunch of research on this .
06:47If you know, congratulations Waterloo in tone on Twitter.
06:51Ali is from our model performance team.
06:55Actually, I'm not sure if he's still in tone.
06:57But yes, he is from Waterloo.
06:59In short, he wrote a great article about the mathematics of Turbo Quant, and the main advantage is that you can represent the KV cache with 4 bits instead of 8. You save twice as much space and bandwidth
07:13, effectively getting twice as much speed when moving the KV cache in system memory.
07:19But there is a significant drawback to, uh, Turbo Quant.
07:22It turns out that additional calculations need to be performed during the direct pass for decoding, and this reduces the TPS by more than half.
07:29And this is simply an unacceptable compromise for many industrial use scenarios.
07:34So we've looked carefully at TurboQuant, but we don't use it for any of these real-world workloads.
07:40We still use traditional full quantization NVFP4.
07:44However, this is actually a great method for those who run models locally.
07:49So if you're running a model, especially a language model with a long context, on your local computer, on a GPU in the basement, you have very limited memory.
07:59This is the main " bottleneck".
08:02So anything that can free up memory from the KV cache and allow you to fit longer sequences there will be very valuable.
08:10And the additional calculations of the direct pass will be a less significant disadvantage.
08:15So, TurboQuant is still a fantastic piece of research, a truly remarkable method that ultimately turned out to be less suitable for use in data centers than it might have seemed at first glance.
08:27Instead, we focus on NVFP4 with an emphasis on weight quantization rather than KV cache, trying to preserve the bounds in the quantized weights as much as possible and making sure we don't flatten
08:39our probability distributions.
08:42For the KV cache itself, we focus on KV-aware routing, offloading, KV sharing, using NCCL, Nvidia Dynamo, and other tools to move the KV cache around the system and potentially
08:57offload it to CPU, regular memory, etc., rather than trying to use TurboQuant to compress it.
09:06We also focus on cross- modality quantization, considering how to apply the benefits of NVFP4 not only to language models, but also to image and video processing models.
09:18Ali also wrote a lot of interesting things about this on Twitter, be sure to check it out.
09:22So, despite everything, KV cache remains very important.
09:25And let's talk about him.
09:27Let's talk about KV cache compaction .
09:29Again, a brief overview.
09:30In KV cache, if you enter the same query with the same prefix, you can reuse the tokens you calculated during the previous fill.
09:37This makes your entire system faster and more efficient.
09:42In general, KV cache is lossless memory.
09:45There are only a few sources of lossless memory when we think about our output system.
09:51We have the request content, the context, and we have the KV cache.
09:54And it will scale linearly depending on the amount of data you transfer.
09:58And now, if you think about , you know, sequences that are a million tokens long, it becomes quite significant.
10:04So , many people are thinking about how to compress memory?
10:06How to compress context?
10:08Agent systems will compress context.
10:11RAG, search, all these methods that we've been talking about for years are kind of compressing a large context into something that can be passed to a model.
10:19You can write to files.
10:21All of these things scale sublinearly with the amount of data you have.
10:25But what if there was a middle ground?
10:28What if there was a way to get a fairly significant compression of the data you remember, with almost complete preservation of the information?
10:38So, we have many different ways to think about what to cache.
10:42You know, recent compaction techniques have shown that we can replace the cache with a much shorter one.
10:48We have work like attention matching and cartridges that have produced really promising results with high compression ratios, but both of them work at output time.
10:59Again , one of the techniques, or one of the topics I want to talk about , is learning to infer.
11:05So in this case, I want to introduce something called Still from the Base 10 research team, where synthesis on top of a cache, where we store a learned representation of
11:15information, rather than the information itself or a deterministic subset of it, is amortized through learning.
11:22So Charlie and Mudit from my post-workout training team recently gave a great presentation at Coster Compile.
11:30It's on YouTube.
11:31I would recommend you to watch it if you are interested in learning about KV compactification.
11:37Unfortunately, I have neither the time nor the genius to explain everything here .
11:42But the basic mechanism is that Still is a kind of bottleneck that takes a fixed set of learned query vectors, cross-matches them against the full KV cache, and produces a set of compact keys and values in a single direct
11:57pass.
11:58This creates a fast differentiated compressed memory that the LLM can pay attention to as if it were a real context.
12:04So, if you are interested in KV compactification, be sure to check out Charlie and Mudit's work.
12:10It was really great to learn.
12:13So, those are two methods.
12:15We talked about quantization, we talked about caching.
12:19The latter is a speculative execution, and there have been many changes here.
12:23So, to recap : speculative decoding—we take draft tokens, validate them during a direct pass, and use this to generate more than one token per pass.
12:34This helps significantly with the number of tokens per second and is a lossless optimization , which is great because we don't have to worry about quality at all.
12:43Now, regarding the history of speculation, we started with speculative decoding.
12:47All of this is in the book.
12:49You have SpecDec, which uses a small model from the same family to generate token drafts.
12:54It turns out that small models are not very good at generating token drafts, they are just good small models.
13:01So we as an industry invented a bunch of new methods, like Medusa, where you just add heads to the model, and then Eagle 3 came along, the idea of which was: what if instead of a tiny model, we train
13:12a billion-parameter model on the hidden states of the target model to generate drafts?
13:19And it worked really well.
13:22So, as of February this year, Eagle 3 was the best method in speculative execution.
13:30Now we have Dflash.
13:31Dflash is even better.
13:32This is diffusion for speculative execution.
13:35Dflash creates a sequence of token drafts instead of a single token.
13:39The model is a diffusion language model, meaning that it generates an entire sequence of tokens in the same way that video or image generation generates a sequence of frames or pixels, iteratively improving
13:53them, instead of autoregressive token generation.
13:57Dflash models may be two or four times slower, but they predict eight or 16 tokens simultaneously in this window, while Eagle only does one at a time.
14:07Therefore, a single direct Dflash pass is faster than the entire Eagle draft generation phase and predicts more tokens.
14:13These tokens can leverage cross-talk between themselves and typically create a higher level of adoption, because in speculative execution, adoption is everything.
14:23So, in practice, we see more than a threefold improvement thanks to Dflash.
14:29This is measured on a single B200 with 38B quantization, and we see that compared to Eagle this is a significant improvement in both token acceptance rate and number of tokens per second.
14:43These Dflash models are trained with an attention mask for bidirectional draft generation.
14:51So the target model will provide context, and within each block we will have a subset of pure tokens being selected, and the attention mask will provide causal consistency, but it will still allow for
15:02bidirectional attention, whereas in a traditional autoregressive model you only look at tokens in one direction.
15:10That's why we can take advantage of this diffusion-based architecture.
15:17And so I thought I was done, but a few days ago Dspot appeared.
15:22We have already launched Dflash into production.
15:24And Dspot is a new study.
15:27So all I can say about this is, “Hey, it exists.
15:30This is cool.
15:31We are studying it."
15:33The difference compared to Dflash is that it still has the same diffusion model, but also combines it with a sequential model.
15:41The idea is that we will improve the adoption rates by having these two models work together, instead of using just the iterative speculator, just the diffusion speculator, or just the autoregressive
15:52speculator.
15:53So, Dspot is very interesting, but we don't have any results from using it in production yet.
15:59Hopefully we'll have them next time .
16:02However, we have the results of the productive use of continuous retraining of the speculator.
16:09We return to Dflash again; the idea is that speculative decoding depends very much on the actual content of the requests and responses you see on your system.
16:19So, if you continuously retrain on these requests and responses in your production system, you can get an improvement in token acceptance rates from 20% to 2 times.
16:32In fact, this is very difficult to implement.
16:35This requires a lot of memory, and you should make sure you have permission to use the data you are processing in this way.
16:41This requires huge computing power, you need to constantly move all this information around, and if you change the base model, you also have to change the speculator model.
16:51But looking to the future, I believe that continuous speculation for very large systems will be a worthwhile optimization.
17:01So what's next in inference?
17:02What follows is my personal opinion, assumptions, and publicly available information; if I knew anything about what was actually being prepared for release, I wouldn't be able to talk about it.
17:13So, this is just what I think will happen.
17:17You know, I went through three hardware cycles: the Ampere, Hopper, and Blackwell releases.
17:22It always takes time from the moment chips are shipped to their installation in data centers for the entire software stack to be able to fully utilize their capabilities.
17:33But there's one thing that particularly excites me: it looks like the performance of NVFP4 on a Rubin system will be fantastic.
17:43So the more we can borrow methods from local inference and confidently run models in the NVFP4 data format, the better we can leverage the amazing performance of future Rubin systems.
17:57I believe that disaggregation and system-wide communication will become increasingly important.
18:03We are seeing great early results from PD disaggregation, and the ability to move information like KV cache data across the system will become even more important.
18:14As I said, the topic of training for breeding will continue to have a significant impact on the development of the industry.
18:24So thank you very much to everyone who came to this lecture.
18:27I am on Twitter.
18:28I am on LinkedIn.
18:29And I also give away free books.
18:32You can download the PDF using the QR code or come with me to the Base 10 booth to get your copy of "Influence Engineering."
18:38We have a lot of them there.
18:40Maybe there will be enough for everyone.
18:42If not, we'll ask Koyo to bring some more from the office.
18:45So yes, I'll be down by the Base 10 booth.
18:47Thank you very much again everyone and have a great day.