WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=75ckHC2LU_0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/75ckHC2LU_0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.986 --> 00:00:17.204
Um, I'm here to talk about news in impact engineering.

s2
00:00:17.204 --> 00:00:20.300
So, hello, I'm Philip, and I'm here because I wrote a book.

s3
00:00:20.300 --> 00:00:23.726
Um, this is my third year at the AI ​​ Engineer World's Fair.

s4
00:00:23.726 --> 00:00:26.780
This is my favorite conference in the world.

s5
00:00:26.780 --> 00:00:29.159
This is the main event of the year every time.

s6
00:00:29.159 --> 00:00:33.811
I really started my journey as a speaker and engineer right here in 2024.

s7
00:00:33.811 --> 00:00:36.974
I came back in 2025 and did a bunch of things .

s8
00:00:36.974 --> 00:00:38.090
I'm here again.

s9
00:00:38.090 --> 00:00:42.519
I like it here, and I am very grateful to the organizers for always inviting me.

s10
00:00:42.519 --> 00:00:46.239
Um, I wrote this book called " Influence Engineering."

s11
00:00:46.239 --> 00:00:51.186
We released it three or four months ago, and I'm just amazed by the reaction.

s12
00:00:51.186 --> 00:00:56.633
Um , we did more than...I, yes, I posted it on February 23rd.

s13
00:00:56.633 --> 00:01:11.015
Um, at this point we've sold over 11,000 hard copies, we're approaching 30,000 digital copies, and 24 million people in the world, or 24 million Twitter accounts, so let's see how many people

s14
00:01:11.015 --> 00:01:14.677
actually saw something from the influence engineering.

s15
00:01:14.677 --> 00:01:21.041
And despite this, you know, wonderful reception, there's one question that people ask me all the time.

s16
00:01:21.041 --> 00:01:23.733
Why did you even have to do this?

s17
00:01:23.733 --> 00:01:27.936
So, why write a book about something that is changing so quickly?

s18
00:01:27.936 --> 00:01:37.742
Well, you know, I think a lot of the principles of impact engineering are pretty well established at this point , and there's a lot that we can, you know, learn from and kind of iterate

s19
00:01:37.742 --> 00:01:43.701
over generations of models; but today I'm here to talk about what's new in impact engineering.

s20
00:01:43.701 --> 00:01:49.776
This is the first public supplement with all the new information that has appeared since the book was released.

s21
00:01:49.776 --> 00:01:53.396
We'll review the principles of impact engineering a bit.

s22
00:01:53.396 --> 00:01:59.931
And then let's talk about everything that happened after February 23, 2026 in the world of influence.

s23
00:01:59.931 --> 00:02:04.618
We'll talk about what happened to Turbo Quant and a little bit about KV-compactness .

s24
00:02:04.618 --> 00:02:10.324
We'll be spending a lot of time on D-flash and some other new cool stuff in speculative decoding.

s25
00:02:10.324 --> 00:02:21.722
And then I'll get a little philosophical, make a little prediction about what I think will happen in the sphere of influence in the near future, which I'll be happy to talk about, hopefully, the

s26
00:02:21.722 --> 00:02:25.337
next time you see me here.

s27
00:02:25.337 --> 00:02:25.849
Class.

s28
00:02:25.849 --> 00:02:28.530
So, let's begin.

s29
00:02:28.530 --> 00:02:35.572
So, you know, one thing that I've picked up now from a lot of conversations with people about influence is a few common principles.

s30
00:02:35.572 --> 00:02:42.582
And one of the most important topics: I was on a podcast with Salo the other day.

s31
00:02:42.582 --> 00:02:48.307
We talked about inference, and you know, two types of inference engineering have emerged.

s32
00:02:48.307 --> 00:03:04.235
There is local inference, where the main strategy is to simply make the model run on any hardware, compressing it using quantization, distillation, pruning, whatever, splitting it across whatever GPUs you have at home.

s33
00:03:04.235 --> 00:03:09.367
First you make her work, then you make her a little smarter.

s34
00:03:09.367 --> 00:03:20.037
You fix all the catastrophic problems that the model compression caused and try to get it back to a baseline level of intelligence at a batch size of one.

s35
00:03:20.037 --> 00:03:30.672
And then there's my world—the world of data centers with a batch size of N, where the main thing is to simply run the model, configure the LLM build to make it work, and then make it less slow.

s36
00:03:30.672 --> 00:03:35.190
Do things like KV cache-aware routing, speculation, disaggregation.

s37
00:03:35.190 --> 00:03:40.249
And you know, within these two worlds, I think we have a lot to learn from each other.

s38
00:03:40.249 --> 00:03:49.652
In this talk, I'll focus on advances in inference engineering for data centers, because that's what I know, but there's a lot of cool stuff happening in the on-premises world too

s39
00:03:49.652 --> 00:03:50.432
.

s40
00:03:50.432 --> 00:03:57.937
So, in a book about inference engineering, I usually assume that the scale is a finished product.

s41
00:03:57.937 --> 00:04:07.597
And I think the moment of transfer from learning to conclusion is important to remember, it's a good way to delineate these areas.

s42
00:04:07.597 --> 00:04:15.133
However, lately I've been noticing more and more that many optimizations for inference come from a special learning process.

s43
00:04:15.133 --> 00:04:19.064
So the lines between learning and inference are becoming increasingly blurred.

s44
00:04:19.064 --> 00:04:22.140
And this is an interesting point worth remembering.

s45
00:04:22.140 --> 00:04:32.179
We see a cycle where you get faster inference, which yields more data, which you use to train a better model, which yields even faster inference, and you just keep

s46
00:04:32.179 --> 00:04:34.768
doing it until you become extremely rich.

s47
00:04:34.768 --> 00:04:44.300
So, with regard to learning for inference, we have a number of new methods to discuss within what I call the "big three."

s48
00:04:44.300 --> 00:04:53.677
So we'll talk about news in quantization, news in caching, in particular the KV cache mechanism, and news in the area of ​​ speculation.

s49
00:04:53.677 --> 00:05:00.341
Because that's, you know, there's a lot more to the world of inference, including what I'll talk about at the end.

s50
00:05:00.341 --> 00:05:08.459
But when it comes to practical, everyday questions about how to speed up a Model X, people usually resort to these three methods.

s51
00:05:08.459 --> 00:05:11.140
So, first of all .

s52
00:05:11.140 --> 00:05:13.372
I am publishing a book.

s53
00:05:13.372 --> 00:05:14.419
This is February.

s54
00:05:14.419 --> 00:05:15.926
I feel great.

s55
00:05:15.926 --> 00:05:20.000
I think, "Wow, everything I need to know about inference in one place."

s56
00:05:20.000 --> 00:05:25.987
And here, uh, we have news in the world of quantization.

s57
00:05:26.571 --> 00:05:38.633
So, for a quick overview: quantization is the use of a less precise numerical format to save bandwidth and computation to improve TTFT and TPS.

s58
00:05:39.477 --> 00:05:47.496
This usually depends on the equipment, saves money, but may slightly reduce the quality of the model.

s59
00:05:47.496 --> 00:06:00.890
By the way, if you want to hear my full story on quantization, I spoke at AI Engineer Miami last month about how quantization isn't all that "evil" and there are many ways to preserve quality during the

s60
00:06:00.890 --> 00:06:02.506
process.

s61
00:06:02.506 --> 00:06:16.837
I was happy with my explanation of quantization, and then 20 million people saw Octo Quant, and it even caused a short-term drop in memory stocks because everyone thought, "Oh, memory is going to be a lot more efficient

s62
00:06:16.837 --> 00:06:18.040
now."

s63
00:06:18.040 --> 00:06:20.279
"We don't need flash memory anymore."

s64
00:06:20.279 --> 00:06:22.499
Which was a mistake.

s65
00:06:22.499 --> 00:06:34.164
But, in any case, it was a new approach to quantization, popular this year, that uses polar coordinates and allows quantizing the KV cache to 4 bits.

s66
00:06:34.164 --> 00:06:43.718
It became incredibly popular, and I thought, "Wow, I did n't even think about that, how does that even work?"

s67
00:06:43.877 --> 00:06:47.479
So our team did a bunch of research on this .

s68
00:06:47.479 --> 00:06:51.669
If you know, congratulations Waterloo in tone on Twitter.

s69
00:06:51.669 --> 00:06:55.187
Ali is from our model performance team.

s70
00:06:55.187 --> 00:06:57.590
Actually, I'm not sure if he's still in tone.

s71
00:06:57.590 --> 00:06:59.553
But yes, he is from Waterloo.

s72
00:06:59.553 --> 00:07:13.881
In short, he wrote a great article about the mathematics of Turbo Quant, and the main advantage is that you can represent the KV cache with 4 bits instead of 8. You save twice as much space and bandwidth

s73
00:07:13.881 --> 00:07:19.476
, effectively getting twice as much speed when moving the KV cache in system memory.

s74
00:07:19.476 --> 00:07:22.398
But there is a significant drawback to, uh, Turbo Quant.

s75
00:07:22.398 --> 00:07:29.849
It turns out that additional calculations need to be performed during the direct pass for decoding, and this reduces the TPS by more than half.

s76
00:07:29.849 --> 00:07:34.696
And this is simply an unacceptable compromise for many industrial use scenarios.

s77
00:07:34.696 --> 00:07:40.877
So we've looked carefully at TurboQuant, but we don't use it for any of these real-world workloads.

s78
00:07:40.877 --> 00:07:44.868
We still use traditional full quantization NVFP4.

s79
00:07:44.868 --> 00:07:49.259
However, this is actually a great method for those who run models locally.

s80
00:07:49.259 --> 00:07:59.547
So if you're running a model, especially a language model with a long context, on your local computer, on a GPU in the basement, you have very limited memory.

s81
00:07:59.547 --> 00:08:02.219
This is the main " bottleneck".

s82
00:08:02.219 --> 00:08:10.190
So anything that can free up memory from the KV cache and allow you to fit longer sequences there will be very valuable.

s83
00:08:10.190 --> 00:08:15.392
And the additional calculations of the direct pass will be a less significant disadvantage.

s84
00:08:15.392 --> 00:08:27.744
So, TurboQuant is still a fantastic piece of research, a truly remarkable method that ultimately turned out to be less suitable for use in data centers than it might have seemed at first glance.

s85
00:08:27.744 --> 00:08:39.512
Instead, we focus on NVFP4 with an emphasis on weight quantization rather than KV cache, trying to preserve the bounds in the quantized weights as much as possible and making sure we don't flatten

s86
00:08:39.512 --> 00:08:42.488
our probability distributions.

s87
00:08:42.488 --> 00:08:57.813
For the KV cache itself, we focus on KV-aware routing, offloading, KV sharing, using NCCL, Nvidia Dynamo, and other tools to move the KV cache around the system and potentially

s88
00:08:57.813 --> 00:09:06.413
offload it to CPU, regular memory, etc., rather than trying to use TurboQuant to compress it.

s89
00:09:06.413 --> 00:09:18.178
We also focus on cross- modality quantization, considering how to apply the benefits of NVFP4 not only to language models, but also to image and video processing models.

s90
00:09:18.178 --> 00:09:22.854
Ali also wrote a lot of interesting things about this on Twitter, be sure to check it out.

s91
00:09:22.854 --> 00:09:25.286
So, despite everything, KV cache remains very important.

s92
00:09:25.286 --> 00:09:27.124
And let's talk about him.

s93
00:09:27.124 --> 00:09:29.585
Let's talk about KV cache compaction .

s94
00:09:29.585 --> 00:09:30.742
Again, a brief overview.

s95
00:09:30.742 --> 00:09:37.862
In KV cache, if you enter the same query with the same prefix, you can reuse the tokens you calculated during the previous fill.

s96
00:09:37.862 --> 00:09:42.206
This makes your entire system faster and more efficient.

s97
00:09:42.439 --> 00:09:45.939
In general, KV cache is lossless memory.

s98
00:09:45.939 --> 00:09:51.382
There are only a few sources of lossless memory when we think about our output system.

s99
00:09:51.382 --> 00:09:54.555
We have the request content, the context, and we have the KV cache.

s100
00:09:54.555 --> 00:09:58.298
And it will scale linearly depending on the amount of data you transfer.

s101
00:09:58.298 --> 00:10:04.080
And now, if you think about , you know, sequences that are a million tokens long, it becomes quite significant.

s102
00:10:04.080 --> 00:10:06.870
So , many people are thinking about how to compress memory?

s103
00:10:06.870 --> 00:10:08.703
How to compress context?

s104
00:10:08.703 --> 00:10:11.498
Agent systems will compress context.

s105
00:10:11.498 --> 00:10:19.923
RAG, search, all these methods that we've been talking about for years are kind of compressing a large context into something that can be passed to a model.

s106
00:10:19.923 --> 00:10:21.942
You can write to files.

s107
00:10:21.942 --> 00:10:25.759
All of these things scale sublinearly with the amount of data you have.

s108
00:10:25.759 --> 00:10:28.262
But what if there was a middle ground?

s109
00:10:28.262 --> 00:10:37.174
What if there was a way to get a fairly significant compression of the data you remember, with almost complete preservation of the information?

s110
00:10:38.200 --> 00:10:42.453
So, we have many different ways to think about what to cache.

s111
00:10:42.453 --> 00:10:48.157
You know, recent compaction techniques have shown that we can replace the cache with a much shorter one.

s112
00:10:48.157 --> 00:10:59.210
We have work like attention matching and cartridges that have produced really promising results with high compression ratios, but both of them work at output time.

s113
00:10:59.210 --> 00:11:05.092
Again , one of the techniques, or one of the topics I want to talk about , is learning to infer.

s114
00:11:05.092 --> 00:11:15.514
So in this case, I want to introduce something called Still from the Base 10 research team, where synthesis on top of a cache, where we store a learned representation of

s115
00:11:15.514 --> 00:11:22.313
information, rather than the information itself or a deterministic subset of it, is amortized through learning.

s116
00:11:22.313 --> 00:11:30.354
So Charlie and Mudit from my post-workout training team recently gave a great presentation at Coster Compile.

s117
00:11:30.354 --> 00:11:31.482
It's on YouTube.

s118
00:11:31.482 --> 00:11:37.812
I would recommend you to watch it if you are interested in learning about KV compactification.

s119
00:11:37.812 --> 00:11:42.561
Unfortunately, I have neither the time nor the genius to explain everything here .

s120
00:11:42.561 --> 00:11:57.968
But the basic mechanism is that Still is a kind of bottleneck that takes a fixed set of learned query vectors, cross-matches them against the full KV cache, and produces a set of compact keys and values ​​ in a single direct

s121
00:11:57.968 --> 00:11:58.728
pass.

s122
00:11:58.728 --> 00:12:04.999
This creates a fast differentiated compressed memory that the LLM can pay attention to as if it were a real context.

s123
00:12:04.999 --> 00:12:10.503
So, if you are interested in KV compactification, be sure to check out Charlie and Mudit's work.

s124
00:12:10.503 --> 00:12:13.267
It was really great to learn.

s125
00:12:13.267 --> 00:12:15.755
So, those are two methods.

s126
00:12:15.755 --> 00:12:19.005
We talked about quantization, we talked about caching.

s127
00:12:19.005 --> 00:12:23.052
The latter is a speculative execution, and there have been many changes here.

s128
00:12:23.052 --> 00:12:34.841
So, to recap : speculative decoding—we take draft tokens, validate them during a direct pass, and use this to generate more than one token per pass.

s129
00:12:34.841 --> 00:12:43.817
This helps significantly with the number of tokens per second and is a lossless optimization , which is great because we don't have to worry about quality at all.

s130
00:12:43.817 --> 00:12:47.460
Now, regarding the history of speculation, we started with speculative decoding.

s131
00:12:47.460 --> 00:12:49.382
All of this is in the book.

s132
00:12:49.382 --> 00:12:54.380
You have SpecDec, which uses a small model from the same family to generate token drafts.

s133
00:12:54.380 --> 00:13:01.277
It turns out that small models are not very good at generating token drafts, they are just good small models.

s134
00:13:01.277 --> 00:13:12.872
So we as an industry invented a bunch of new methods, like Medusa, where you just add heads to the model, and then Eagle 3 came along, the idea of ​​ which was: what if instead of a tiny model, we train

s135
00:13:12.872 --> 00:13:19.011
a billion-parameter model on the hidden states of the target model to generate drafts?

s136
00:13:19.011 --> 00:13:22.604
And it worked really well.

s137
00:13:22.604 --> 00:13:28.987
So, as of February this year, Eagle 3 was the best method in speculative execution.

s138
00:13:30.219 --> 00:13:31.076
Now we have Dflash.

s139
00:13:31.076 --> 00:13:32.110
Dflash is even better.

s140
00:13:32.110 --> 00:13:35.639
This is diffusion for speculative execution.

s141
00:13:35.639 --> 00:13:39.826
Dflash creates a sequence of token drafts instead of a single token.

s142
00:13:39.826 --> 00:13:53.400
The model is a diffusion language model, meaning that it generates an entire sequence of tokens in the same way that video or image generation generates a sequence of frames or pixels, iteratively improving

s143
00:13:53.400 --> 00:13:57.087
them, instead of autoregressive token generation.

s144
00:13:57.087 --> 00:14:07.111
Dflash models may be two or four times slower, but they predict eight or 16 tokens simultaneously in this window, while Eagle only does one at a time.

s145
00:14:07.111 --> 00:14:13.138
Therefore, a single direct Dflash pass is faster than the entire Eagle draft generation phase and predicts more tokens.

s146
00:14:13.138 --> 00:14:23.841
These tokens can leverage cross-talk between themselves and typically create a higher level of adoption, because in speculative execution, adoption is everything.

s147
00:14:23.841 --> 00:14:29.727
So, in practice, we see more than a threefold improvement thanks to Dflash.

s148
00:14:29.727 --> 00:14:42.791
This is measured on a single B200 with 38B quantization, and we see that compared to Eagle this is a significant improvement in both token acceptance rate and number of tokens per second.

s149
00:14:43.800 --> 00:14:50.410
These Dflash models are trained with an attention mask for bidirectional draft generation.

s150
00:14:51.077 --> 00:15:02.976
So the target model will provide context, and within each block we will have a subset of pure tokens being selected, and the attention mask will provide causal consistency, but it will still allow for

s151
00:15:02.976 --> 00:15:10.572
bidirectional attention, whereas in a traditional autoregressive model you only look at tokens in one direction.

s152
00:15:10.572 --> 00:15:17.402
That's why we can take advantage of this diffusion-based architecture.

s153
00:15:17.402 --> 00:15:22.644
And so I thought I was done, but a few days ago Dspot appeared.

s154
00:15:22.644 --> 00:15:24.891
We have already launched Dflash into production.

s155
00:15:24.891 --> 00:15:27.108
And Dspot is a new study.

s156
00:15:27.108 --> 00:15:30.812
So all I can say about this is, “Hey, it exists.

s157
00:15:30.812 --> 00:15:31.705
This is cool.

s158
00:15:31.705 --> 00:15:33.070
We are studying it."

s159
00:15:33.070 --> 00:15:41.610
The difference compared to Dflash is that it still has the same diffusion model, but also combines it with a sequential model.

s160
00:15:41.610 --> 00:15:52.528
The idea is that we will improve the adoption rates by having these two models work together, instead of using just the iterative speculator, just the diffusion speculator, or just the autoregressive

s161
00:15:52.528 --> 00:15:53.212
speculator.

s162
00:15:53.212 --> 00:15:59.696
So, Dspot is very interesting, but we don't have any results from using it in production yet.

s163
00:15:59.696 --> 00:16:02.914
Hopefully we'll have them next time .

s164
00:16:02.914 --> 00:16:09.620
However, we have the results of the productive use of continuous retraining of the speculator.

s165
00:16:09.620 --> 00:16:19.810
We return to Dflash again; the idea is that speculative decoding depends very much on the actual content of the requests and responses you see on your system.

s166
00:16:19.810 --> 00:16:32.053
So, if you continuously retrain on these requests and responses in your production system, you can get an improvement in token acceptance rates from 20% to 2 times.

s167
00:16:32.053 --> 00:16:35.340
In fact, this is very difficult to implement.

s168
00:16:35.340 --> 00:16:41.498
This requires a lot of memory, and you should make sure you have permission to use the data you are processing in this way.

s169
00:16:41.498 --> 00:16:51.307
This requires huge computing power, you need to constantly move all this information around, and if you change the base model, you also have to change the speculator model.

s170
00:16:51.307 --> 00:16:59.637
But looking to the future, I believe that continuous speculation for very large systems will be a worthwhile optimization.

s171
00:17:01.170 --> 00:17:02.621
So what's next in inference?

s172
00:17:02.621 --> 00:17:13.252
What follows is my personal opinion, assumptions, and publicly available information; if I knew anything about what was actually being prepared for release, I wouldn't be able to talk about it.

s173
00:17:13.252 --> 00:17:17.012
So, this is just what I think will happen.

s174
00:17:17.012 --> 00:17:22.752
You know, I went through three hardware cycles: the Ampere, Hopper, and Blackwell releases.

s175
00:17:22.752 --> 00:17:33.375
It always takes time from the moment chips are shipped to their installation in data centers for the entire software stack to be able to fully utilize their capabilities.

s176
00:17:33.375 --> 00:17:43.724
But there's one thing that particularly excites me: it looks like the performance of NVFP4 on a Rubin system will be fantastic.

s177
00:17:43.724 --> 00:17:57.712
So the more we can borrow methods from local inference and confidently run models in the NVFP4 data format, the better we can leverage the amazing performance of future Rubin systems.

s178
00:17:57.712 --> 00:18:03.291
I believe that disaggregation and system-wide communication will become increasingly important.

s179
00:18:03.291 --> 00:18:14.619
We are seeing great early results from PD disaggregation, and the ability to move information like KV cache data across the system will become even more important.

s180
00:18:14.619 --> 00:18:23.862
As I said, the topic of training for breeding will continue to have a significant impact on the development of the industry.

s181
00:18:24.690 --> 00:18:27.446
So thank you very much to everyone who came to this lecture.

s182
00:18:27.446 --> 00:18:28.758
I am on Twitter.

s183
00:18:28.758 --> 00:18:29.694
I am on LinkedIn.

s184
00:18:29.694 --> 00:18:32.185
And I also give away free books.

s185
00:18:32.185 --> 00:18:38.973
You can download the PDF using the QR code or come with me to the Base 10 booth to get your copy of "Influence Engineering."

s186
00:18:38.973 --> 00:18:40.803
We have a lot of them there.

s187
00:18:40.803 --> 00:18:42.177
Maybe there will be enough for everyone.

s188
00:18:42.177 --> 00:18:45.115
If not, we'll ask Koyo to bring some more from the office.

s189
00:18:45.115 --> 00:18:47.982
So yes, I'll be down by the Base 10 booth.

s190
00:18:47.982 --> 00:18:52.064
Thank you very much again everyone and have a great day.