WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=J4_jCrTxMkk NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/J4_jCrTxMkk.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.400 --> 00:00:12.719 Okay. s3 00:00:12.719 --> 00:00:13.840 Hello everybody. s4 00:00:13.840 --> 00:00:21.520 Welcome to uh compression at the edge, the panel that we'll be conducting for the next bit here. s5 00:00:21.520 --> 00:00:23.359 Uh very nice to meet you all. s6 00:00:23.359 --> 00:00:25.600 I'll be your trusty moderator today. s7 00:00:25.600 --> 00:00:26.720 My name is Chris Alex. s8 00:00:26.720 --> 00:00:28.640 I'm a product research engineer at Nvidia. s9 00:00:28.640 --> 00:00:29.679 I work on Neotron. s10 00:00:29.679 --> 00:00:30.240 Let's go. s11 00:00:30.240 --> 00:00:34.000 Okay, we are joined by Daniel. s12 00:00:34.000 --> 00:00:34.960 Yes, hello everyone. s13 00:00:34.960 --> 00:00:36.160 I'm from Enslaf. s14 00:00:36.160 --> 00:00:38.719 Um, yeah, thanks for coming everyone. s15 00:00:38.719 --> 00:00:39.280 Excellent. s16 00:00:39.280 --> 00:00:39.760 And s17 00:00:39.760 --> 00:00:40.000 hello. s18 00:00:40.000 --> 00:00:44.800 Hi, I'm build NVIDIA model optimizer and we conduct a lot of models. s19 00:00:44.800 --> 00:00:45.760 Let's go. s20 00:00:45.760 --> 00:00:46.559 I'm Marv. s21 00:00:46.559 --> 00:00:49.920 I work as a machine learning engineer at Hugging Face. s22 00:00:49.920 --> 00:00:50.320 Awesome. s23 00:00:50.320 --> 00:00:51.120 I'm Parth. s24 00:00:51.120 --> 00:00:53.360 I work at Volama. s25 00:00:53.360 --> 00:00:57.520 So, compression uh a big topic. s26 00:00:57.520 --> 00:01:03.440 We're gonna we're going to set some some context hopefully uh in order to to kind of launch into this. s27 00:01:03.440 --> 00:01:18.880 So m maybe just in each of your own words if you want to kind of define what you think about compression, let us know like how you engage with uh with technology that ultimately is designed to make models that are bigger be a little bit smaller, right? s28 00:01:18.880 --> 00:01:21.360 Like that's the that's the general idea. s29 00:01:21.360 --> 00:01:23.600 So uh maybe we'll just go in reverse order. s30 00:01:23.600 --> 00:01:26.640 Par if you want to kick us off and what is compression to you? s31 00:01:26.640 --> 00:01:39.759 Yeah, I think honestly with Olama and for those of you who are not familiar, one of the easiest ways to run local models um and for us what honestly like rose to popularity was s32 00:01:39.759 --> 00:01:47.119 being able to run a larger model on a relatively small machine through quantization, which I'm sure we'll talk a lot about today. s33 00:01:47.119 --> 00:01:55.600 Um, and to me, compression is so so important because it actually makes these giant models viable for most people. s34 00:01:55.600 --> 00:02:01.040 I think for me it's just um shrinking something without losing information. s35 00:02:01.040 --> 00:02:02.960 But there's absolutely zero free lunch. s36 00:02:02.960 --> 00:02:12.879 So at the end of the day, you still spend on something whether it's latency or uh quality uh at the short um and I kind of agree. s37 00:02:12.879 --> 00:02:22.000 I feel like compression is much more than that definition because it democratizes the models for everyone at edge devices at your computer. s38 00:02:22.000 --> 00:02:33.120 I'm sure you are all running some uh Gemma 4, Kuwant at the moment or QN 3.6 those are the hot ones these days and like it's it just works so well. s39 00:02:33.120 --> 00:02:38.800 Um so yeah uh this is my definition at least it democratizes things. s40 00:02:38.800 --> 00:02:39.440 Cool. s41 00:02:39.440 --> 00:02:43.440 So the way I think about is same cost more intelligence. s42 00:02:43.440 --> 00:02:55.360 So compression accelerates and enables to give like u like a quick example originally we started with training in FP32 right and now we are talking about FP4 so that is 8x s43 00:02:55.360 --> 00:03:03.840 uh more compression and same almost same intelligence without not much degradation yeah same cost more intelligence s44 00:03:03.840 --> 00:03:13.519 yeah like how we see quantization is like you know you take like a big model like GLM 5.2 to it's 1.5 tab which is definitely ginormous. s45 00:03:13.519 --> 00:03:18.400 Um but then the trick is you can actually quantize it and shrink it to 250GB. s46 00:03:18.400 --> 00:03:20.959 Um so you can make it 86% smaller. s47 00:03:20.959 --> 00:03:24.560 Um but with tricks of quantization it will not become 86%. s48 00:03:24.560 --> 00:03:31.120 So it's not 86% dumber right if you compress it by 86% it doesn't become like you know terrible useless. s49 00:03:31.120 --> 00:03:45.200 Um and so what we show with dynamic quantization if you quantize some layers to you know higher precision and you leave most of the layers in like one bit or two bit and some you know very important layers is 16 bit you can still recover 76% s50 00:03:45.200 --> 00:03:46.239 of all accuracy. s51 00:03:46.239 --> 00:03:53.200 Um so you so the trick for quantization is if you quantize the correct layers you will not make the model you know literally useless. s52 00:03:53.200 --> 00:03:57.519 Um and compression I guess is very important for you to run on your local computers. s53 00:03:57.519 --> 00:04:00.720 Um you know make democratization of AI. s54 00:04:00.720 --> 00:04:01.360 Yeah. s55 00:04:01.360 --> 00:04:06.400 So like, okay, we we we've been talking today about like there this inflection point, right? s56 00:04:06.400 --> 00:04:18.799 Recent moments that like uh kind of led to this uh this more this resurgence of the importance of local AI and and uh you know open models, right? s57 00:04:18.799 --> 00:04:20.720 Own your own intelligence. s58 00:04:20.720 --> 00:04:22.720 You guys were like ahead of the curve though, right? s59 00:04:22.720 --> 00:04:27.919 Like I mean you guys were thinking about this before it was it was cool to think about it, right? s60 00:04:27.919 --> 00:04:40.160 So, I'd love to hear from each of you like when was the moment that you knew that like quantization or compression like is the path forward for it and and we're going to talk a lot today about like consumer s61 00:04:40.160 --> 00:04:50.960 right uh hardware that means like your RTX cards your your your Macs things like that right but uh it goes well beyond that what was the moment you kind of first like s62 00:04:50.960 --> 00:04:58.400 got the quantization or compression bug you know that made you say like oh this is going to be big man and we'll just go way back down the line. s63 00:04:58.400 --> 00:04:59.759 Daniel, take us away. s64 00:04:59.759 --> 00:05:00.080 Yeah. s65 00:05:00.080 --> 00:05:03.840 Um, so I think like the biggest moment was Deep Seek R1. s66 00:05:03.840 --> 00:05:11.039 Definitely when it got released, it was quite dramatic for the world because we finally have some sort of like reasoning model that worked very well. s67 00:05:11.039 --> 00:05:12.560 Um, and it was open source. s68 00:05:12.560 --> 00:05:16.080 Um, and the biggest problem though it was very big. s69 00:05:16.080 --> 00:05:19.919 Um, and you know, running it locally was extremely complex. s70 00:05:19.919 --> 00:05:25.120 Um, and so you know we when we started off like you know we decided okay let's do some tricks. s71 00:05:25.120 --> 00:05:28.320 let's quantize some layers to higher precision. s72 00:05:28.320 --> 00:05:30.479 Um, and it randomly worked. s73 00:05:30.479 --> 00:05:35.680 Um, and so like we were quite surprised like you know 1.58 bit quant worked well. s74 00:05:35.680 --> 00:05:39.199 Um, and we were yeah we just posted about this and we were quite surprised. s75 00:05:39.199 --> 00:05:41.440 Okay, look model seem to be working well. s76 00:05:41.440 --> 00:05:50.240 Um, and then obviously over time we had you know Quen 3.6 3.5 GMA 4 you know every single time there's a new even Nvidia's you know open source models Neimatron s77 00:05:50.240 --> 00:05:52.240 they just keep getting better and better and better. s78 00:05:52.240 --> 00:05:55.759 Um the only problem sometimes is they get bigger and bigger and bigger. s79 00:05:55.759 --> 00:06:04.240 So that's the only problem and so you need to focus more on like you know how to make the model smaller and smaller and smaller and so like you know recently with GLM 5.2 who you know was quite s80 00:06:04.240 --> 00:06:14.800 you know it's a very good local model and although it's ginormous and so making it work very well in the local device is complicated um but I think like you know I guess it's a long history s81 00:06:14.800 --> 00:06:23.919 of these models um but we're expecting more you know whatever the next model is the next jama the next any model um we're very excited for the future s82 00:06:23.919 --> 00:06:37.919 yeah it's like a ongoing arms race between how big can we make the models versus how small can we make them and they still work right uh uh maybe some thoughts for for from you like what when did you think like ah man quantization s83 00:06:37.919 --> 00:06:39.280 is is it s84 00:06:39.280 --> 00:07:00.240 so I originally started working on pruning uh so okay so that was maybe 3 years back that was when CV models were all the rage then sadity happened um yeah so pruning typically you typically lose uh quality so you need to fine-tune it a little bit and then we lls came out and quantization was a easy compression s85 00:07:00.240 --> 00:07:10.160 thing you could do you don't uh lose lot of accuracy the model it so what I learned was some kind of compressions are really good you can architect techniques s86 00:07:10.160 --> 00:07:31.520 so that u you can chop it down a lot without u losing quality so quantization is really good sparity is really good but not as much as quantization so I did not have like a exact moment of like you know where I got this numeric bug it uh slowly grew on me and uh so for example NVFP4 s87 00:07:31.520 --> 00:07:44.319 right I was fortunate to work on NVFP4 experiment math when I joined Nvidia and the NVFP4 is very genius uh it we have been doing like a lot of research and experiments to s88 00:07:44.319 --> 00:07:49.840 uh make NVFP4 better but it is like a really solid uh numeric format. s89 00:07:49.840 --> 00:07:51.599 So yeah, so that was Yeah. s90 00:07:51.599 --> 00:08:02.960 So basically like you know uh some compressions are really good and you can cleverly architect uh understanding things better and uh not lose quality like like you know Dan said uh mix precision s91 00:08:02.960 --> 00:08:08.000 uh quantization you can compress it 10 chop it at 10 without losing much quality. s92 00:08:08.000 --> 00:08:08.240 Yeah. s93 00:08:08.240 --> 00:08:11.199 So that was a slowly growth for me. s94 00:08:11.199 --> 00:08:12.080 Should I? s95 00:08:12.080 --> 00:08:13.120 Yes please. s96 00:08:13.120 --> 00:08:21.599 So, um I think for me the biggest wow moment of quantization was back in the day there was something called Q Laura s97 00:08:21.599 --> 00:08:35.039 and and the fact that we could actually fine-tune stuff on a toaster essentially that is the collab tier T4 for me that was like a big wow although it's super slow s98 00:08:35.039 --> 00:08:49.200 it's it's okay um but on top of it um for instance like we have libraries called TRL and bits and bites and have that enables all of this and like you can even do this with the very advanced techniques like GRPO s99 00:08:49.200 --> 00:09:00.959 for instance actually you can train stuff on the um I mean you need to collocate on stuff but like with very small amounts of VRAM and then recently we s100 00:09:00.959 --> 00:09:21.600 uh in case you missed it we have acquired Lama CPPA kind of AUI hired llama CPP and then I kind of pivoted to the llama CPP world and I was like wow Because like the second wow moment for me was the fact that I could run open claw and hermes agent on like very long horizon tasks with s101 00:09:21.600 --> 00:09:27.519 Q1 3.6 quants which wouldn't be able to which wouldn't be possible before. s102 00:09:27.519 --> 00:09:36.800 Um I just know uh I tried many models actually with that and then Q1 3.5 was like wow it can do a lot of coding it can fix its own s103 00:09:36.800 --> 00:09:38.160 harness and stuff. s104 00:09:38.160 --> 00:09:42.800 So yeah, that was the two big moments for me actually. s105 00:09:42.800 --> 00:09:43.279 Yeah. s106 00:09:43.279 --> 00:09:47.680 Um I think I'll speak to it more from like a a consumer point of view. s107 00:09:47.680 --> 00:09:50.480 Um I this is back in like 2023. s108 00:09:50.480 --> 00:09:52.800 Uh I was kind of hacking around my own stuff. s109 00:09:52.800 --> 00:09:54.720 This is prior to me being at a Lama. s110 00:09:54.720 --> 00:09:59.839 Um but one of the coolest things was I wanted to run kind of my AI on my computer. s111 00:09:59.839 --> 00:10:02.880 You know, I was still in school at the time, broke, did not have a lot of money. s112 00:10:02.880 --> 00:10:05.200 And so I was like, "Okay, I need to run this for free somehow. s113 00:10:05.200 --> 00:10:07.360 So what's like the best option?" s114 00:10:07.360 --> 00:10:16.720 Um, and so I came across a Llama at the time and I ran the model kind of locally at on my computer and I think it was Llama 3 at the time as well s115 00:10:16.720 --> 00:10:18.160 very long time ago. s116 00:10:18.160 --> 00:10:22.880 Um, and honestly for me that's kind of how I got into the whole world of local models. s117 00:10:22.880 --> 00:10:25.600 It's like okay this is actually workable. s118 00:10:25.600 --> 00:10:27.040 I can like make it do things. s119 00:10:27.040 --> 00:10:32.640 It's able to output something and you know back then it wasn't as good as it is you know outputting now. s120 00:10:32.640 --> 00:10:40.640 As Murvey said, Quentin 3.6 is insane and Gemma 4 and all there's all these new models which can have so much more capability baked into them. s121 00:10:40.640 --> 00:10:46.480 But for me, it was honestly the idea of like being able to even just run something locally. s122 00:10:46.480 --> 00:10:54.640 Um, and it started with like Llama 3 with Lama and being able to actually have it run and like complete tasks. s123 00:10:54.640 --> 00:10:58.240 Um, and prior to that, you know, I'd done built a lot of models before. s124 00:10:58.240 --> 00:11:02.720 Um, and I knew like how much work it goes like is that goes into building them. s125 00:11:02.720 --> 00:11:04.480 Um, and running them has always been hard. s126 00:11:04.480 --> 00:11:09.200 So to me, quantization like really makes that happen for a lot of people. s127 00:11:09.200 --> 00:11:09.440 Yeah. s128 00:11:09.440 --> 00:11:11.680 I mean, I think everyone's experience is the same, right? s129 00:11:11.680 --> 00:11:23.920 like uh probably most of the people in this room the first time that you run one of these models that like you know only existed behind an API or or a hosted environment on on some cloud GPU somewhere s130 00:11:23.920 --> 00:11:34.480 uh on your on your computer or on your you know your your your gaming laptop or your your gaming whatever it is I mean it's just uh I think from that moment on you're like s131 00:11:34.480 --> 00:11:47.440 making these things small is pretty cool there there is no a something that we have to discuss right and and and and and anyone can jump in to to follow up to this, but I'll pose it first to you, Daniel. s132 00:11:47.440 --> 00:11:51.040 Uh you said like models are 86% smaller, right? s133 00:11:51.040 --> 00:11:53.839 But like they're not 86% dumber. s134 00:11:53.839 --> 00:11:56.560 Uh how that sounds absurd, right? s135 00:11:56.560 --> 00:12:08.639 Like uh and and beyond how like how do you actually like verify or think about verifying that like this model is uh you know gone through the process of some some form of compression, right? s136 00:12:08.639 --> 00:12:09.680 Whatever it happens to be. s137 00:12:09.680 --> 00:12:13.680 How do I now like determine that that model is is not garbage, right? s138 00:12:13.680 --> 00:12:17.600 Like that that that we haven't chopped out 86% of its brain. s139 00:12:17.600 --> 00:12:18.720 Yeah, that's a great question. s140 00:12:18.720 --> 00:12:35.120 Um, so I think generally speaking, if you compress a model down by 86%, you would assume, you know, if you randomly select parts of the model to compress like delete or something like that or set set them to be like if you round it to like the closest number, you would most likely it will be not 86% s141 00:12:35.120 --> 00:12:35.519 dumber. s142 00:12:35.519 --> 00:12:36.959 It will be 100% dumber. s143 00:12:36.959 --> 00:12:39.200 So if you do that methodology that will not work. s144 00:12:39.200 --> 00:12:44.320 Um and so the main trick of language models is you should leverage the architecture of the language model itself. s145 00:12:44.320 --> 00:12:51.920 So language models generally have like you know 36 layers you know 50 layers many many layers each of the layers have different importance. s146 00:12:51.920 --> 00:12:59.360 Um and so like you know for example the most the first layer is actually very important and then the last layer is also very important but then the middle layers are kind of useless. s147 00:12:59.360 --> 00:13:07.839 Um and so the main reason why they're not that useful um is because when you train a language model um with like you know let's say one trillion parameters s148 00:13:07.839 --> 00:13:17.279 um you have to use many many tokens right so like a language model can be trained with like 30 trillion tokens um but we're still not there yet in terms of saturating all of the weights s149 00:13:17.279 --> 00:13:28.000 um so once you okay maybe in the future once we train to 300 trillion tokens okay maybe you can't do compression anymore okay maybe that's another topic but at the current stage the trick is s150 00:13:28.000 --> 00:13:42.480 um 86% of the weights do not need to be there in the model um because of the training algorithm because of you know back propagation some of the weights are very close to zero and you can literally just set them to zero um and so that's one of the tricks um and also you have to do you s151 00:13:42.480 --> 00:13:55.040 know layer by layer analysis you know if you quantize layer 1 what will happen to accuracy if you quantize layer two what will happen to accuracy and so on so on so on um and so you can also think of this as like a combinatorial s152 00:13:55.040 --> 00:14:03.360 optimization problem um you don't just do okay layer one and then do layer 2, you also have to do like you know 32 choose two layers or choose three layers. s153 00:14:03.360 --> 00:14:05.120 So it becomes very complicated. s154 00:14:05.120 --> 00:14:17.360 Um and so like you know it's a very then you get some combinatorial explosion problem you know it's not just layer by layer you know within the layer which number of this specific tensor is not quantizable or not. s155 00:14:17.360 --> 00:14:27.839 Um for example there is something called a um super weights there is a super weights paper which shows that if you quantize one number just one of the entire model your model becomes 20% dumber. s156 00:14:27.839 --> 00:14:32.560 Um, and so you need to find this specific one number and then you cannot quantize this. s157 00:14:32.560 --> 00:14:37.199 Um, so there's like very weird mechanisms of language models during training. s158 00:14:37.199 --> 00:14:42.000 Um, and yeah, there's a lot of whole research going into like quantizing models correctly. s159 00:14:42.000 --> 00:14:43.680 Um, yeah. s160 00:14:43.680 --> 00:14:46.000 Anyone else with thoughts to add here? s161 00:14:46.000 --> 00:14:47.120 Uh, yeah. s162 00:14:47.120 --> 00:14:47.839 So, okay. s163 00:14:47.839 --> 00:14:51.199 So, particularly your question was about how we evaluate. s164 00:14:51.199 --> 00:14:51.360 Yeah. s165 00:14:51.360 --> 00:14:52.480 Like how Yeah, exactly. s166 00:14:52.480 --> 00:14:57.440 Like how beyond just how do we make the model smaller like how do we know that it's not dumb? s167 00:14:57.440 --> 00:15:01.600 We just run all the benchmarks mostly the AA benchmarks. s168 00:15:01.600 --> 00:15:06.959 So uh model optimizer team we publish lot of check quantiz checkpoints on hugging face hub. s169 00:15:06.959 --> 00:15:09.920 You can check the Nvidia model optimizer space. s170 00:15:09.920 --> 00:15:17.120 Uh so we with FP4 we target for less than 1% accuracy degradation overall on all a benchmarks. s171 00:15:17.120 --> 00:15:17.519 Yeah. s172 00:15:17.519 --> 00:15:18.880 So uh yeah. s173 00:15:18.880 --> 00:15:19.199 Yeah. s174 00:15:19.199 --> 00:15:32.639 So that is um and and they're like you know we we uh we we try to use simple uh strategies because that way we can um push out uh models faster. s175 00:15:32.639 --> 00:15:45.519 Um on top of that um learning is that u the strong like like you know um the symbol and strong nicely designed number formats like FP4 preserve like lot of accuracy s176 00:15:45.519 --> 00:15:59.600 and then u we also see this um disproportionate sensitivity to some layers for example uh linear uh attention projection layers are very sensitive the the uh KV QKB layers are sensitive s177 00:15:59.600 --> 00:16:08.800 whereas we we by default keep them in uh FP4 while we put say like you know other uh layers in FP8 or BF16. s178 00:16:08.800 --> 00:16:12.560 We use this uh gradient based sensitivity analysis. s179 00:16:12.560 --> 00:16:15.120 It runs like a linear programming solver. s180 00:16:15.120 --> 00:16:29.360 But yeah, but my main um learning has been um that uh FP4 or like like you know really nicely designed uh number formats help a lot and we Yeah. s181 00:16:29.360 --> 00:16:34.160 And then a lot of benchmarking the boring stuff in which we spent a lot of time. s182 00:16:34.160 --> 00:16:34.800 Yeah. s183 00:16:34.800 --> 00:16:44.399 Maybe ju maybe just for everyone here who is like uh you know I mean I I think hopefully most of us understand like BF-16 and like you know uh s184 00:16:44.399 --> 00:16:47.279 FP8 like what what the hell is NVFP4? s185 00:16:47.279 --> 00:16:48.000 Oh cool. s186 00:16:48.000 --> 00:16:57.759 Okay so FP4 um the number is that it's a floating point 4bit number but the genius isn't it is a microlock scaled uh number. s187 00:16:57.759 --> 00:17:03.759 So microlock scaling means uh every u you choose a group. s188 00:17:03.759 --> 00:17:13.760 In in the case of FP4, you choose 16 elements and you can share one scaling fac one one extra um FP8 number between these 16 elements. s189 00:17:13.760 --> 00:17:29.840 So this was originally u invented by uh uh bits and bites Tim that yeah so uh Nvidia adopted like a similar but different design of uh four bits but every uh 16 bits share one uh 8 bit value s190 00:17:29.840 --> 00:17:42.480 uh to scale them and yeah that has been like that has been tremendous it uh gives significant improvements over like other formats yeah of 4bit. s191 00:17:42.480 --> 00:17:43.200 Yeah. s192 00:17:43.200 --> 00:17:43.440 Yeah. s193 00:17:43.440 --> 00:17:50.080 So it's it's not as simple as like uh we just make the numbers smaller, you know, it's a lot more going into it than that. s194 00:17:50.080 --> 00:17:51.600 I'd be interested to hear from you guys. s195 00:17:51.600 --> 00:18:01.760 Uh like I think a lot of the time when we talk about compression or quantization or any technique that makes the model more accessible, right? s196 00:18:01.760 --> 00:18:07.679 We're talking about it through the lens of like uh so that I can run it on my on my toaster like you said, right? s197 00:18:07.679 --> 00:18:18.720 Uh but like is there any value to compression or these techniques that exist for like a business that presumably has access to I would hope more more than a toaster s198 00:18:18.720 --> 00:18:20.799 part maybe you want to s199 00:18:20.799 --> 00:18:34.320 yeah I think for sure like you have a variance in kind of just you know hardware that a employee has versus hardware which uh scales up and you know they're running their own clusters. s200 00:18:34.320 --> 00:18:45.440 Um, and the way that at least we look at it from is you're you should be able to run whatever model you want locally on kind of that individual's computer as well. s201 00:18:45.440 --> 00:18:50.559 Um, and you know with compression in particular, you run through a lot of different challenges. s202 00:18:50.559 --> 00:18:53.039 So accuracy is for sure one of them. s203 00:18:53.039 --> 00:18:57.280 Um, but you know actually making use of it through a harness. s204 00:18:57.280 --> 00:19:01.039 Um, seeing you know what the end result is when you actually try it out. s205 00:19:01.039 --> 00:19:17.919 you know there's so many things that I feel can't be captured by a uh model optimizer or after quantizing it or you know certain benchmarks and it's literally me you know running through putting it in cloud code or something and running the model it's like no it doesn't feel just right s206 00:19:17.919 --> 00:19:30.960 um so I think you know the benchmarks are a great indicator of pointing in the right direction um but having the model behavior kind of stay in line with that I think is still kind of being worked done. s207 00:19:30.960 --> 00:19:38.240 And when it comes to businesses, you want to be able to give them the option to not just, you know, deploy their own models on like their own infrastructure, s208 00:19:38.240 --> 00:19:44.080 but also empower them to be able to run it on, you know, individual machines. s209 00:19:44.080 --> 00:19:51.440 I think benchmarking often times only work for the verifiable tasks rather than the, you know, the Vive itself. s210 00:19:51.440 --> 00:20:01.200 And like I now that I think about it actually it could have been cool to have like a quant arena or something but unfortunately like there was a paper last year by s211 00:20:01.200 --> 00:20:07.520 um singal I think from cohhere uh that showed that arenas are very much hacked. s212 00:20:07.520 --> 00:20:23.840 Uh so that that's also not the way but anyway I feel like most of the jobs still do not require like a fable level thing like we are kind of in a bubble as a software developers right so we are like wow fable s213 00:20:23.840 --> 00:20:37.120 but at the same time really most of the jobs do not require that and on top of it like for the businesses of some of the things that we are saying I feel like because I'm very open source pled it's like super obvious s214 00:20:37.120 --> 00:20:43.840 But at the same time many people don't know about it like you can just serve a lot of like you can have much more concurrency. s215 00:20:43.840 --> 00:20:47.600 Uh at the same time you can do like in-house deployment and stuff. s216 00:20:47.600 --> 00:20:56.480 So like um increasing concurrency cuts the compute and then yeah like most of the time people just don't need that. s217 00:20:56.480 --> 00:21:13.919 But like for instance like I've been hearing a lot from hugging face users and stuff and like it's not only the quantization but I know companies that actually distill midsize models to like very small ones for like reranking or whatever that doesn't require like LLM outputs s218 00:21:13.919 --> 00:21:16.559 uh cutting like millions of costs. s219 00:21:16.559 --> 00:21:30.400 So like it's not only quantization but like there is so much more that is out there for compression in my opinion and it just creates a ton of value for businesses and people aren't aware of it like because people in this room are like s220 00:21:30.400 --> 00:21:41.280 you know you are interested you you read about it you just assume that people know a lot about it but actually they don't and it's kind of shocking to me but yeah s221 00:21:41.600 --> 00:21:48.640 uh I another question that comes up all the time right we're talking about model compression uh you were talking about like GLM52, right? s222 00:21:48.640 --> 00:21:51.120 And let's let's shrink it down as much as we can. s223 00:21:51.120 --> 00:22:07.120 Uh back there on the the station we're running uh you know a reap quant of uh well compress of uh of GLM but like why would I do that when like uh Neotron 3 Nano exists or Gemma Small exists or Quentyn Tiny you know exists? s224 00:22:07.120 --> 00:22:19.280 like why why should I care about compression when a lot of these model shops are kind of putting out like uh small enough models that you can run them in kind of their native precision or very close to their native precision. s225 00:22:19.280 --> 00:22:32.559 So there is a paper showing that if you want to do compression um it's actually most likely better use of resources if you train a ginormous model then you quantize it down um and so there is this like formula where they show s226 00:22:32.559 --> 00:22:51.600 um comparing for example a very small model like a 35 billion um you know 35 billion Bflow 16 so 16 bit versus say like um four times bigger like 120 billion at 4bit um and which one's better right so essentially they're the same size in terms of disk case, but which one intelligence- wise is better? s227 00:22:51.600 --> 00:22:59.120 And from those experiments, they show that the bigger model quantized to 4bit is actually much better um than a 35 billion 16 bit. s228 00:22:59.120 --> 00:23:04.159 So in general, most likely what will happen is we'll get bigger and bigger and bigger and bigger models. s229 00:23:04.159 --> 00:23:10.960 Um and you know like okay currently now you know GLM you know 1.5 terabytes you know oh okay it's not that big. s230 00:23:10.960 --> 00:23:28.320 Um but what happens if it's 15 terabytes then okay we must do quantization we must do compression this will not fit in anyone's not not even enterprises can now service and so like local models you know you must do compression if they're getting bigger and bigger and bigger um and you know to extract any value out of s231 00:23:28.320 --> 00:23:40.080 it and you know okay I guess the DGX station has a lot of memory I guess that's very useful um but you know once we have 10 trillion parameter models that will also not fit and so like you know we need to do compression s232 00:23:40.080 --> 00:23:42.159 you I guess quantization for that. s233 00:23:42.159 --> 00:23:44.480 Um but in general the small ones are very useful. s234 00:23:44.480 --> 00:23:48.720 Um but I think the bigger ones compressed down um is slightly more useful. s235 00:23:48.720 --> 00:23:51.200 Um but there's also another trick you can do model routing. s236 00:23:51.200 --> 00:23:57.679 For example, for the small ones you can do it for the for example you can use the big ones for planning and then you execution with the small ones. s237 00:23:57.679 --> 00:24:01.039 Um but the small ones are still better because they're much faster. s238 00:24:01.039 --> 00:24:09.200 So even if you have a big one and you compress it down you'll probably get like five tokens to 10 tokens per second if you don't have enough GPU power. s239 00:24:09.200 --> 00:24:12.240 Um and for the small ones you can get 200 tokens per second. s240 00:24:12.240 --> 00:24:19.679 Um so I guess like depending on your use case um you also have to consider speed throughput you know what is your GPU that you have and stuff like that. s241 00:24:19.679 --> 00:24:34.480 Um yeah I guess I mean very cool very cool obviously uh so par with Olama I I I imagine mo most of the people here know Olama if if you're if you're not using it give it a try. s242 00:24:34.480 --> 00:24:52.159 uh you know this is one of those topics where you is Olama useful in a world where we don't have this kind of compression right like how how much uh is the fact that we can compress intelligence to kind of run on like my Mac or my s243 00:24:52.159 --> 00:25:02.400 uh you know my my my home workstation right like how much is that uh ecosystem important to to you know people or businesses or organizations like Ola s244 00:25:02.400 --> 00:25:19.120 yeah I think uh It's pivotal obviously for in in order to have like these models be so viable running on your personal computers and being able to actually make work of them as Murva was saying you know qu 36 just running on computer running open claw being able to s245 00:25:19.120 --> 00:25:30.320 fix its own harness that's really possible through some level of compression now there are you know other techniques which kind of some of the model labs will do especially while releasing s246 00:25:30.320 --> 00:25:39.440 smaller size models um things like Q8 at or you know with GPOSS it was MXFP4 which is a different format. s247 00:25:39.440 --> 00:25:52.080 So I do think as long as there's a demand for being able to run kind of personal language models even if compression didn't exist there would be analogous techniques or you know maybe model training would look a little bit different. s248 00:25:52.080 --> 00:25:54.960 Um obviously that comes everything comes at a trade-off. s249 00:25:54.960 --> 00:26:06.080 Um, as Daniel was mentioning, you know, you can have a really large model being condensed down into something smaller so you can run it versus even just having um a full precision model but smaller in parameter size. s250 00:26:06.080 --> 00:26:08.559 And they both come with different trade-offs. s251 00:26:08.559 --> 00:26:19.679 it matter of fact like how how do you see the open source community help drive this right like I think quantiz I mean we all remember Tim Demer's we all remember s252 00:26:19.679 --> 00:26:32.799 Qura you know is like that that that image right where there's a big stack of things that would fall over if it weren't for bits and bittes holding it all up like like how have you seen the open source community flourish or grow around s253 00:26:32.799 --> 00:26:36.559 uh not just quantization but compression more generally s254 00:26:36.559 --> 00:26:39.200 I wanted to add something to what Daniel says. s255 00:26:39.200 --> 00:26:46.799 By the way, if you don't know about their work, they have like immense pipelines to actually do the quants and then verify them and stuff. s256 00:26:46.799 --> 00:26:53.840 So like I highly recommend you to check out onslaught which is like in my opinion the best quants you will find over there. s257 00:26:53.840 --> 00:26:54.240 Yes. s258 00:26:54.240 --> 00:26:55.039 Oh, thank you. s259 00:26:55.039 --> 00:27:06.080 Um I wanted to say for instance like to add to your point like two years ago I think two years ago we trained like small VLM where there were no small vision language models. s260 00:27:06.080 --> 00:27:11.760 there was lava and then there was a jump to big models and then nobody was training like smaller ones. s261 00:27:11.760 --> 00:27:27.120 Um, we figured out that like um training like we trained like 1 B 500M and 260 uh 56M and like 256M was not good and like 500M was actually s262 00:27:27.120 --> 00:27:42.320 somewhat good and it was actually able to run on your iPhone which was like shocking to me and um I noticed that over time it feels like u people have been releasing like smaller models which is great. s263 00:27:42.320 --> 00:27:48.799 But like if you have like advanced models and then everybody is just releasing midsize models, you can quantize them. s264 00:27:48.799 --> 00:27:52.880 And I feel like that's the biggest value that quantization somewhat offers. s265 00:27:52.880 --> 00:28:01.840 And for instance, like to coming to your question about the community and stuff, for instance, this year what happened was that for the Q1 3.6 release, Q1 team asked the s266 00:28:01.840 --> 00:28:06.799 people, okay, we are going to release only one checkpoint, which is kind of heartbreaking. s267 00:28:06.799 --> 00:28:09.440 And everybody said the midsized ones. s268 00:28:09.440 --> 00:28:20.320 And this kind of goes to show the value of like how quantization is adopted and everybody's uh using those checkpoints and then quantizing them instead of asking for smaller models. s269 00:28:20.320 --> 00:28:33.039 I feel like you get more and more and more intelligence and then over time you shrink them and then suddenly it the the the bigger models make the bigger shifts in terms of like s270 00:28:33.039 --> 00:28:44.640 um how much you can quantize them and still you do not lose the information and the quants become smaller than the bigger models of yesterday which to me is like phenomenal. s271 00:28:44.640 --> 00:28:47.909 Um as for the community can you rephrase your s272 00:28:47.909 --> 00:28:47.919 [laughter] s273 00:28:47.919 --> 00:28:49.360 question again? s274 00:28:49.360 --> 00:28:51.679 I think you answered it actually. s275 00:28:51.679 --> 00:28:52.480 It's a great job. s276 00:28:52.480 --> 00:28:53.760 You Yeah, s277 00:28:53.760 --> 00:28:54.559 thank you. s278 00:28:54.559 --> 00:28:56.559 I mean, you know, s279 00:28:56.559 --> 00:29:00.799 I I got to ask this question since you're you're on the you're up here with us. s280 00:29:00.799 --> 00:29:06.399 So, we uh Nvidia Neotron releases NVFP4 with every uh release. s281 00:29:06.399 --> 00:29:11.679 NVFP4 is uh as described basically a numeric format, right? s282 00:29:11.679 --> 00:29:16.720 Uh but the idea is that it's like a smaller version of the model and it retains a lot of accuracy, right? s283 00:29:16.720 --> 00:29:20.240 uh which is the whole idea of of compression and quantization. s284 00:29:20.240 --> 00:29:23.919 Uh I just like to hear like how hard is it to do that right? s285 00:29:23.919 --> 00:29:29.919 Like so we kind of heard Daniel talk about it from this dynamic uh quantization strategy, right? s286 00:29:29.919 --> 00:29:39.120 Uh it's already sounds very difficult like how difficult as an engineering challenge is it to like make the model small without it shedding the bed. s287 00:29:39.120 --> 00:29:39.919 Uh I see. s288 00:29:39.919 --> 00:29:40.320 Okay. s289 00:29:40.320 --> 00:29:53.120 So if you're doing post- training quantization which is uh you take the model that uh the uh like the release model the BF16 model and do post- training quantization on it. s290 00:29:53.120 --> 00:29:56.559 It is fairly u like you know easy to do. s291 00:29:56.559 --> 00:30:06.720 Uh so we quantize to FP4 models like uh a uh large uh GLM or like you know those like trillion size models. s292 00:30:06.720 --> 00:30:22.399 uh you need uh like um a we have blackwell nodes so we put it on blackwell uh in in a couple of hours the condise checkpoint is ready but then our pain starts there because now we have to evaluate these models match the model card s293 00:30:22.399 --> 00:30:34.640 actually that is where we spend a lot of our time now coming to uh uh training based methods okay so for large models and mediumsiz models I would say like you know 20 s294 00:30:34.640 --> 00:30:39.120 billion parameters plus or 30 million parameter plus den size. s295 00:30:39.120 --> 00:30:42.960 Uh this PTQ usually works out of the box. s296 00:30:42.960 --> 00:31:00.080 Uh with uh some selective quantization like the heristics like Dan mentioned right like we use some heristics such as sparse MOS can be aggressively quandisized like yeah and we also have this auto quandis which uses this automatic sensitivity analysis and napsack solver. s297 00:31:00.080 --> 00:31:00.720 Yeah. s298 00:31:00.720 --> 00:31:00.960 Yeah. s299 00:31:00.960 --> 00:31:02.240 So all those right. s300 00:31:02.240 --> 00:31:08.320 So for medium and large models PDQ works out of the box very easy relatively super easy to do. s301 00:31:08.320 --> 00:31:21.760 Uh then if it is smaller model say like you know less than 20B size model we have to do some uh quantization aware distillation etc to recover accuracy that is yeah so training based methods are a little bit more painful. s302 00:31:21.760 --> 00:31:25.760 They're becoming more painful especially with this reasoning models. s303 00:31:25.760 --> 00:31:29.440 So you need to have the original data set and it's not just about the original data set. s304 00:31:29.440 --> 00:31:43.440 Internally we have the original neotend data set but even then it is a pain because these are multi-rain uh multi-stage trained RL models like for done with now models are being trained with uh multiple teacher distillation s305 00:31:43.440 --> 00:31:47.519 where each teacher is like an expert in coding or reasoning or something. s306 00:31:47.519 --> 00:31:58.080 So it becomes really difficult to uh get uh good uh training data to uh train the model with with QAD uh yeah to recover accuracy. s307 00:31:58.080 --> 00:32:06.799 If we if we train if we do QA with wrong data it uh it most commonly breaks the model rather than helping it. s308 00:32:06.799 --> 00:32:07.600 Yeah. s309 00:32:07.600 --> 00:32:07.919 Yeah. s310 00:32:07.919 --> 00:32:09.519 So again it sounds hard. s311 00:32:09.519 --> 00:32:13.120 Uh I mean it's still an engineering challenge. s312 00:32:13.120 --> 00:32:14.399 No I want to stay corrected. s313 00:32:14.399 --> 00:32:25.120 I want uh to encourage people to like you know use quandies model mostly PTQ will uh work like you know if you are like looking at like 30B yeah it should work out of the box and s314 00:32:25.120 --> 00:32:36.559 there are so many tools right now like model op like unsloth has tools uh hug huggy face in the ecosystem tons of tools right to to make these big these big models small s315 00:32:36.559 --> 00:32:54.720 you you know something that I want to pick your guys' brains about is architecture for models in like the llama era of let's call it open AI right which is like uh every model was like the same you know the architecture was basically the same the kinds of things that you saw in the guts of the model s316 00:32:54.720 --> 00:33:05.840 were the same and now we're entering like a very cursed era of technology right where uh you know everyone's doing the architecture just a little bit differently we're using this hybrid attention s317 00:33:05.840 --> 00:33:16.000 they're using this uh you know linear attention uh you know you know variant uh you know we're we're exploring a lot and we're trying a bunch of new things. s318 00:33:16.000 --> 00:33:28.000 How has that like shifted the difficulty of quantization now that it's not just like one problem repeated with different uh you know you know numbers of layers like is that something that is s319 00:33:28.000 --> 00:33:35.919 makes it difficult to keep up with maybe maybe par you know from from Olaf's perspective like how hard is it to keep up with this? s320 00:33:35.919 --> 00:33:36.399 Yeah. s321 00:33:36.399 --> 00:33:39.279 Um there's kind of two facets to it I would say. s322 00:33:39.279 --> 00:33:42.240 Uh the first is actually just the model implementation itself. s323 00:33:42.240 --> 00:33:54.960 U there's you know been times where a model lab would come to us early and we're kind of working with them to implement the model uh beforehand and we do this except you know they come and sometimes there's like five different variations s324 00:33:54.960 --> 00:33:57.840 of a model and you know we need to have them all working. s325 00:33:57.840 --> 00:33:59.679 So the implementation is one side of it. s326 00:33:59.679 --> 00:34:03.200 Um and I'm sure other people also have to put a lot of work in it. s327 00:34:03.200 --> 00:34:11.599 But the other kind of other side is you kind of have to run through the quantization bit and seeing you know kind of which one works best. s328 00:34:11.599 --> 00:34:19.599 And atama we kind of do a UX thing of like giving a default uh model quantization for most things uh for most models. s329 00:34:19.599 --> 00:34:29.520 Um, and a big part of that is actually, you know, us spending the time one quantizing it, but then seeing if it actually works well with like different harnesses s330 00:34:29.520 --> 00:34:40.879 and, you know, is it actually usable after and so we find that sometimes when you have a very small parameter size model, um, you don't get like great quantization s331 00:34:40.879 --> 00:34:41.679 after that. s332 00:34:41.679 --> 00:34:54.960 Um so we sometimes leave it in higher precision um as a default just because we actually want people to have a better experience uh versus you know quantizing it down quantiz yeah yeah quantizing it down to too little of a precision s333 00:34:54.960 --> 00:34:58.240 um and not having the model actually work well. s334 00:34:58.240 --> 00:35:07.839 So it's always a challenge um both from like the implementation perspective but then actually you know running into the quantization bit and making sure it still works correctly. s335 00:35:07.839 --> 00:35:18.960 And and Daniel, like is it harder to do quantization now that like everything is some hybrid or linear variant or and it's also all like sparse or variations on sparsee s336 00:35:18.960 --> 00:35:24.240 like how much harder is it today than it was when it was everything was just long all the way down. s337 00:35:24.240 --> 00:35:24.480 Yeah. s338 00:35:24.480 --> 00:35:28.880 Like in the olden days you know every model was dense the transformer plus plus. s339 00:35:28.880 --> 00:35:30.880 So it's called transformer plus architecture. s340 00:35:30.880 --> 00:35:32.400 You know it's an old transformer. s341 00:35:32.400 --> 00:35:32.640 Okay. s342 00:35:32.640 --> 00:35:35.119 Plus RMS lay plus some extra tricks. s343 00:35:35.119 --> 00:35:36.560 That's called transformer plus. s344 00:35:36.560 --> 00:35:44.880 And then now it's like oh my it's like transformer plus+ plus and then plus this thing plus that thing minus this thing minus that thing a different activation function s345 00:35:44.880 --> 00:35:59.599 linear attention here sliding window window window attention you know how many layers are sliding window how many layers are global oh let's delete global do something else blah blah blah blah blah um you know everyone likes to do their own thing and you know they like to compare you know like okay this one does better for long s346 00:35:59.599 --> 00:36:15.040 context you know this one does worse for long context or something like this so there there's always like reasons why they like to change the architecture Um you know some folks even change some of the you know layer norm epsilons like you know change 1 eus 5 to 1 eus 6 okay which one's better and so on. s347 00:36:15.040 --> 00:36:19.760 So they do a lot of abilations you know they do a lot of testing you know this one seems to be better than this one. s348 00:36:19.760 --> 00:36:24.400 Um and yes it has complicated compression and quantization dramatically. s349 00:36:24.400 --> 00:36:33.920 Um you know you have your old huristics okay this works well for for this model but then when you go to the world oh you can quantize thee layers to like one bit and it doesn't break. s350 00:36:33.920 --> 00:36:38.800 Um but then you know when you go to the linear attention world you cannot quantize the linear attention layers. s351 00:36:38.800 --> 00:36:48.880 So if you quantize the linear we found that if you quantize the linear attention layers okay it looks like it's doing good but then when you do long context benchmarks you know when you actually use the model in real production it becomes gibberish. s352 00:36:48.880 --> 00:36:52.720 Um and so like there are some layers you cannot quantize with these new architectures. s353 00:36:52.720 --> 00:36:57.839 Some layers you can you know quantize very low to like one bit you know you can even delete some layers if you like. s354 00:36:57.839 --> 00:37:01.760 Um and so it's like these new architectures complicate the process. s355 00:37:01.760 --> 00:37:06.400 Um but to be honest very happy with this because we need more different architectures. s356 00:37:06.400 --> 00:37:14.800 We don't want everyone to be like thinking the same way and you know open source has been you know the open model era has been like there's so many different architectures s357 00:37:14.800 --> 00:37:19.599 and it's very good to have like you know a variety of different opinions and architectures. s358 00:37:19.599 --> 00:37:20.320 Yeah. s359 00:37:20.320 --> 00:37:21.520 I mean that's that's dope. s360 00:37:21.520 --> 00:37:21.760 Yeah. s361 00:37:21.760 --> 00:37:22.560 Hell yeah. s362 00:37:22.560 --> 00:37:26.079 Uh I we we got about five minutes left. s363 00:37:26.079 --> 00:37:26.640 Question. s364 00:37:26.640 --> 00:37:27.440 Can we ask questions? s365 00:37:27.440 --> 00:37:31.359 Uh I I I want to get one more question with these guys and then yes. s366 00:37:31.359 --> 00:37:32.400 Uh okay. s367 00:37:32.400 --> 00:37:36.160 So compression is an art not a science right now. s368 00:37:36.160 --> 00:37:38.880 It's a it's a it's an artisal science let's say. s369 00:37:38.880 --> 00:37:41.520 Uh but where is it going? s370 00:37:41.520 --> 00:37:46.560 Uh you know what does what does compression look like uh in six months and 18 months? s371 00:37:46.560 --> 00:37:50.320 Maybe we'll just go we'll start with Marv and then we'll we'll we'll do a loop around. s372 00:37:50.320 --> 00:37:52.000 Can we start from s373 00:37:52.000 --> 00:37:52.221 Yeah, s374 00:37:52.221 --> 00:37:52.240 [laughter] s375 00:37:52.240 --> 00:37:52.720 sure. s376 00:37:52.720 --> 00:37:54.480 We'll start. s377 00:37:54.480 --> 00:37:55.119 Would you like s378 00:37:55.119 --> 00:37:55.920 Yeah, sure. s379 00:37:55.920 --> 00:37:57.920 Um, so s380 00:37:57.920 --> 00:38:06.560 I think the way what we've seen so far is compression being so critical for any model launch that comes around. s381 00:38:06.560 --> 00:38:10.720 So it seems that model labs are starting to think more about it as well. s382 00:38:10.720 --> 00:38:14.160 Um, the folks at on do a phenomenal job. s383 00:38:14.160 --> 00:38:16.800 Fingers crossed they keep putting some great stuff out. s384 00:38:16.800 --> 00:38:27.200 Um, but more than that, I think because of, you know, the model architecture is changing and kind of this awareness that model labs are having, we're starting to see more things like QAT s385 00:38:27.200 --> 00:38:28.720 uh, come out from the labs themselves. s386 00:38:28.720 --> 00:38:32.800 And I'm sure like there's ways to push even that further. s387 00:38:32.800 --> 00:38:42.320 But I think it's going to be a mix of the labs kind of becoming a little bit more interested, but also the community kind of doing the great work they already have been and pushing that further. s388 00:38:42.320 --> 00:38:46.400 Can't go now that I thought about the question. s389 00:38:46.400 --> 00:38:58.800 So basic um what I think is most of so previously we were just running models on servers and stuff but I think we can just finally push the edge because like there was this increasing s390 00:38:58.800 --> 00:39:05.680 demand for privacy and everything especially for like sensitive data personal data and so on. s391 00:39:05.680 --> 00:39:13.040 So like I see compression being a super hot topic and um we are developing a lot of cool stuff with Llama CPP. s392 00:39:13.040 --> 00:39:16.560 So I would like I would like it if you could stay tuned for that. s393 00:39:16.560 --> 00:39:19.760 Um so that it runs like everywhere. s394 00:39:19.760 --> 00:39:31.839 Um so I think I see the the so previously I saw that the we were constantly scaling the model model parameters and then the data diversity and then going down. s395 00:39:31.839 --> 00:39:40.480 I see it as like going even further to the phones and stuff because it wasn't working like we tried a lot and then it wasn't working. s396 00:39:40.480 --> 00:39:45.440 I see intelligence going to phones thanks to the quants and everything. s397 00:39:45.440 --> 00:39:46.880 So yeah, s398 00:39:46.880 --> 00:39:47.839 going to phones. s399 00:39:47.839 --> 00:39:49.359 Let's go. s400 00:39:49.359 --> 00:39:50.000 Yeah. s401 00:39:50.000 --> 00:39:50.400 Okay. s402 00:39:50.400 --> 00:39:56.480 So the way I think about compression, uh the whole space is going to go more broader. s403 00:39:56.480 --> 00:40:00.320 So we will focus this discussion mostly on weight compression. s404 00:40:00.320 --> 00:40:02.800 So uh okay, going back to FP4 once again. s405 00:40:02.800 --> 00:40:10.800 Uh so it also do weight compression and uh math acceleration because it it do the uh gem in 4bit. s406 00:40:10.800 --> 00:40:21.760 So uh yeah so for in terms of weight compression we might be able to go to like one maybe two or three bit more but in terms of u s407 00:40:21.760 --> 00:40:37.440 quantization alone we might be like close like close to the par optimality uh yeah let's let's see uh then there are more like you know more type of compressions so KB cache compression right so yeah so people are still s408 00:40:37.440 --> 00:40:45.599 mostly using 8 bit uh 4-bit large models retain that quality very well but smaller models we see some drop. s409 00:40:45.599 --> 00:40:56.960 So yeah so looking forward to KV cache uh maybe comb action plus quantization uh like pushing uh the long uh horizon reasoning uh broader than sparsity. s410 00:40:56.960 --> 00:41:07.119 So uh so okay so by the way sparsity is uh has been part of Nvidia hardware but it has not been uh like broadly adopted. s411 00:41:07.119 --> 00:41:15.440 uh that is because um quantization does not degrade accuracy that much but spec uh sparity causes accuracy degradation a bit more. s412 00:41:15.440 --> 00:41:27.599 So in Rubin there is this cool feature called dynamic activation sparity so it can uh improve attention math etc. So yeah so looking forward to that and then coming back to like you know all these heterogeneous s413 00:41:27.599 --> 00:41:45.440 architectures right so with each release the yeah the particularly the attention architecture is getting more and more complex deepseek started it I blame them they started with MLA now like you know sparse attention indexed attention like you know lot of skips soft max yeah so it's getting broader s414 00:41:45.440 --> 00:41:53.599 and yeah so it is part of this process where we make models cheaper and and they are they are having compounding effects Perfect. s415 00:41:53.599 --> 00:41:58.800 So, so we're going to quantize more things uh more is basically the idea. s416 00:41:58.800 --> 00:41:59.920 That's that's pretty dope. s417 00:41:59.920 --> 00:42:01.440 I I don't I don't mind that. s418 00:42:01.440 --> 00:42:02.560 Take us home. s419 00:42:02.560 --> 00:42:02.800 Yeah. s420 00:42:02.800 --> 00:42:07.280 So, I think like the world we're going to get more and more bigger models. s421 00:42:07.280 --> 00:42:15.040 Um and you know, it would be very cool if we can run them locally on our phones, you know, on your laptops with your GPUs. s422 00:42:15.040 --> 00:42:21.119 Um and you like imagine in a world where, you know, the best Frontier models will be able to run on your computers. s423 00:42:21.119 --> 00:42:22.400 You know, that would be so cool. s424 00:42:22.400 --> 00:42:24.640 um you know now you can control your own destiny. s425 00:42:24.640 --> 00:42:31.440 you do not need to be, you know, controlled by some model labs and now you can do whatever you like on your computer, right? s426 00:42:31.440 --> 00:42:34.160 You can do your own fine-tuning, you can customize it. s427 00:42:34.160 --> 00:42:37.440 Um, you can, you know, everything becomes yourself. s428 00:42:37.440 --> 00:42:38.240 You own it. s429 00:42:38.240 --> 00:42:48.720 Um, and so like, you know, I see a world where the world in the future, all of the intelligence will be fully democratized, you know, via GPUs, via the phone, you know, any single hardware. s430 00:42:48.720 --> 00:42:57.760 And you you will have, you know, very capable AIs on your laptop, on your local devices running, you know, and it's also going to be efficient, right? s431 00:42:57.760 --> 00:43:00.560 You don't want your computer to like, you know, lose battery and die. s432 00:43:00.560 --> 00:43:05.920 Um, but I feel like, you know, with all of these techniques, um, you know, we can have a future where AI is fully democratized. s433 00:43:05.920 --> 00:43:07.520 And that's, yeah, that's where I see it. s434 00:43:07.520 --> 00:43:08.000 Yeah, s435 00:43:08.000 --> 00:43:09.200 not a bad future. s436 00:43:09.200 --> 00:43:13.599 Uh, again, we can't do it without any all of you in the room. s437 00:43:13.599 --> 00:43:16.449 So, big round of applause for the the panel here. s438 00:43:16.449 --> 00:43:18.449 [applause] s439 00:43:19.040 --> 00:43:20.240 Thank you so much guys. s440 00:43:20.240 --> 00:43:21.359 We we have one question. s441 00:43:21.359 --> 00:43:22.480 We'll do one question. s442 00:43:22.480 --> 00:43:22.880 All right. s443 00:43:22.880 --> 00:43:32.240 So you talked about all the different permutations of craziness that are happening in the model space and the proliferation of various techniques to right. s444 00:43:32.240 --> 00:43:46.079 Um but we often you find some weird Frankenstein model out there that you want to try out but you have no clue how well it actually performs on the original benchmarks of the model before all the modifications. s445 00:43:46.079 --> 00:43:49.119 So I'm just Is there any good resource out there? s446 00:43:49.119 --> 00:43:53.280 Maybe this is a job for this place. s447 00:43:56.240 --> 00:44:13.920 All these benchmarks again on these models again to see how well they perform after all the freakish things were done to them to see how well they're going for this for that s448 00:44:13.920 --> 00:44:15.520 etc. Right. s449 00:44:15.520 --> 00:44:16.880 a resource like that. s450 00:44:16.880 --> 00:44:19.040 Just curious. s451 00:44:19.200 --> 00:44:21.520 Yeah, if you find something like that, let me know. s452 00:44:21.520 --> 00:44:22.319 Yeah. s453 00:44:22.319 --> 00:44:22.720 Yeah. s454 00:44:22.720 --> 00:44:23.920 I I think I think that's it. s455 00:44:23.920 --> 00:44:32.880 Basically, the question is what is the resource I can look at to find out what is, you know, the of the suite of crazy quants or compresses that exist for a model? s456 00:44:32.880 --> 00:44:35.839 How do I find the ones that are good at the things that I care about? s457 00:44:35.839 --> 00:44:36.960 Is anyone doing that? s458 00:44:36.960 --> 00:44:41.359 I think right now it is not being done comprehensively. s459 00:44:41.359 --> 00:44:53.760 Uh we do have some so for some models when we do our dynamic quantizations we do release benchmarks um and we do not do so generally what our view is accuracy benchmarks can be very complicated because you have to do sampling s460 00:44:53.760 --> 00:44:56.160 how many trials you need to do and you have to average. s461 00:44:56.160 --> 00:44:59.760 So there is another better method in our view um KI divergence. s462 00:44:59.760 --> 00:45:05.599 So KOD is the distance between the uncontized version which is Bflat 16 and your quantized version. s463 00:45:05.599 --> 00:45:09.920 And you can calculate some sort of distance between the quantized version and the uncquantized version. s464 00:45:09.920 --> 00:45:14.319 And your goal is to make the distance zero and the size smaller. s465 00:45:14.319 --> 00:45:16.880 Do you do it over the output logits? s466 00:45:16.880 --> 00:45:17.119 Yes. s467 00:45:17.119 --> 00:45:26.160 So you check the output logs, you pass some sort of collaboration data, you shove it in and then you have some you check the output logits between the Bflat 16 and then the main, you know, the quantized version. s468 00:45:26.160 --> 00:45:28.560 And then the goal is make this distance zero. s469 00:45:28.560 --> 00:45:29.920 Um and you make the model smaller. s470 00:45:29.920 --> 00:45:30.720 That's KLD. s471 00:45:30.720 --> 00:45:34.800 Yeah, that's K. Um KLD, uh you can check out the paper. s472 00:45:34.800 --> 00:45:37.839 Accuracy is not all you need if you want to learn more about that one. s473 00:45:37.839 --> 00:45:38.960 That's it for us guys. s474 00:45:38.960 --> 00:45:42.160 Thank you so much again to our excellent panel. s475 00:45:42.160 --> 00:45:43.171 Thank you. s476 00:45:43.171 --> 00:45:45.171 [applause] s477 00:45:48.411 --> 00:45:50.411 [music] s478 00:45:57.581 --> 00:45:59.581 [music]