WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XV2oYi7kojc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XV2oYi7kojc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.719 --> 00:00:17.520
Hey everyone, we are about to start this presentation.

s3
00:00:17.520 --> 00:00:20.305
Uh it's called the desktop frontier

s4
00:00:20.305 --> 00:00:20.720
[snorts]

s5
00:00:20.720 --> 00:00:33.280
and um [clears throat] it's basically about where we started and how far we've come with local and open source models.

s6
00:00:33.280 --> 00:00:41.520
Um how many like just a quick question how many of you here follow me on X?

s7
00:00:42.239 --> 00:00:43.840
I'm I'm amazing.

s8
00:00:43.840 --> 00:00:44.640
Love you all.

s9
00:00:44.640 --> 00:00:45.760
Love you all.

s10
00:00:45.760 --> 00:00:58.800
Uh so you know I sometimes every now and then I would say a prediction uh here is a new one within roughly 18 months we are going to have the equivalent of GLM 5.2

s11
00:00:58.800 --> 00:01:05.760
class intelligence running on a single RTX 5090 with 32 GB of VRAM.

s12
00:01:05.760 --> 00:01:10.080
Um that's basically late 2027.

s13
00:01:10.080 --> 00:01:12.400
Um this this is conservative.

s14
00:01:12.400 --> 00:01:15.520
We might actually get there faster.

s15
00:01:17.840 --> 00:01:34.479
So uh you know for a long time uh the story has been bigger models bigger models bigger models how can we get to the next 5 trillion how can we get to the 20 trillion and I'm not saying that there won't ever be like a gap between frontier intelligence

s16
00:01:34.479 --> 00:01:48.399
um and u you know open source models there will always be a gap but that gap um will shrink and the efficiency of the models will get exponentially better.

s17
00:01:51.840 --> 00:01:58.079
So the the term that I like to think about is impact per parameter.

s18
00:01:58.079 --> 00:02:01.280
Um you know what capability are we talking about?

s19
00:02:01.280 --> 00:02:03.680
What could a model do?

s20
00:02:03.680 --> 00:02:10.080
Um what footprint like hardware footprint did it have last year in comparison to now?

s21
00:02:10.080 --> 00:02:21.920
and uh what hardware does that use and what hardware does it need to use a year ago and uh you know are we moving down for the same kind of quality

s22
00:02:21.920 --> 00:02:32.959
on the on that hardware again as I was saying earlier I used to run lama 2 on an RTX 3090 it's now running qu 3.5 3.6 6 27 billion parameter.

s23
00:02:32.959 --> 00:02:35.519
That's better than Lamas 405.

s24
00:02:35.519 --> 00:02:45.680
That's a 400 billion 400 billion plus parameters model that you beat with a 27 billion parameter model a year and a half after.

s25
00:02:48.560 --> 00:02:55.120
So um yeah, as I was saying, similar capabilities are moving into smaller hardware footprint.

s26
00:02:55.120 --> 00:03:09.760
Um benchmark scores are one thing but also you know a year ago this time a year ago we didn't have any local models that were able to successfully run within clo code

s27
00:03:09.760 --> 00:03:24.080
right it wasn't until GLM 4.5 that came out in late July and GLM 4.5 air required at least four RTX uh 3090s or an RTX Pro 6000.

s28
00:03:24.080 --> 00:03:27.840
Now that footprint for hardware is not needed anymore.

s29
00:03:27.840 --> 00:03:36.720
All that you need is a single RTX 1390 1590 and you have something much more capable, much more intelligent.

s30
00:03:36.720 --> 00:03:41.120
So is this trend just random or is there more to it?

s31
00:03:41.120 --> 00:03:44.560
That's a question that everyone should ask.

s32
00:03:44.560 --> 00:03:56.799
Um, is it just by by random chance that we've gotten this far from models that were weren't able to sustain more than 4,000 um tokens in terms of context lengths

s33
00:03:56.799 --> 00:04:06.319
and now we have things that are million parame million uh tokens locally on your hardware that you own.

s34
00:04:06.319 --> 00:04:10.080
It's not by chance, you know, it's not just a coincidence that we got here.

s35
00:04:10.080 --> 00:04:11.920
Um, there is research being done.

s36
00:04:11.920 --> 00:04:14.080
There is efficiency gains to be made.

s37
00:04:14.080 --> 00:04:20.799
There are architecture hacks that compound and they will continue to compound.

s38
00:04:21.680 --> 00:04:23.680
And I think I like this line.

s39
00:04:23.680 --> 00:04:26.960
It's not that small models are beating big models.

s40
00:04:26.960 --> 00:04:32.880
It's that newer, more efficient models are beating older, less efficient ones.

s41
00:04:33.120 --> 00:04:49.120
Uh so yeah capability density uh is you know the literature I back this up with uh uh nature machine intelligence uh calls this pattern densing law and uh basically

s42
00:04:49.120 --> 00:05:02.000
um you know every three and a half months we are having 50% fear parameters whether that's in dense or activated that's a different story but we're getting way more intelligence

s43
00:05:02.000 --> 00:05:05.280
out of the models that we're running.

s44
00:05:09.840 --> 00:05:24.000
So, you know, right now where we're at, it's uh GLM 5.2. That's uh our, you know, biggest player and um it's 744 billion parameters total with only 40 billion parameter activated.

s45
00:05:24.000 --> 00:05:26.720
And that supports up to 1 million contexts.

s46
00:05:26.720 --> 00:05:35.600
You can run this in MVFP4 on a machine uh on a GGX station or on a server with eight RTX Pro 6000.

s47
00:05:35.600 --> 00:05:43.360
That's something that you like a GX station is something that you can sit under your desk and it's running this kind of frontier intelligence.

s48
00:05:43.360 --> 00:05:49.199
Whether you know it it's on one benchmark it actually beats GBT 5.5 extra high.

s49
00:05:49.199 --> 00:06:02.240
Doesn't that mean that we're getting somewhere with local and open source models that we can compete with the frontier that we're not that far off from the best that you can get from the cloud?

s50
00:06:02.400 --> 00:06:19.039
We also have Neatron 3 ultra which proved that NVFB4 training more efficient training can be done on uh on hardware right that's that's very important that means that the footprint even for training these models for fine-tuning them for making small and specialized models as I was talking earlier

s51
00:06:19.039 --> 00:06:31.759
could be more efficient could be done cheaper and could be you know could deliver you value in terms of economics way sooner or you know for much less money than you used

s52
00:06:33.120 --> 00:06:33.520
Yeah.

s53
00:06:33.520 --> 00:06:38.080
So, you know, again, Lama 2, uh, that was a 70 billion parameter model.

s54
00:06:38.080 --> 00:06:41.840
If you try to run that right now, you're you'd laugh at it, right?

s55
00:06:41.840 --> 00:06:54.400
That used to take eight RTX 1390s to load up and it those same eight RTX3090s could run something like 15 parallel agents right now with Quen 3.5 27.

s56
00:06:54.400 --> 00:06:58.560
That's that's a massive jump in terms of performance gains.

s57
00:06:58.560 --> 00:07:08.240
Um so the densing law basically means that we have similar or better um capabilities with significantly fewer parameters.

s58
00:07:08.240 --> 00:07:09.440
That's the impact per parameter.

s59
00:07:09.440 --> 00:07:21.039
As I was saying I want everybody to live here thinking about this term and you know thinking where are we going to get a year from today as I was saying earlier

s60
00:07:21.039 --> 00:07:22.160
everyone here has a phone.

s61
00:07:22.160 --> 00:07:25.599
I'm assuming raise your hand if you have a phone.

s62
00:07:25.599 --> 00:07:28.560
If you didn't raise your hand, we know you lie about other things as well.

s63
00:07:28.560 --> 00:07:31.199
So, come on, guys.

s64
00:07:31.199 --> 00:07:38.240
So, [clears throat] you know, you can run you can now run GBT40 quality on your iPhone.

s65
00:07:38.240 --> 00:07:39.759
That that's massive.

s66
00:07:39.759 --> 00:07:43.039
That thing require data centers to serve.

s67
00:07:43.039 --> 00:07:47.675
So, why wouldn't you invest, you know, in sovereign

s68
00:07:47.675 --> 00:07:48.080
[snorts]

s69
00:07:48.080 --> 00:07:48.479
AI?

s70
00:07:48.479 --> 00:07:59.280
Why wouldn't you as a consumer, as an individual, as a smalls size business, middlesiz business, enterprise, why wouldn't you want to be in control of the models that you're on?

s71
00:07:59.280 --> 00:08:04.400
Why wouldn't you want to make sure that nothing gets taken away from you?

s72
00:08:04.400 --> 00:08:18.639
That every little thing can be optimized for you later on, that the performance gains can be made specially and specifically for your use cases, and that you can save more money that way in the long run.

s73
00:08:19.120 --> 00:08:33.120
and you know ODS for consumers it's basically the way that we support individuals but enterprises also and I think that there is something that we like as a community we need to think about deeply

s74
00:08:33.120 --> 00:08:45.440
we need enterprises for open source AI to win we need these people that are using the cloud right now that are basically supporting data centers being built for cloud providers

s75
00:08:45.440 --> 00:09:00.480
to come on this side to own their own hardware to own the stack fully end to end so that we can keep delivering open source models so that there is an incentive for open source providers to actually come up with models so that we can come up with new licenses

s76
00:09:00.480 --> 00:09:22.240
that allows open source to thrive and so again um open weight and the frontier I think I yeah sorry that was a misclick um [clears throat] you know so smaller models started bunching above the weight after lama 2 with mistral 7B

s77
00:09:22.240 --> 00:09:32.560
one of my favorite models if you try to build that model right now and um clo code or oven code it's not going to work but it used to take so much

s78
00:09:32.560 --> 00:09:43.920
in terms of hardware right that you would now get from a 9b mill model that I can run with telegram with or with hermes for example and uh do a lot of stuff with

s79
00:09:43.920 --> 00:09:46.160
so we've come a long Okay.

s80
00:09:46.160 --> 00:09:51.920
Uh we had that we had mixed trial 8 by 7B which you know everybody knows is an MOE.

s81
00:09:51.920 --> 00:10:02.959
Then the progression went uh from that to Lamas 3 you know Lamas 3 8B was one of my favorites still is um it had unique identity in my opinion.

s82
00:10:02.959 --> 00:10:10.640
Uh then we had like the 70 billion which was like the the thing that I would run basically on my 8 RTX3090s at home.

s83
00:10:10.640 --> 00:10:23.040
Then there was like the 405 the 400 billion plus parameter lamas which again required a lot of hardware and if you put it now against 3.5 the 27 billion parameter

s84
00:10:23.040 --> 00:10:34.000
would lose against it that's in the span of what two years two years and some no I think I think less than two years that's summer 2024 to March 2026

s85
00:10:34.000 --> 00:10:48.959
that's uh that's about 21 months and the next big thing in my opinion gamma 27V and then we had the Quinn 2.5 and that that was the moment that I was like okay we actually are making progress and the gap was shrinking

s86
00:10:48.959 --> 00:11:01.760
between open source models and uh the frontier uh really lamas 3 saved like you know it really helped us a lot and then um Gwen 2.5 delivered a massive improvement

s87
00:11:01.760 --> 00:11:12.079
and there was a lot of fine-tuning and experiments that could be done on that one there was amazing papers and um they helped the community immensely in my opinion.

s88
00:11:12.079 --> 00:11:18.399
Then the next big thing was Deepseek R1 in my opinion and uh reasoning becoming something that you can run at home.

s89
00:11:18.399 --> 00:11:22.079
That was a massive MOE almost 700 billion parameters.

s90
00:11:22.079 --> 00:11:28.560
Um you know you had to have like a very beefy server to actually get it up and running.

s91
00:11:28.560 --> 00:11:40.640
Um and then you know the improvements that came from just more training on that one and Deepseek R1 that was released in May last year made massive jump again.

s92
00:11:40.640 --> 00:11:49.279
So it showed that post training could deliver more improvements on the same on the same checkpoints.

s93
00:11:53.279 --> 00:11:57.279
Then GBT open source like GBT OSS 12B.

s94
00:11:57.279 --> 00:12:00.000
Anyone remembers that one from last summer?

s95
00:12:00.000 --> 00:12:01.519
Yeah.

s96
00:12:01.519 --> 00:12:03.519
Nobody here used it.

s97
00:12:03.519 --> 00:12:05.040
Come on, guys.

s98
00:12:05.040 --> 00:12:07.058
I I need some help here.

s99
00:12:07.058 --> 00:12:07.920
[laughter]

s100
00:12:07.920 --> 00:12:14.720
Uh it was it was it was one of the first open source models that were able to successfully do tool calling.

s101
00:12:14.720 --> 00:12:17.360
Um and uh it was a step forward.

s102
00:12:17.360 --> 00:12:22.480
It showed us that we can do more with uh with the hardware that we have at running at home.

s103
00:12:22.480 --> 00:12:35.760
Uh that was a footprint shift right from like you know that massive 700 billion parameters deepseek uh R1 that was yeah 671 billion parameters to something that was 1/5

s104
00:12:35.760 --> 00:12:42.959
of its size in GBTSS with comparable maybe better more agentic performance.

s105
00:12:42.959 --> 00:12:43.738
Um

s106
00:12:43.738 --> 00:12:45.738
[snorts]

s107
00:12:48.399 --> 00:13:03.360
then the moment of uh quinc 3.5 that's 397 that's three that's 397 billion parameters uh that's uh that's a BVOE and uh you know what's funny is that about

s108
00:13:03.360 --> 00:13:17.839
uh it's about 15 times the size of uh the Quinn 3.6 and I'm here I'm comparing 3.5 to 3.6 6 of the dense 27 billion parameter model and that dense model beats it and that dense model

s109
00:13:17.839 --> 00:13:21.839
has 40% higher number of activated parameters.

s110
00:13:21.839 --> 00:13:24.240
So it's not that far off.

s111
00:13:24.240 --> 00:13:34.399
That's massive amount of performance gains and a very small amount of time with massively different footprint in terms of hardware uh requirements.

s112
00:13:34.399 --> 00:13:37.760
And that trend happened in like what two three months.

s113
00:13:37.760 --> 00:13:40.639
So you know how far could we go from here?

s114
00:13:40.639 --> 00:13:51.440
Um how far before we get to you know uh a recent model that there was some news about you know that is uh finally relaunch the game.

s115
00:13:51.440 --> 00:14:02.959
How far before open source delivers something of that quality that you could run on your own hardware and you can control and will not be taken away from you and will not refuse

s116
00:14:02.959 --> 00:14:05.760
a request from you.

s117
00:14:08.320 --> 00:14:22.480
So again these are just some benchmarks where you can see that an iteration on the 27 billion parameter model a little bit more post training proved it across all benchmarks and made it

s118
00:14:22.480 --> 00:14:29.920
one against a model that is almost 15 size for 15 times its size.

s119
00:14:32.800 --> 00:14:38.639
And again, remember this is 27 billion parameters activated versus 17 billion parameter activated.

s120
00:14:38.639 --> 00:14:50.079
It's still massively the same amount like you know it's it's only 40% less in terms of the amount of time it would take to process things but it's 15 times smaller.

s121
00:14:50.079 --> 00:14:52.480
That's a lot.

s122
00:14:53.040 --> 00:15:04.720
So again um how long until the prediction I made earlier becomes plausible when I said that we're going to have the equivalent of GLM 5.2 2 running on an RTX

s123
00:15:04.720 --> 00:15:06.560
uh 1590.

s124
00:15:06.560 --> 00:15:11.440
This is the mass 17 months and this is a conservative math.

s125
00:15:11.440 --> 00:15:22.959
Uh earlier this year in December, I had a a very viral post that I predicted that we're going to have the quality of OBS 4.5 running locally at home on a single RTX

s126
00:15:22.959 --> 00:15:24.240
uh Pro 6000.

s127
00:15:24.240 --> 00:15:27.040
That happened by March.

s128
00:15:30.959 --> 00:15:43.440
So a question um hardware purchase today does it get more valuable as models become more efficient and smaller in size?

s129
00:15:43.600 --> 00:15:44.639
That's a good question.

s130
00:15:44.639 --> 00:16:02.800
So why are you funding other people to build data centers so that you can subscribe to them and pay subsidized tokens and then later on get those subsidies are going to go away and you're not going to be able to run those models and they will have so many limitations.

s131
00:16:02.800 --> 00:16:07.759
So might as well ask yourself why not own the hardware yourself and be in control.

s132
00:16:07.759 --> 00:16:16.000
Um, so yeah, the forwardl lookinging question is basically what will a DGX station be able to run in three, six, 12, 18 months from now?

s133
00:16:16.000 --> 00:16:20.800
That's something that there is a reason that I'm not selling any of my RTX3090s if you follow me.

s134
00:16:20.800 --> 00:16:22.880
And I have a lot of hardware, guys.

s135
00:16:22.880 --> 00:16:31.920
U but I'm interested in seeing what I could do with them in a year or two from now more than in the amount of money I would get for them today.

s136
00:16:31.920 --> 00:16:33.600
This is not a financial advice by the way.

s137
00:16:33.600 --> 00:16:36.639
Like let me make that very clear.

s138
00:16:36.800 --> 00:16:51.199
Um so yeah um the disk side frontier potential um you know an Nvidia GX station could run a lot of uh today it could run GLM 5.2 to what will it be able to run tomorrow

s139
00:16:51.199 --> 00:16:55.261
6 months 18 months two years from today we know that you know RTX3090

s140
00:16:55.261 --> 00:16:56.079
[snorts]

s141
00:16:56.079 --> 00:17:09.439
is the amber uh architecture from 2020 sells at higher value than MSRP today and it's still being utilized for a lot of use cases so what well at GGX station the actively

s142
00:17:09.439 --> 00:17:18.959
developed blackwell architecture will be able to run in a few months a couple of years that's a Good question.

s143
00:17:18.959 --> 00:17:38.240
So the question you have to ask yourself um if an RTX30590 with 32 GB of VRAM runs in the equivalent of a GLM 5.2 and 18 months and this is the question that everybody should be asking themselves and I want you all to be looking at the screen taking this very seriously.

s144
00:17:38.240 --> 00:17:40.240
Okay.

s145
00:17:40.480 --> 00:17:43.200
Should you buy a GPU?

s146
00:17:44.480 --> 00:17:46.720
Thank you.